Engineering with agile - is it possible?

Engineering is the application of scientific, economic, social, and practical knowledge, in order to design, build, and maintain structures, machines, devices, systems, materials and processes. http://en.wikipedia.org/wiki/Engineering


The engineering approach in software development can be seen as a top-down process.
Starting from the problem, first of all it has to be deeply understood by analytics before the solution is proposed by designers. Finally it is implemented as a working thing by programmers in a selected programming language.
A good example of software engineering approach to software development is the Rational Unified Process (RUP). RUP consists of four major steps, that if followed, should result in a software system that fulfills specific technological and market needs. Each step requires the stakeholders to be equipped with specific competences (requirement engineers, analytics, designers, programmer and testers) and tools. RUP was proven to be useful in the development of many complex systems; however it requires the high organizational level of the stakeholders. 
Even if it is not specified explicitly, it is impossible to build a software by using RUP in organizations that do not have strictly defined hierarchy and communication channels of competitions. Moreover RUP requires knowledge management activities. Therefore RUP is primarily used in big corporations that are able to handle such complex hierarchies of stakeholders, and are able to effectively evaluate their outcome. Every RUP based project requires four phases that result in specific artifacts:

1)     Inception (requirement analysis) is used to identify the requirements as well as the scope of the software solution that is going to be released. Tasks should determine the needs or conditions required to meet the expectations, taking into account the potential conflict of requirements of the various stakeholders, such as beneficiaries or users.

2)     Elaboration (architecture and design identification) tries to identify an architecture that has a good chance of working. The architecture is often defined with diagrams, which explore the technical infrastructure, the major business entities and their relationships. The design is derived in a modeling session, in which issues are explored until the team is satisfied that they understand what needs to be delivered.

3)     Construction (software implementation), where the main focus is on the development of components and other features of the system. This is the phase in which the majority of the coding takes place. In larger projects, several construction iterations may be developed in an effort to divide the use cases into manageable segments that produce demonstrable prototypes.

4)     Transition (integration and tests). The primary objective here is to 'transit' the system from development into production, making it available to the end user and understood by him. The activities of this phase include training the end users and maintainers, and beta testing the system to validate it against the end users' expectations. The product is also checked if it satisfies the required quality level.

Critics say that RUP is a ‘high ceremony methodology’ because it demands all the requirements to be collected before starting the design phase. Once they are collected they need to be frozen before starting the development. However; it is very common that the requirements are not known in details or even a customer may require the features that are not needed at the end. Once the RUP process starts, all change requirements are recognized as additional cost. Also the process that once started, requires bureaucracy (every document must be approved in hierarchy of stakeholders), therefore it is considered as a slow and demanding method of software development.

An emergent behavior or emergent property can appear when a number of simple entities (agents) operate in an environment, forming more complex behaviors as a collective. This stream includes: neural networks, genetic programming, expert systems and many others AI activities.

With emergence as a key concept, it is also easier to understand what agile software development methodologies are proposing. If we take a look at a software as a result of work of group of programmers, where each one has a different background in field of software development and different psychological skills, it starts to become clear why it is so difficult to build the system related to the specific needs. Without consistent specification and prior educational task it is even harder. The agile methodology approaches this problem by focusing on building the ground for optimal cooperation between the stakeholders. The success of agile methodologies, that is currently observed, proves that even without specification and education, we can still build effective programs - what a surprise, with smaller budget and in a shorter time. Agile methods break tasks into small increments with minimal planning, and do not directly involve long-term planning. Agile Manifesto reads, in its entirety, as follows:

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

o   Individuals and interactions over processes and tools

o   Working software over comprehensive documentation

o   Customer collaboration over contract negotiation

o   Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

Agile methodologies, even if it is not clearly stated in manifesto, try to develop software products in emergent (self-organizing) way. Each agile programmer can be seen as a process that produces the part of computer system which in the end is a superposition of emergent work of the team. The core idea is the organization of the environment that will allow for emergent work e.g.:

-          Programming is made by  teams of competitive programmers, that handle the knowledge management inside the team

-          The team produces programs as well as other programs that are to prove that the  earlier mentioned ones are working

-          There is an automated process that verifies the core properties of the system of programs that is going to emerge

One can say that the computer language is used to communicate with a computer, however nowadays, computer language is becoming more a communication language between stakeholders involved into software engineering process. Agile methodology these days, tends to be the standard for production of complex systems.
Without a specification it is impossible to build the system in any methodology based on engineering, while agile methods emphasize face-to-face communication over written documents to overcome this limitation.  This observation proves that agile methodologies are focused on psychological aspects rather than formal ones, therefore they are opposite to engeenering (see.
http://en.wikipedia.org/wiki/Engineering). Is it possible to merge them together?