Friday 1 February 2013

Engineering with agile - is it possible?

Engineering is the application of scientific, economic, social, and practical knowledge, in order to design, build, and maintain structures, machines, devices, systems, materials and processes.

The engineering approach in software development can be seen as a top-down process.
Starting from the problem, first of all it has to be deeply understood by analytics before the solution is proposed by designers. Finally it is implemented as a working thing by programmers in a selected programming language.
A good example of software engineering approach to software development is the Rational Unified Process (RUP). RUP consists of four major steps, that if followed, should result in a software system that fulfills specific technological and market needs. Each step requires the stakeholders to be equipped with specific competences (requirement engineers, analytics, designers, programmer and testers) and tools. RUP was proven to be useful in the development of many complex systems; however it requires the high organizational level of the stakeholders. 
Even if it is not specified explicitly, it is impossible to build a software by using RUP in organizations that do not have strictly defined hierarchy and communication channels of competitions. Moreover RUP requires knowledge management activities. Therefore RUP is primarily used in big corporations that are able to handle such complex hierarchies of stakeholders, and are able to effectively evaluate their outcome. Every RUP based project requires four phases that result in specific artifacts:
1)     Inception (requirement analysis) is used to identify the requirements as well as the scope of the software solution that is going to be released. Tasks should determine the needs or conditions required to meet the expectations, taking into account the potential conflict of requirements of the various stakeholders, such as beneficiaries or users.
2)     Elaboration (architecture and design identification) tries to identify an architecture that has a good chance of working. The architecture is often defined with diagrams, which explore the technical infrastructure, the major business entities and their relationships. The design is derived in a modeling session, in which issues are explored until the team is satisfied that they understand what needs to be delivered.
3)     Construction (software implementation), where the main focus is on the development of components and other features of the system. This is the phase in which the majority of the coding takes place. In larger projects, several construction iterations may be developed in an effort to divide the use cases into manageable segments that produce demonstrable prototypes.
4)     Transition (integration and tests). The primary objective here is to 'transit' the system from development into production, making it available to the end user and understood by him. The activities of this phase include training the end users and maintainers, and beta testing the system to validate it against the end users' expectations. The product is also checked if it satisfies the required quality level.
Critics say that RUP is a ‘high ceremony methodology’ because it demands all the requirements to be collected before starting the design phase. Once they are collected they need to be frozen before starting the development. However; it is very common that the requirements are not known in details or even a customer may require the features that are not needed at the end. Once the RUP process starts, all change requirements are recognized as additional cost. Also the process that once started, requires bureaucracy (every document must be approved in hierarchy of stakeholders), therefore it is considered as a slow and demanding method of software development.

An emergent behavior or emergent property can appear when a number of simple entities (agents) operate in an environment, forming more complex behaviors as a collective. This stream includes: neural networks, genetic programming, expert systems and many others AI activities.
With emergence as a key concept, it is also easier to understand what agile software development methodologies are proposing. If we take a look at a software as a result of work of group of programmers, where each one has a different background in field of software development and different psychological skills, it starts to become clear why it is so difficult to build the system related to the specific needs. Without consistent specification and prior educational task it is even harder. The agile methodology approaches this problem by focusing on building the ground for optimal cooperation between the stakeholders. The success of agile methodologies, that is currently observed, proves that even without specification and education, we can still build effective programs - what a surprise, with smaller budget and in a shorter time. Agile methods break tasks into small increments with minimal planning, and do not directly involve long-term planning. Agile Manifesto reads, in its entirety, as follows:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
o   Individuals and interactions over processes and tools
o   Working software over comprehensive documentation
o   Customer collaboration over contract negotiation
o   Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Agile methodologies, even if it is not clearly stated in manifesto, try to develop software products in emergent (self-organizing) way. Each agile programmer can be seen as a process that produces the part of computer system which in the end is a superposition of emergent work of the team. The core idea is the organization of the environment that will allow for emergent work e.g.:
-          Programming is made by  teams of competitive programmers, that handle the knowledge management inside the team
-          The team produces programs as well as other programs that are to prove that the  earlier mentioned ones are working
-          There is an automated process that verifies the core properties of the system of programs that is going to emerge

One can say that the computer language is used to communicate with a computer, however nowadays, computer language is becoming more a communication language between stakeholders involved into software engineering process. Agile methodology these days, tends to be the standard for production of complex systems.
Without a specification it is impossible to build the system in any methodology based on engineering, while agile methods emphasize face-to-face communication over written documents to overcome this limitation.  This observation proves that agile methodologies are focused on psychological aspects rather than formal ones, therefore they are opposite to engeenering (see. Is it possible to merge them together?

Software Models - what it is all about?

The rapid development of software engineering methodologies that has occurred with increasing complexity of computer programs is related to the need for a way to ascribe and analyze the software intensive systems at the time of their formation. 
In November 1997 the OMG consortium established the standard of software design and analysis. This standard is nowadays well known as a Unified Modeling Language (UML), and is mainly focused on graphical modeling of software intensive systems with diagrams. To support the design of large-scale industrial applications, sophisticated CASE tools[1], which provide a user-friendly environment for editing, storing, and accessing multiple UML diagrams, are available on the market. 
The spectrum of UML diagrams is divided into two main branches: structural and behavioral. Structural diagrams emphasize the things that must be present in the system being modeled. The ‘must’ means here that we cannot say that the software is realized fully, if it does not implement the diagram of structure. This is dual to diagrams of behavior, which emphasize what must happen in the system being modeled. In the formal way we can distinguish those two classes of diagrams in the meaning of time or modality. 
While structure diagrams must always be true, the behavioral diagrams must be true in some – strictly defined - circumstances. Therefore, while structural diagram represent specification for “ALWAYS MUST BE …” the behavioral diagram is “IN A SPECIFIC SITUATION THE PROGRAM MUST SATISFY A CONDITION THAT…”. This duality led us to think about separation of the diagrams in strict, logical manner. The structural diagram specifies the terminology which must be obeyed, while the behavioral diagram describes the situation that takes place in specific space and time – the world (in terms of modality) description.
UML allows for modeling the structure of relationships between use cases, which can be viewed as behavioral and structural projects requirements. In addition, UML allows the modeling software, that is object-oriented in terms of classes, objects their hierarchy and their collaboration as well. To build the bridge between use-cases and classes (that are explicitly implemented by programmers) is a key task for a software designer. 
Using several use-case diagrams, the software designer tries to build the object-oriented design that fulfills all the requirements specified, trying to preserve non-functional (like: performance, traceability, scalability, etc…) demands given by chosen technology. The work is usually done also by using UML diagrams, but UML diagrams are in fact only graphical artifacts  and therefore it is a designer responsibility to preserve  consistency of the described here knowledge.

Spectrum of UML diagrams

The architectural view of UML is based on 4+1 View Model[2] - a view model for "describing the architecture of software-intensive systems, based on the use of multiple, concurrent views". The views are used to describe the system from the viewpoint of different stakeholders. There are four major views: (1) the logical view that should be delivered by the object model of the design, (2) the development view, which depicts the development environment of a software product, (3) the process view that represents concurrency and synchronization aspects of a design and (4) the physical view that elaborates on the mapping between software and hardware components (see Figure 15). In addition, selected use cases or scenarios are utilized to illustrate the architecture serving as the +1 view.
  • The (1) - logical view  describes the functionality that the system provides, supported with UML diagrams, including Class diagram, Communication diagram and Sequence diagram, which are used to represent the logical view.
  • The (2) development view illustrates the system from a programmer's perspective (and software management) and it uses the UML Component diagram to describe system components. UML Diagrams are used to represent the development view, including the Package diagram.
  • The (3) process view deals with the dynamic aspects of software, as it explains the system processes, how they communicate, and it also focuses on the runtime behavior of the system. It addresses e.g.: concurrency, distribution, integrators, performance, and scalability. UML Diagrams that represent process view are e.g.: the Activity/Collaboration diagram.
  • The (4) physical view depicts the system from a system engineer's point-of-view, as it is concerned with the topology of software components on the physical layer and is represented with UML Diagrams, including the Deployment diagram. The description of architecture is illustrated by a small set of use cases, or scenarios which become a 
  • +1 view that describes sequences of interactions between objects, and between processes. Those UML diagrams are used to identify architectural elements, to illustrate and validate the architecture design and to serve as a starting point for tests of an architecture prototype. UML Diagrams are used to represent the scenario view, including the Use case diagram. 

4+1 Architectural View Model

Model Driven Engineering

Modeling is essential to every engineering activity as every action here is preceded by the construction (implicit or explicit) of a model. If the model is incorrect, the action may be inappropriate. According to the definition, a model is an abstraction of phenomena in the real world; a metamodel is yet another abstraction, highlighting properties of the model itself. A model conforms to its metamodel in the way that a computer program conforms to the grammar of the programming language in which it is written. 
Model Driven Engineering (MDE) is the successor of CASE tools as well as a unification of methodologies based on UML approach. The architecture of metamodeling (called Model Driven Architecture (MDA), introduced in 2001 by the Object Management Group (OMG)) is the basis for building MDE software systems. MDE can be shortly ascribed in the following comparison: in object-oriented engineering “everything is an object” and in MDE “everything is a model”.

Object-oriented vs. model driven engineering

Every model in software development forms a graph; therefore software modeling activity can be perceived as an activity that tries to construct graphs which model the software conceptualization. The way to transform one graph into another is the key idea that lies behind MDE.  For transforming one graph into another there is a requirement of unidirectional function which is realized with graph transformation languages. 
Support for change propagation QVT is based on the Meta-Object Facility (MOF)[3]. MOF is designed as a four-layered architecture that provides a meta-meta model at the top layer called the M3 layer – that itself forms a language used by MOF to build metamodels, (called M2-models) and it is also able to describe itself, so no additional Mn... layers that are required to complete the unification. The most prominent example of a model in M2 is the UML metamodel, the model that describes the UML. These M2 models describe elements of the M1, and thus M1 models as well. Those would be, for example, models written in UML. The last layer is the M0, used to describe real-world objects. Because of the similarities between the MOF M3 models and UML structure models, MOF metamodels are usually modeled as the UML class diagrams. A supporting standard of MOF is XMI[4], which defines an XML-based exchange format for models on the M3-, M2-, or M1-Layer.
MDE requires systematic use of Model Transformation Languages (MTL). The OMG has proposed a standard called QVT for Queries/Views/Transformations, that is an implementation of MTL, however the model transformation is a general technique that tries to construct one model (lower) from the another (higher) and therefore there exist other, very usable, transformation languages. As each model is a graph, model transformation is based on graph transformation. The QVT transformation has a support of model integration rules, model consistency checking and uni/bidirectional model transformations for a declarative or operational specifications. It is also equipped with either textual or graphical notation.
The modern MDE vision does not use models only as a simple documentation but as a formal input for software tools implementing precise operations. As a consequence model-engineering frameworks have progressively evolved towards solid proposals like the MDA defined by the OMG. Carrier of information in here is the OWL [HKP+09] which enables the exchange of models (or portions thereof) between different systems, thus ensuring the implementation of the concept of re-use (in the phase of modeling and design system).  This approach is similar to the OASE approach. The Ontology Definition MetaModel (ODM) is to make the concepts of MDA applicable to the engineering of ontologies. The features available in UML are mapped here to OWL elements (OWL can be seen as XML representations of DL).

Four layers of the modeling hierarchy

Service Oriented Modeling

Service-oriented modeling (SOM) is the discipline of modeling of service‑oriented systems within a variety of architectural styles. It encourages viewing software entities as 'assets'. It refers to these assets collectively as 'services' that have properties of physical objects e.g.: location, price…, which can replicate and organize themselves. The Service Oriented Architecture (SOA) is an application of SOM defined as “a paradigm for organizing and utilizing distributed compatibilities that may be under the control of different membership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired consistent with measurable preconditions and expectations”[6]. Services evolved as a result of observation, that nowadays we are occupied with loosely coupled, interoperable software, available in a form of small pluggable units rather than large, centralized software. Those pluggable units need to be fault-tolerant pieces of software that are continuously improving their quality basing on business demand.  
 Conceptual architecture offers mechanisms for describing the proposed technological solution; logical architecture discipline is chiefly concerned with asset reuse, utilization and consumption, while the physical architecture is the resulting tangible architecture construct (that itself is not a purpose of SOM). Architectural concepts in SOM are related to the categories of “machines” like:
1)     Workflow Machine: based on states of execution, each of which is assigned a certain goal to fulfill.
2)     Connecting Machine: describes communication methods and mediating mechanisms between various software assets in a distributed technological environment. 
3)     Time Machine: is characteristically associated with time and calendar scheduling of imperative business and technological missions. 
4)     Transformation Machine: fills incompatibility gap between operating systems, communication protocols, data formats, etc.
5)     Rendering Machine: describes presentation layer solutions that enable users to communicate with backend services.
6)     Data Machine: enables data manipulation and handling activities, data serving, such as data aggregation, validation, searching, and enrichment.

Each machine is ought to be implemented as a service that has lifecycle composed of circular process: from design to runtime. Going further, services can be seen as software assets analogous to living cell of complex biological organism. In this case, the organism is a metaphor of a complex business system that involves many services in order to satisfy its needs. Cloud computing, the modern offspring of grid computing is nowadays considered as a field where service-oriented modeling can be used as an effective modeling method. Service-oriented modeling has much in common with object-oriented modeling and MDE, as all of them are trying to use the metaphor of physical objects to represent the software assets. Going further, we can say (paraphrasing MDE metaphor) that in SOM “everything is a service”.


Programming is a process of generating domain specific languages that create words in other, higher level languages[7]. MDE is a methodology of building such the domain-specific languages by using the language of MOF meta‑metamodel.  In SOM everything is service and every activity is a service activity, including activities of the people involved.[8]. The language of SOM is a language of services, which is also used by human-beings. We classify languages in the following manner:
1)     Computer languages (used by programmers during service life-cycle) including domain specific languages and general purpose ones,
2)     Inter-service communication languages (used by services to communicate),
3)     Human-service interaction languages (realized by User-Interface),
4)     Natural languages (used to communicate between programmers and service users)
The separation between human (the constructor) and machine (the material) is blurred nowadays. In crowd-sourcing services people starts to be services, controlled by machines within large service clusters[9]. To improve communication between human and machine, the innovations in user-computer interactions is needed, especially in natural language processing and human-machine interaction. On the other hand, the software developer without support of machine is not able to construct the software in a right way - moreover, it is starting to be clear that complexity of software requires usage of formal methods, that can form sort of rails on which programmer can be safely conducted over the software development process.

[1] e.g.: Rational Rose (made by Rational Software Corporation - now IBM) is used for object-oriented analysis and design),  Telelogic TAU (made by Telelogic - now IBM) modeling tool supporting automated code generation and model verification, StarUML (Open Source) – UML modeling tool, etc…
[2]    A framework that defines a coherent set of views to be used in the construction of a system architecture, software architecture, or enterprise architecture. A view is a representation of a whole system from the perspective of a related set of concerns.
[6] Web Services are the example of successful application of SOA - built upon the infrastructure of WWW and HTTP protocol.
[7] A famous aphorism of David Wheeler is: “All problems in computer science can be solved by another level of indirection.”
[8] E.g.: The Amazon Mechanical Turk (MTurk) is a crowdsourcing Internet marketplace that enables computer programmers (known as Requesters) to co-ordinate the use of human intelligence to perform tasks that computers are unable to do yet. It is one of the suites of Amazon Web Services.
[9] A collection of distributed and related services that are gathered because of their mutual business or technological commonalities.