The Agile Concurrent Software Process

ABSTRACT

The major activities required in a software development life cycle (SDLC) were identified in the waterfall model. The agile concurrent software process model proposes that the activities identified in the waterfall model are not done sequentially; these activities progress concurrently with varying intensities during the entire software life cycle. Also, a focus on a working software subset as opposed to documents, giving high importance to customer satisfaction and sensitivity to human factors go a long way in contributing towards a software project's overall success.

Software Process Models tell us about the major steps that need to be followed in a software development life cycle (SDLC). Software development life cycle, or the software development process is basically a breakdown of total work that needs to be done into several well defined activities for creating a software product. The concept of software process model is thus a central idea in Software Engineering, which is a discipline involved in study and dissemination of sound software development and management practices.

Software Process Models have evolved during the last forty years. In the late 1960s, “software crisis” was identified as the lack of ability to develop correct and reliable software for large applications that had become possible due to advances in hardware technology. Software projects were typically plagued by cost and time overruns and the performance of software developed was often not reliable. Software Engineering emerged as an important discipline, similar to production or operations management in industry. People started putting their experiences and intuition together to discover the basic principles which, if followed, would make a significant contribution towards achieving success in creation of software products.

It was understood that software had a “life-cycle” starting from the decision to make that software to the time the software became operational. The term software development life-cycle was based on a basic software process model, indicating the stages software would go through in its life-cycle. The first response to the software crisis was the discovery of the waterfall software process model. It was thought that the root cause of the problem was that the requirements were not understood well enough before the software design and proper design was not done prior to coding. Applying the engineering paradigm of freezing requirements before design and freezing design before manufacture, the waterfall model stipulated that software should be made in a sequential series of stages, like requirements' analysis, design, detailed design, coding, integration, installation, operations and maintenance. Slowly, it became apparent that it was not possible to have water-tight compartments and it was desirable to have feedback between successive stages. Thus a design error found during coding would lead to a correction of design and then the coding would continue. This was in line with the engineering practice for a design error found during manufacture is first corrected in design and than manufacturing is done as per the modified design. Also, taking the clue from engineering again, it was felt that making a prototype of the system in early stages would help in understanding the requirements better and would eventually help in making a better system.

Figure 1: The Waterfall Software Process model

The problems encountered with the waterfall model are listed in the classic paper from Parnas and Clements [1]. It was felt that software development was too intricate a process to be captured in a sequential waterfall model. Normally requirements are not understood at the beginning (the first stage) and when a project goes on with erroneous requirements, it is doomed to be a failure. Similarly design errors found late in the testing or integration stages would put the project in jeopardy as some requirements would not be achieved. One had to live with these errors for the rest of the life cycle. To overcome these problems, the evolutionary model, was formulated. In this model, software would be developed incrementally, Initially, a subset of requirements would be taken up. These were designed, coded and tested. With time, more and more requirements would be added. This helped in better control as the size of software under development, at any time, was smaller. However, the problem was that if the subsets were not well chosen, the simpler ones got developed earlier and would not work well with the more complex developed later. Or the design for the earlier developed parts was not suitable for the parts developed later.

The spiral model [2] was a further improvement on the evolutionary model. It integrates the concepts of project objectives, risk management, prototyping and project cost to the evolutionary model. In the spiral model the development is done in “cycles” of a spiral. Each cycle starts with determination of objectives, alternatives and constraints for the cycle. With a good project overview, the objectives of a cycle can be decided. Having the objectives in place, the next step in a cycle is to evaluate alternatives and identify and resolve risks. In order to manage the risks involved in a cycle, prototyping, simulation, bench-marking, etc. are done. Thus the project team is able to zero-in on the best methods and tools for development. The next step in the cycle is to develop as per the objectives and methods decided for the cycle. In an early cycle, the development might involve development of concept of operation or software requirements, whereas in a later cycle it might mean development of software design, detailed design, or code. Development also involves validation of the developed output. After development, the last part of the cycle is to reflect on the work done in the cycle and plan for the next cycle. In an early cycle, it might be requirements or life cycle plan, whereas later on, it might be an update of earlier plans (based on the experiences of the cycle), development plan or integration and test plan. Each cycle has an associated cost. The radial dimension of the spiral is the project cost and the angular dimension is the project progress for a cycle. The spiral model was a big improvement as it stressed on evolutionary incremental development, risk management, prototyping, and project overview and planning.

Figure 2: The Spiral Software Process model

If we look at the evolution of software process models, certain trends are evident. First, software can not be developed in an ad-hoc code and fix way, we have to be clear about software process model principles so that we take definite steps forward in the right direction. There are clear-cut activities like understanding and statement of requirements, software architecture and design, coding, testing, integration, system testing and installation. However since software is intangible and software products affect the environment in a many ways, timely feedback is necessary for making a correct usable product. Also, it helps the developers to make a scaled-down prototype, see how it works and then make a bigger prototype or the final product. Also, we need to keep in mind the big picture all the time, the project overview, so that we are aware about the relationship of the parts with the whole and make sure that the parts add up to the whole.

The second point is regarding the overhead of following the software process model. With all the knowledge-base of standards, templates, reports and stages, this can become a lot of work and might even overshadow the actual software design and development work. This is similar to making such elaborate arrangements for a child’s birthday celebrations that the child (and his or her happiness) is forgotten in the process. Another analogy is to a situation where management processes become too complicated and the production suffers. One of the objectives of engineering disciplines like industrial engineering has been to simplify processes so that the efficiency is improved. Closer home, simplicity and efficiency have always been key virtues in software development. The key question to be asked is: “Is the software process model being followed too heavy such that it is a hindrance instead of being a facilitator for the software project?”

One problem with the conventional software process models like the waterfall model was that the initial stages like requirements’ analysis would usually take a long time. Since the project time span would have been fixed in advance (with a possible penalty for delay in completion), there would be resistance to increasing overall project time span. An attempt would be made to “speed up” the coding work. The developers would have anticipated this well in advance and would have started development and testing of key components while requirements’ study was being done. In a real life project following the waterfall or evolutionary model, multiple activities would be going on concurrently at any time. There would be a main activity like requirements’ analysis or design or coding. Then there would be other activities with lesser intensity. There might be a spurt in a particular activity at a time. Like, if the hardware vendor delivers the new computer, the developers might spend some focused time on developing a prototype on the new machine. But it is an inescapable fact that in the waterfall model with a bigger span or spiral model with a smaller span of activities per cycle, it is not that a single activity being done at a time; there are multiple activities with different intensities that are going on at any given time. So, for example, for a typical software project, the time spent during a month during requirements’ study might be as shown in Figure 3. Although, primarily, the study of requirements is being done, some design work, coding, testing and integration are also being done.

Figure 3: An example of time spent on various activities during Requirements’ study

In a real project, all the major activities progress at different rates all the time. For example, formulation of requirements might be being done at a particular time. But the study of requirements might trigger some design, coding or even integration activity. We might want to see whether a certain requirement is feasible or not. Or whether a desired performance level can be achieved for a particular requirement. Or simply, activities like design, coding, testing take a lot of time. Why wait till the end of requirements phase to start design when it can be started in parallel and we can get a head-start. Thus for example assuming we were in the requirements’ phase, when a particular sub-activity of coding done, work done for coding accumulates. In this case, some design, coding, testing and integration work was also getting accomplished although we were in the requirements’ phase.

The basic requirements of the software process as well as the need to develop high quality software in the shortest span of time has made us more pragmatic and open to new ideas. The waterfall model helped to identify the major software process activities. The evolutionary model clarified that complex software systems need to be developed in an incremental way. The spiral model emphasized on the need to practice risk management, develop prototypes and proceed in an incremental way. The concurrent model tells us that all activities are under progress at all times, albeit with different intensities. The agile software process model takes this forward and says,

1. at all times during the life cycle, keep in mind the final product which does the job, the working programs,
2. as it is humanly not possible to define requirements accurately, the process model should be efficient at handling changes in requirements during the entire life cycle,
3. software is developed for customer’s requirements; customer satisfaction is most important,
4. for the success, the level of motivation of software developers is very important, and
5. communication, cooperation, teamwork, and not competition, should be the keywords.

“Talk is cheap. Show me the code.”

Linus Torvalds

The importance of programming cannot be over-emphasized. Stroustrup has observed, “… the details of the problem and the concepts of the solution often become clearly understood only through the effort to express them in a program and trying to get it run acceptably …” [3]. Programming is the way to ensure that definition of requirements, design, hardware and system software chosen are all tuned to the software to be developed. Programming brings out whether there are some requirements which are obscure and would require special algorithms and tools. If programming is not done, these difficult and unpleasant problems would be kept aside and these would eventually become a bottleneck for the project. Programming, likewise, checks the efficacy of design; whether the design is easy to implement or is it too cumbersome and whether the initial results point to the desired level of performance. The same thinking has led to development of prototypes at different project stages in all models, waterfall, evolutionary, spiral and, of course, concurrent.

Does the emphasis on programming mean return to the pre-software engineering days of 1960s, when code and fix was the norm? Definitely not. We, now, understand the importance of requirements, design, programming, testing, integration, system testing very well and there are clear methodologies for these activities. What we are saying is that these activities need to be done in parallel and that coding and testing need to be done almost right through the life cycle, first to clarify requirements, then to achieve the right design and later on keep the system in sync with new requirements.

Since requirements are not known precisely at the beginning and become clear only as we proceed on the project, we need to be geared up to accommodate changes in requirements. Each change in requirements, in fact, takes us closer to the “real requirements”. One of the reasons for the success of Unix has been the usage of the system in parallel with the development efforts and any error or feedback was promptly corrected or incorporated and system was quickly tried with modification. Thompson and Ritchie have observed, “… nearly from the start, the system was able to, and did maintain itself. … If the designers of a system are forced to use that system, they quickly become aware of its functional and superficial deficiencies and are strongly motivated to correct them before it is too late. Because all our source programs were always available on-line, we were willing to revise and re-write the system and its software when new ideas were invented, discovered, or suggested by others” [4]. Early correction of errors is very important. It can orient the work in the right direction and save a lot of effort which would otherwise be required in carrying on with the errors. The most serious errors are the requirement errors. It is important to be positive about changes in requirements. For, if we are not positive, then people will hesitate to point errors in requirements. Consequently, the system development would proceed with erroneous requirements in the wrong direction.

Here customer means the end-user who would be actually using the system on a day-to-day basis. The customer has the best knowledge of the system requirements although they might not be able to tell it to software developers in the beginning. But as the system development proceeds, their feedback is of paramount importance. Because of their “feel” of the end-use environment, they are able to give “key” feedback which can make the development effort successful. Early incorporation of customer feedback leads to further thinking on the part of customers and then more feedback. But all this feedback is worth its weight in gold. The agile software process stresses on customer collaboration and a quick incorporation of feedback leads to a system that is close to user needs. The feedback’s follow a bell shaped curve. Early in the life cycle there is less feedback. It suddenly picks up when the customer is able to see a tangible system. It reaches a maxima and then the rate of feedback starts falling down sharply. It is at this point that the system has come close to “real” requirements. By early coding and customer involvement, the agile process attempts to shift this curve towards west.

Figure 4: Change requests increase, reach a maximum and then decrease

The agile software process stresses on the human factors. Capable and committed people, with a high level of morale, make invaluable human resources. We need to value quality over quantity. For, a good quality 100 line code might do the same work as a poor-quality 1000 line code. A good design could drastically reduce the amount of total work required. For all this to happen, the motivation of developers has to be high. It is generally felt that people are not motivated because of high financial compensation but because of good working conditions. This might mean reasonable hours of work, empowerment, right training, hardware and other resources and a good organizational setup. Communication between team members, between the team and the customers and between the management and the team is all very important. An environment needs to be created where communication is informal and relaxed. It is a myth that “healthy” competition leads to great effort and good results. Rather than thinking of work as a 100 meters athletics competition, we need to think of it as a mountaineering expedition, where teamwork is most important, and where people collaborate to do the work together. Ritchie has attributed the success of Unix to the “calm, communicative and noncompetitive environment” at Bell Labs [5].

The guiding principle of agile software process is that the development teams are technically strong, cohesive, self-organizing, able set their own goals and meet them. An agile team periodically sits down and reflects on the work done and thinks about the ways of becoming more effective. It then goes about fine-tuning its methods so as to improve. A self organizing team is able to make good software architectures, efficient programs and tune these programs to “real” user requirements quickly and effectively.

Is the agile software process too idealistic to be successful in real life? To answer this question, we need to look at the various doubts that have been expressed of the agile process. The first skepticism was that if the requirements were continually changing, the software life cycle would go on for ever and would never finish. This had the financial implication that the project cost and time frame were a part of software contract and the software development organizations would lose money if the project was not “managed” well. The second skepticism was whether self-organizing software teams would deliver the results.

To answer these questions we need to get back to basics. The extent of success of any economic activity depends on the quality of the management. The best possible form of management is that of creative leadership, a sort of statesmanship in the economic world. The key to success is the teamwork between between the customer organization and the software development organization. Rather than thinking in terms of “them” and “us”, it is best to think as “us” only. If the project is successful, both the customer and the developer benefit; if it is not, both lose. A lot of projects are not a success and are close to a failure; these projects are a problem because of the drain they cause to the resources of both customer and developer. Some sort of management statesmanship and initiative at the working level are required to create this teamwork. The “ideal” teamwork might not happen, but we need to understand the ideal and the closer we are to the ideal, the more successful we are. Once there is teamwork and mutual trust, the requirements would “not keep changing” but converge towards “real requirements”. The bell shaped curve of customer feedback would get past the maxima and come downwards. The project might take a little longer, but would be better than a work away from user requirements or a project abandoned because of cost and time overruns. Ultimately, both the customer and developer organizations would have to rise above small financial gains and losses and look at the bigger gains of project success. For the customer, a successful project might mean more operational efficiency and the increasing returns from better efficiency would compensate for a slight increase in project cost by many times. For the developer organization, an agile process gives the best results and the project would in all probability an overall success, even if the costs increase a little. In an environment of mutual trust both the customer and developer organizations might discuss the increase in cost and come to a mutually beneficial solution.

That brings us to the second question. Do self-organizing teams deliver good products in the given time-frame so that the project is economically successful? There can be only one answer to this, the affirmative. The management challenge lies in creating and nurturing such teams. Of course, any commercial organization would have adequate checks and balances that these teams do not do something wrong. But it is necessary (for success) to create and empower such teams. The teams should have the power to set their goals. This is better than thrusting some goals on the team. If the goals are set by the team themselves, the chances of their meeting the goals is better. Once a team is able to set their goals, they should be empowered to get the resources needed for meeting the goals. When the goals and resources make economic sense, that is, the revenue earned by the team is more than the costs, we need not have any doubts about the economic viability of self-organizing teams.

We tried to trace the development of software process models. From the waterfall model to agile model, the principles of software process models have become clear for practice. If we follow the agile software process practices and the concurrent process model and create self-organizing strong development teams, we would be on the right track for software project success.