Agile software development is a powerful solution to the perennial problems of unpredictable software quality, customer dissatisfaction, and compromised projects. This paper is a brief introduction to the structure and the philosophy of an agile software development process.

The Structure of Agile Process

The first activities in an agile process identify the business case for the project, and identify the major requirements and major risks on the project, as well as identify the product’s major features, i.e., those features that drive and determine the product architecture. Only a small percentage of features actually constrain the product architecture; most features are “followers” and can be readily implemented in whatever architectural metaphor we chose. One of the important goals of these early activities is to obtain a vision of the product, a “big picture” if you will, but the picture is more broad and shallow than narrow and deep ― more like a Road Atlas showing the entire United States rather than a photograph of a 200 foot section of U.S. Interstate 40, showing every pothole and billboard.

Assume we have 25 features to incorporate into the product we are building. Rather than trying to get a full understanding of all 25 features, gathering all of the requirements, and defining all of the design at the beginning of the project ― we partition these features across the planned duration of the project. Agile development is invariably based on an iterative process structure. In an iterative process, we use a divide and conquer approach to solving a large problem: we take the total project and divide it into smaller pieces. These partitions are called iterations, and an iteration is really a sub-project within the project. With our broad and shallow project vision in hand, we now start to develop the narrow and deep understanding of the work ahead of us.

Each iteration has a defined goal: to deliver working software that implements one or more of the features we have chosen to develop in that iteration. It is also possible that an iteration may only deliver part of a feature: e.g., an implementation of the feature in which no exceptions or errors occur, or only a credit card payment method is supported. In another iteration we might implement the exception and error handling, or additional payment methods.

Let me contrast this approach with the traditional, waterfall development process. In the waterfall process we deliver the whole product at the end of the project lifecycle. The reason we do this is that the waterfall process says that development must support the customer’s perspective of the product, which is assumed to be thus: only the full, complete product has value for the customer. That is an arguable assumption that I will not pursue here.

But what if we approach development from a different perspective: the development perspective. What has value for the developer, or development organization acting as the vendor to this customer? The value for the developer is knowing they are always developing the right product, and you cannot obtain this value of validation if you wait 3, 6 or 12 months before you deliver the first working examples of the product. From the development perspective we want to deliver real, working product software early and often, so we can get immediate feedback on whether we are still on the path to hit the target our customer has given us.

Here is something very interesting about these contrasting approaches. In the waterfall approach we adopt the customer’s perspective and place a burden of credulity on the development organization: we can only believe that we are producing what the customer wants, and we support our belief with lots of documentation, and mega-plans. But only by delivering the demonstrable, working software can we transform our belief into objective knowledge that we actually have met the customer’s desires.

In contrast, in an agile approach, we adopt the developer’s perspective that requires proof that we are meeting the customer’s needs, and we present that proof as demonstrable, working software that can be validated by that very customer or an authorized representative. Not only does the developer get continuous, rapid feedback from the customer, but the customer sees real, working software that either does, or does not, satisfy his requirements. If the software is valid, the customer sees real results for his money. If the software is not quite correct, changes can be made quickly because the scope of the change is usually limited to features in the current iteration. So deviations from what is expected are inherently limited and minimal. And either way, confidence and trust is established between the customer and development organization because the customer sees working code, not just promises in large piles of documentation.

Figure 1 below illustrates what we have been describing. On the top half we have a typical division of activities based on the waterfall process. At the end of our 6-month project we deliver ― for the first time ― the product software to the customer for acceptance testing or production installation.

In the bottom half of Figure 1 we have partitioned our 25 features across 6 iterations of about 4 weeks each. Each iteration is a small sub-project in which the tasks of (R)equirements gathering, (A)nalysis, (D)esign, (I)mplementation and Test are performed. At the end of each iteration real, working software is delivered to the customer for her to validate that our implementation conforms to her expectations. Each delivery contains only a few of the total features, but each delivery adds to the previous deliveries. Therefore, at the end of iteration 6, which is the end of the project, we deliver only 4 new features, but these are in addition to the 21 features we delivered in the previous iterations.

In summary, the structural differences in the iterative approach are:

1.   Partition the project into discrete, sequential sub-projects called iterations.

2.   Conduct each iteration as itself a project, with its own requirements, analysis and design challenges, still constrained by the goals of the overall project.

3.   Incrementally add features to the working code base, and incrementally test all current and previous features.

4.   Deliver working software at the end of each iteration. Deliver to either an internal group for validation, or externally for the customer to validate.

5.   Re-evaluate the entire project at the end of each iteration, making any necessary adjustments based on new learnings from the current iteration.

The Philosophy of Agile Process

A recurring question is: Does this mean iterative development is the same as agile development? No, the distinction that defines agile development processes is not structural, it is philosophical and psychological. Whether we are developing software by a waterfall process or an agile iterative process, what we do does not differ significantly. But how we do what we do is quite different.

In Table 1 below I have listed 4 psychological properties that are present in every software project. In the two right-hand columns I have described how these properties are addressed between the traditional waterfall approach, and an agile approach.

Table 1: Process Properties

Uncertainty & Risk

All projects must address uncertainty and risk. At Evanetics we see these as the fulcrum for success on any project. Risk is always present and cannot be eliminated, but must be actively managed and it can be minimized. Uncertainty is inherently transient. Uncertainty implies questions that do not have answers ― yet.  

In the waterfall approach it is common to see efforts to eliminate risk and uncertainty by defining it away: make decisions early and get, or force, answers early, often earlier than justified by the maturity of the project. These early decisions are documented in early artifacts such as Functional Specifications, and then “frozen”. Because the planning is up-front, changes and accommodations are resisted. This approach implicitly embraces the presumption that most, if not all, obstacles and risks can be identified and solved early in the project.

Agile process recognizes that this attempt to get early, final answers is itself a major risk on projects. The agile approach accepts and accommodates uncertainty when it is not an obstacle to progress. That is, you don’t have to get the answers until you actually need the answers. Deferring the decision activity allows the project team to learn over time, and to bring their best, current understanding to the risks and uncertainty facing them. Similarly, an agile process does not “freeze” understanding. Rather, agile processes expect projects to evolve, and strive to provide a framework for adaptation to change rather than fighting against change. On an agile project, an absence of change in project documentation or understanding is a sign of pathology, not health.

Fear of Failure

The fear factor motivates us to construct barriers to prevent forces from impeding our success. Fear leads us to institute control structures of various complexity, and often a high-level of ceremony and artifacts to assure that we are in control in a world of unpredictable forces. In organizations with strong separation among functional groups, fear leads to further isolation and creates an environment where business groups do their narrow job and then throw their deliverables over the wall to the next, downstream group. Seldom are the complex control  structures, or the “silo” organizational structure recognized as potential, or actual, risk factors on the project.

In the agile world we recognize fear of failure as a negative perspective on a very positive property: desire to succeed. Since all projects differ in their staffing, goals, deliverables and scope, in an agile process control structures are instituted at a project level as they are needed, not as a result of corporate edict. Within a three to four week iteration only very short-term control mechanisms are needed, as well as minimal ceremony and artifacts. The agile process needs so little overhead because iterations are short and offer rapid feedback to the stakeholders, and the single goal of the agile process is to prove it with code.  In an agile process the tasks assigned in an iteration are only those tasks needed to produce the code for that iteration. Similarly, there is no “throw it over the wall” in an agile project. There is one “Team”, populated with the developers, the business experts, the Quality Assurance members, the Project Manager, and customer representatives. One team. One focus. They are all inside the wall.

Predictability

Predictability is a critical business requirement. One cannot run a business without some level of commitment to time of delivery, quality of product, number and type of resources needed, etc. But the scope of predictability can be very much in the eye of the beholder, or the process being pursued.

In a waterfall project much emphasis is on up-front definition. It is quite common in these projects to see the production of large Gantt charts from a software tool. These Gantt charts often describe tasks and personnel assignments months, or years, into the future. Having a large, pretty printout is seductive: one can think that since the content is so precise, that it is also accurate. One can also fall into thinking that the predictions made at the beginning of a project should still be true in the middle, or end of a project. This is a dangerous trap.

In an agile project we plan broad and shallow over the life of the project, but we plan narrow and deep only in the current and next iteration. We know that accuracy is more important than precision, and we seek precision only for the tasks “in our headlights”. In an agile project predictability is dynamic. This means that at any moment we will have an estimate that is our most accurate as of that moment…but it will may change tomorrow or after the next iteration.

This dynamic aspect is disturbing to people accustomed to up-front, static predictions. But the agile approach is based on accumulated learnings on the project: if my estimate 3 months into a project is different from the one made at the beginning of the project, which can we be confident is the more accurate?

Process Variance

In non-agile projects it is common to detect a view that the team must stick to “The Plan”. This is an expected psychological posture when we acknowledge the intense up-front effort invested in coming up with The Plan. When intellectual capital is invested in construction, resistance to changing ― or destroying ― what we previously built is inevitable.  This is the same psychological reason why programmers who do not follow a Test-Driven Development approach cannot effectively unit test their own code.

Agile projects adapt the development plan many times as necessary to accommodate external forces. The agile philosophy recognizes that the planning is much more important than The Plan. In an agile process we measure the actual output of the team in each iteration, and we use those actual measures to evaluate the remainder of our project plan. This is enabled by the delivery of working software in each iteration.

One of the best analogies I can offer in this area comes from Kent Beck’s Extreme Programming Explained. Beck presents the metaphor of driving a vehicle to illustrate the process variance in an agile project. Experienced drivers do not blindly drive in a straight line: they make continual, small adjustments to their steering direction, velocity and acceleration based on their current driving context (pouring rain and dark), not on the context that was true (sunny skies) when the driver originally started the trip. To not accommodate these variations based on current learnings is analogous to “launching” a vehicle, not driving it.

Which brings us to the essence of the agile mindset, which I will express as two epithets:

• if you need to do something to achieve your goal, do it; otherwise don’t do it.

• if you need to change something to achieve your goal, change it; otherwise don’t.

But what does it mean “to achieve your goal”? Here the agile process is very clear: your goal is always to deliver the proper working software that meets your customer’s requirements. If your process is directing you to do anything that does not contribute to delivering working software, you are not doing an agile process.

Conclusion

This paper has narrowly focused on the essentials of the structure and philosophy of the agile development process and how it differs from the traditional waterfall process. Agile process is inherently pragmatic, disciplined and adaptive. But becoming an agile organization requires more than producing a project schedule that has features delivered every four weeks. Evanetics wants to be your experienced partner in moving your organization to the productivity of iterative, agile development. Contact us so we can discuss your goals.