There is waterfall and there is Agile; and how the twain shall meet.

Update : 18 Apr 2020 -

I have been holding back on this write up for the last few days as I was not sure it was fully 'done' yet. This for an article on iterative development - the irony is not lost on me; hope you see it too.

On reading through the Waterfall paper in early 2012 on the recommendation of my team-mates, I had the chance to re-evaluate ideas of the sequential phased manner in which I developed at my earlier job. Illuminating as it was to find that the "waterfall" paper did not espouse a rigid one way progression of work in phases, it also sat right with the working experience we had developing within the "waterfall" methodology. Of course I could not find the original article but did find another blog article which elucidate the same thoughts in some detail (see here). As a team we had been practicing SCRUM Agile since 2008.

While the phases left us to develop on features in a 3 month cycle, we were routinely interrupted to fix the issues found in testing phase the development team's "miss" on the "Analysis and Design" documentation handed-over to development. This was usually a situation where one needed to choose their battles mindfully. With the Manager under pressure to deliver the features in prescribed time and the BA and Architect unwilling to accept that there are gaps in the requirements, the only time development could push back on the work coming out of testing -

1. Is it a large enough change to impact the current development on your plate?
2. Is it truly a requirement specification gap, that can be easily established? This would more be a matter of principle. Such gaps are created in specs as the Tech spec writer is not as close to the code base as a developer. (Some food for thought - why do you think following LEAN and AGILE methodologies easily avoids this problem? If you don't have a ready answer - read on.)
3. Will the code change for the minor issue require a relatively large change and another full manual regression? Then this could be negotiated with the BA and the test team :)

It is worth pointing out that all 3 issues can be addressed by a proper CI/CD pipeline.

Notwithstanding these situations, whether responsible or not, development was bound to fix the issues found in testing as by default, any issues found were deemed developers' mistakes or misses.

As such each estimate that we made on the development cycle work, kept enough slack for addressing issues coming out of our earlier development phase. Although the methodology had no room for the inclusion of work outside of current phase, the practical implementation of the work required us to do the exact opposite. From experience, it had always been clear to development and the project managers as well, that development and testing required close working and a proper feedback loop.

Iterative waterfall process used in most development efforts

A confusion in understanding of the difference between a framework and methodology is at the core of the conflict that arises between traditional Project Management practice that has evolved following Waterfall SDLC methodology; but now attempts to fit its operation in the Agile philosophy inspired development frameworks like SCRUM, DAD, SAFE Agile, Extreme Programming to name a famous few.

Agile Crystal Framework

Agile DAD Framework

Agile Extreme Programming Framework

Agile SCRUM Framework

Having silo-ed teams to work on each phase of development is an exact sign of waterfall methodology applied to product development - here each feature (or set of clubbed features) is bound as a project and executed as per the specific waterfall SDLC phases in use.

A close look at the AGILE inspired methodologies provides us the fundamental difference in approach which is the source of all conflict. While an honest implementation of Waterfall methodology is never strictly uni-directional, it still has clearly defined phases, and the development effort is identified as being in a specific phase until an agreed upon date per plan. This also leads to an obsession of tracking dates (and other KPIs) and observing a project in risk only if it does not slips on specific milestone and measures. The trouble is that time slippage is a poor indicator of project / feature health, and also appears with such latency that there is precious little recourse available to the team for corrective action.

In an iterative and incremental methodology the approach is to provide publicly shared "information radiators" that serve as immediate feedback for small corrective actions without larger impact. This also bubbles up the missed or underestimated risks during initial planning, design. (Coming soon - Agile information radiators vs waterfall progress indicators).

Clearly (as I have also experienced), any PM following a sequential model of development like Waterfall is lead to making error in judgement based on his traditional monitoring signal. They either fail to act in time believing the signal to be of no consequence this early in the phase; or veer to the other extreme of having an alarmist reaction - followed by over correction. The whole scenarios plays out repeatedly with variations for different phases.

On repeated such occurrence readers may have witnessed the resulting exasperation of the PM and development teams leading to the PM only asking for end dates, thus relegating himself to only the observer of the project progress (further relying on the same unreliable progress indicators); the development team treats other important project activities outside of their silo and loses out on understanding and getting feedback from the larger impact of their deliverable to the product outcomes. (Coming soon - Real life scenarios of detrimental effect on job satisfaction due to lack of visibility on value add of testing work)

WBS, Product Backlog and the PM's quandary -
There are important differences that also can mean the difference between a good project management and faster response.

Product Backlog -
Fundamentally the story estimate is essentially only that - a best effort estimate based on expert opinion and current level of information - expectantly corrected for known bias. It is tracked often and corrected from immediate feedback. The story needs to be identified fully, with all blockers removed. The backlog at any given time of the project is not fully groomed nor are all tasks / stories fully updated - in fact that would be a big red anti-pattern flag for any agile framework in use. Hope fully I will be able to speak more in another post, at present we run the risk of veering off course.

WBS -
The WBS is actually expected to be fully identified in sum of its individual tasks. All tasks are to be defined fully (this is usually a very non-pragmatic approach for large complex systems) which is difficult for any one team or set of people to define. The unknown - unknowns are always hiding in plain sight :).Every task is broken down into a granularity that can result into actionable work once design phase is completed.

In this context the waterfall PM can be misdirected in his/her belief that each story iteration in the sprint is actually a fixed commitment to complete the tasks in it. If that is missed, it implies to his mind a slippage in the commitment of the deliverable. The fact that a story may be deemed incomplete despite substantial work being completed is alien to his frame of project monitoring and control. The story may have a newly identified dependency, or a requirement gap found right after testing, or simply some other clause of the agreed upon "Definition of Done" or "Acceptance Criteria" not being met. An incomplete story moving to the next sprint is a cause for concern and a directly affects progress indicators like the traditional SCRUM burn down / burn-up charts. Nevertheless, it is an input to the overall progress parameters but can be incorrectly seen as cause for immediate concern. Instead a true slippage will be seen once the team has feedback on their velocity of multiple sprints that tracks completion to a date later than earlier planned.

Harnessing an important Waterfall methodology tracking tool in Agile inspired methodologies -
In looking to move to a backlog tracking method instead of WBS, will provide us with a better control and feedback loop to drive the project. In this we stand to lose a view of the dependencies that are typically visible to us on a Gantt chart. I have found Gantt and fishbone to be excellent tools to understand inherent work dependencies and avoid pitfalls of stalled / bottle-necked work early in the development iterations. While we can have dependencies identified, having a Gantt chart that demonstrates critical dependencies also highlights the important assumptions based on which the team moves forward as well as helps identify critical path. This is also of relevance since external dependencies may be time bound and may put project MVP timelines beyond acceptable limits of cost or time. Generally Gantt charts are plotted in a timeline, but we can still benefit from Gantt with plotting tasks / stories along relative timeline.

A good reference is listed here.

Traditional Waterfall phased approach also pitted testing and development as enemies and lead to an unhealthy relationship very detrimental to the product feature throughput - more on this later.
When applying waterfall performance standards to a team that runs by Agile methodology an often seen example is the exchange between the development and test automation teams. If the test or automation teams believe their contribution to be measured only based the errors found, they are drive to either find trivial errors or only test in isolation to the development team. There is no incentive to actually collaborate with the developers to find scenarios so that test team add to the regression suite while developers program defensively. Although the best case for the overall progress, the test team's contribution in such cases remains hidden, un-appreciated and un-monitored (for case coverage) until it rears its ugly head as production issues. TL;DR team fails because it use the existing standards of progress monitoring which leads to participants working to KPIs instead of results.

Better computation resource, Quicker quality feedback turns the value chain on its head -
CI / CD coupled with the lowered cost of computation allows us to do trials on work done and failures in less costly. Over time developers realized that trying out an idea was the fastest way to hack a solution rather than conjecturing based on limited data / information. The idea of fail-fast could take root as cost of computation resources dropped precipitously - the Waterfall model that actually formalized development process to frugalize on computation resource was not of relevance.

In fact over a period what we have witnessed is the complete commodification of computation resource (mainframes to data centers, then cloud computing), and being able to run development code more freely now transforms the developer's skill as the truly critical resource - thus a potential bottleneck. Perused with this viewpoint it is clear that most new software development approaches address the efficiency of development resource and re usability of developer output rather than computation resources. The only exception to this would be the exception of approaches devised for mission critical or real time systems - these too have seen some movement with available simulations). In fact, the penetration and commodification of data intensive and complex computation projects involving machine learning, data mining, scaled statistical modelling are all a result of reduced computation cost per unit. Such projects were neither feasible earlier and nor could they be delivered with a waterfall approach successfully.

With this is context, it is no leap of imagination to be able to visualize the exact development practices we see in vogue today. Automation of test and deployment code, ability to run regression test repeatedly, large volumes of data used to perform aggregations and information extraction previously not possible without prohibitively expensive and infeasible computation.

The rigor of WATERFALL is needed to be inculcated in the AGILE framework inspired methodologies. Some of the ideas still have relevance and have been included in various form as Agile teams mature -

1. The Paradox of Plenty is a very real risk that is necessary to guard against in modern
methodologies. Teams inculcate good habits in harnessing and leveraging the development resource while taking for granted highly available computation resources that hide other performance inefficiencies like -

less than optimal time -space complexity of core algorithms.
lack of effort at code optimization
de-prioritized technical debt, and its long term costs . eg : fueling poor development practices
poorly designed code that scales poorly
strong platform or environment dependencies as more services become commodified
absence of fault tolerance at a higher layers.

All these lead to brittle code that will work well in the usual environments but will not degrade gracefully. The rigors of keeping track of technical debt and setting aside resource to address it are critical to long term success of the product developed using Agile methodologies.

2. POC (Spikes in SCRUM) for feasibility , Analysis and Designing -

Waterfall methodology formalizes the thought process and helps teams and senior members commit to the idea of providing at least a reference point for architecture. design and risk assessment. The only catch has been that it attempts to capture the unknown complexities and demand variations before they are even evident; as such it attempt to build a solution schematic without being able to fully identify the problem space.
Without over-committing and crystallizing the outcomes, having these exercises to build a reference architecture, design, POC is valuable to future projections. It also supports improved estimations during development cycles. As such, as Agile frameworks are malleable to specific team needs, I have found incorporating these in early sprints of a project to be beneficial. A good yet informative example to start your discovery on what works for your team can be found here.

SAFE Agile as a framework for scaling agile approach across an organization does prescribe it more formally as Agile Architecture. In fact referring the SAFE Agile site periodically would help cultivate a mindset to help everyday decisions.

Although in theory this idea sounds great that a complex design will also be grown iteratively, in practice this has potential for great risk. In particular we can look at examples of complex systems where expert knowledge is required. EG: a search solution may choose to take a ready made solution as it allows for rapid turnaround on the first few requirements; eventually it is not able to support more involved requirements but the plumbing code around it has been developed which will need the team to redo avoidable work. With an eye on the product road map, evolving requirements, technical community support, a proper design and architecture would be required upfront to mitigate such risks. Two important factors to consider here are -

The development team does not have the technical experience or the long term vision on how to evolve the product.
The architects and the senior management are better suited to see the pitfalls and seed the initial architecture and design based on their analysis so as to protect the team from wasting time in trying solutions that wont work. Of course providing visibility and documenting for the team the decisions taken is a critical responsibility of the architects and managers. This is essential to the practice of ANY agile development methodology.

3. Road map prioritization and resource feasibility based on team(s) velocity.

"Agile road maps are built to reflect product strategy but also to respond to changes, such as shifts in the competitive landscape, value propositions, and engineering constraints" (ref). This is the primary difference in thought that requires a periodic revaluation of the road map in an agile approach.

This is not in keeping with the earlier road map and strategy documents brought out annually by the senior members of the team. The product road maps and strategy were something teams visited after they were created by the product team once a year.

Although the product backlog 'funnel' as a visualization is something that I came across only recently, I could not agree any more than I already do that the product backlog should be groomed following the typical 80-20 rule and that stories be fleshed and prioritized to meet the exact road map requirement of the near future. In fact the backlog should be ordered largely based on near term, mid term and long term, with each being addressed differently. For a more formalized understanding of backlog grooming, the reader could look at "WSJF - Weighted Shortest Job First" used in SAFE Agile, or the Kano Model, MoSCoW Model, among other popular useful tools.

Agile Backlog prioritization "Funnel"

This is actually an excellent place to start the discussion around the approach to development with the PM. The WBS dictates deep dives and clear definition of the tasks in the project. Whereas the the backlog will have a lot of "stories" related to the project not full detailed when the work kicks off. this does not leave the PM in a comfortable position; and I suspect you will have to make no effort to address this dichotomy - the PM will come to you instead. In these scenarios it is imperative to remember that you are expected to "respond, not react". This is a golden opportunity to start a conversation, and see it to the "notional" end of the project - milestone by milestone.

4. Documentation in terms of business requirements and Tech specs - living documents!!

It is critical to remember that if your stakeholders and PM are waterfall oriented they derive clarity on the project based on the document delivered out of the analysis and design phases. They understand the true nature and scope of the work "to be" undertaken from the details capture in the document. In all likelihood they also mistake the documentation coming out of agile analysis and design to be the final version. As such, you run the risk of them deeming the contents too light on detail to provide a green for execution. They may also hold on to every goal in the initial documentation as being the team's commitment.

This has been the experience of many a development teams including mine. Despite stakeholders being aware and supportive of agile methodologies, it not a single day transformation from a mindset that has built over years of executing projects a certain way. This is particularly aggravating for Development teams where the core business is not Software and the stakeholders need reeducation for a proper understanding of software development as a trade. It is the responsibility of the Engineering manager and also in their best interest to introduce early and repeatedly the idea of a living document that is versioned and updated as per changing situations. Having highlighted the need for flexibility in scope of deliverable, the Engineering manager will also need to identify an MVP that can be delivered to production to help business feel secure, having some even footing.

In the longer run, business and PM will come to appreciate the quality of documentation that comes out of such exercise and how it organically facilitates documentation of all decisions that can be revisited to understand the thought process and conditions around it.

I would like to highlight that no methodology prescribes any set of documentation but does insist on sufficient documentation. The only caveat on documentation is that it does not become the sole vehicle for information transfer. This is embedded in one of the 4 values of Agile philosophy (thus flows as a principle to any specific methodology implementation) : "individuals and interactions over processes and tools". If you think it should be "working software over comprehensive documentation" instead - think again. :)

Here is quick read for you to get started on understanding the difference in approach to documentation between the two schools of software development methodologies.

Signs that the PM is thinking in Waterfall SDLC when dealing with the agile development team

1. Constantly asks for delivery dates.
2. You get to hear about the commitments made by team and how it is missed.
3. micromanages each task and gets alarmed at each task slippage.
4. Misuses sprint retrospectives to blame individuals for delays or rework
5. Daily stand-up ceremony is used to publicly question technical decisions or new unknowns not captured during planning.
6. Wants to add new stories to tasks in the current sprint. this is a common mistake and seen as being inflexible despite talking about agility.
7. Asks for a percentage done on a feature - this is an innocent question but should not be answered. As an exercise for yourself, think about why it should not - please share your thoughts in the comments.
8. Questions the meaning and utility of Tech Debt and related changes as they add no value to product features.

A cautionary note to alliteratively re-check our own bias through the process as leaders -

After so many years of practicing Agile development cycles, I still have to be mindful in occasionally self-evaluating my thought process.

I write this here to demonstrate to Agile Champions, commonly found mental conditioning; so that they may take advance steps to assuage such fears -

1. The need for perfection in my sub-conscious

2. Fear of criticism - afraid that it may turn out be be valid and logical.

3. Not recognizing that I was taking on too much together. Future iteration with reader inputs will help improve the article in way I could never.

4. Attaching value to only the final product - an imaginary 'ideal' that no one can validate since IT DOES NOT EXIST!

5. Delay in putting out the article one it met the MVP requirement. I believe, it has reasonably capture my thoughts on ground issues faced by teams when moving to Agile.

Search This Blog

Stray Musings on the intersection of Technology, Domains, People & Process