Does Agile Scale?

Ever since the Agile Manifesto was penned twenty years ago, the agile practices it inspired have slowly taken over the world. You will be hard-pressed nowadays to find a software development organisation that would not profess to follow some form of agile approach. According to the latest State of Agile Report, 94% of software organisations across a broad range of industries claim to do so.

The word ‘agile’ itself is now a core part of the modern management lexicon, forming part of MBA curricula and endorsed by the bible of modern management, the Harvard Business Review, in a 2016 article entitled “Embracing Agile.” The digital and transformation divisions of companies large and small all got on the act, cheered on by armies of consultants, and slowly but steadily, all undertook some form of agile transformation. You will thus find agile programmes in organisations as diverse as General Electric, Walmart, the UK Home Office, BP and Barclays Bank.

However, while agile slowly ate the world, something strange happened. As agile spread beyond the software team, many of its practitioners became concerned that many companies were no longer practising ‘true’ agile. Derogatory terms such as “wagile”, short for “waterfall agile” and “Fake Agile” began to be used by agile evangelists, enthusiasts and practitioners concerned about how agile principles were being diluted as they were adopted by an increasingly diverse range of teams and companies.

**Not agile practitioners with different viewpoints, but 16c France wars of religion**

Many of these tensions arose from the fact that ‘Agile’ had strayed far from its home territory of the individual software development team. As an example, the first principle of the Agile Manifesto, the founding charter of the agile movement states that “Individuals and Interactions should be valued over Processes and Tools.” It stands to reason that you’d value a person more than a tool, if for no other reason than common decency. But what does this principle mean in the context of an organisation of 1,000s of employees?

This raised an even more fundamental question. Does Agile even scale at all beyond the individual software development team?

The question. Does “Agile” even scale at all?

Let’s consider the crux of the problem.

Agile (with a capital ‘A’) was developed as a means to allow a single team of software developers to work cohesively, efficiently and creatively together to solve problems that were difficult to predict and to plan. From a business, or enterprise perspective, being agile (small ‘a’) is all about being highly responsive and adaptable to a changing business environment. This could include anticipating or reacting to changes in customer needs, attitudes, social norms or technology-driven disruption.

So we have two different definitions of agile – one in a team context, and another that is enterprise-wide. And these two contexts operate to very different principles, trying to achieve different aims. Team-level optimisation, efficiency and innovation on one side and business success on the other side. So when considering the question as to whether agile scales at all, there are two problems that need to be solved. First, how to scale the software development effort beyond a single team to potentially hundreds of teams, and secondly on how to align an ‘agile at scale’ technology organisation with a wider agile business.

1. Coordination – Like herding cats?

When looking to scale a development organisation, you need to consider two dimensions. The first dimension, rather obviously, is to increase the number of teams, possibly to hundreds of them. The more teams you have, the more complex the challenge of coordination is. As agile is built on self-organising teams, how do you coordinate tens or hundreds of self-teams without top-down control or coordination?

**The principal two dimensions of agile scaling**

Irrespective of what model you follow, unless your business can be delivered by fully autonomous and independent teams, each determining their own direction, priorities and technical strategy, you will need to find some means of providing coordination. This is one of the main aims of what many agile scaling frameworks, such as the Scaled Agile Framework (SAFe), Large Scale Scrum (LeSS) and Scrum@Scale try to achieve. For full disclosure, my personal experience in scaling agile has been with SAFe. If on learning this, you are tempted to stop reading, I invite you to persist a bit further with me.

Harnessing the joint effort of multiple teams often requires various forms of coordinating efforts, including:

Product Backlog. Ensuring that there are means for the teams’ individual product backlogs come together to meet bigger or complex needs
Architecture. Where different components and services need to work together to solve a complex problem, pulling together a common design or plan. This may take the form of an overall architecture or system design.
Portfolio. Where the teams need meet different and often competing business needs, then some means of prioritising these activities are required.
Delivery. If a business output is only achieved through the effort of more than one team over a number of iterations or sprints, then it is likely that the business will expect some form of delivery or progress reporting to give confidence that the teams activities are on track and that they are provided with any support they may require.

The most common criticism by many agile advocates against these centralising activities is that all these coordinating mechanisms can stifle agility and innovation. This is something I rather agree with, and great care should be taken to put in place the ‘minimal viable’ amount of coordinating overlay so as to maintain team-led innovation and problem-solving.

2. Longer Horizon Planning

The second dimension is the time axis. Many software teams do not belong to pure digital businesses. They exist in a context that involves physical goods, operations or people – think of a consumer product company, a travel company, an insurer, a bank, an online delivery company. All of these will need to meet fixed deadlines that are multiple sprints away, often beyond the planning horizon of an individual team, and often with fixed requirements, as for example when needing to meet regulatory and legal obligations.

While it is well and good to release software on demand, or at least with every sprint, any software development team in such companies will need to meet external deadlines. A toy company cannot miss its Christmas deadlines, a TV company must meet the broadcast date for its hit show. For complex projects, such deadlines may be many months, or even further, away. Where such deadlines exist, product development leaders are often required to provide confidence that a shippable product can be released with a given set of features by a given date multiple iterations away. This is often driven by non-digital go-to-market obligations, such as getting catalogues printed, making commitments to customers or ensuring regulatory sign-off.

SAFe makes use of bottom-up team-based planning over multiple sprints, a period called a Programme Increment, which in most implementations takes place over a three months’ period. SAFe proposes that the teams make commitments over the current Programme Increment and predictions or forecasts over the following 2 or 3 quarters, thereby providing business leaders with a view of what can be possibly achieved and committed to over a 1-2 year horizon. While this is just how SAFe addresses this topic, most scaling approaches propose some means for planning over significantly longer intervals than individual iterations or sprints.

Figure 3. SAFe Planning Horizons — ***Look away now – SAFe’s answer to long-horizon release planning***

3. Optimise the System, not the Team!

Team-level agile seeks to optimise the efficiency and optimisation of the team, by maximising the rate of innovation and development. However, where multiple teams are required to produce an output, Systems Theory dictates that by optimising the individual parts of the system (i.e. the teams), you will inevitably end up with a sub-optimal system.

I explored the principles and applications of Systems Thinking in a previous post. The key consideration is to understand the interdependencies, not just at the outset, when creating a plan, but also in terms of the dynamic positive and negative feedback loops inherent to most complex systems. This requires a holistic view of the intended design, which if spans multiple teams, cannot simply be done at the individual team level. Crucially, the aim is to optimise the experience and performance of the whole, even if it means slowing down or creating inefficiencies at the team level.

The design does not necessarily need to be done ‘top-down’ by an architect. Representatives of different teams can self-organise through design workshops or other forms of collaborative design to come up with proper systems-led designs. Indeed, the unpredictability of inherently complex systems (think autonomous driving systems in cars) are best solved collaboratively between teams, rather than to the design of a single architect or technical authority.

***Not Systems Thinking – The Parable of the Blind Men and the Elephant***

4. Organisation – You are not Spotify

One of the issues faced by many companies when establishing agile development of scale is how to organise the teams. One model that is gaining traction is the so-called Spotify model. Well publicised in a video on the Spotify Engineering Culture by Henrik Kniberg, the agile scaling approach adopted by Spotify has been quoted by many agile consultancies as a model that can be applied to many other companies. One characteristic of the approach is dividing a development organisation along product-oriented teams (or squads) grouped together into tribes. The was effectively a traditional matrix cloaked in goofiness.

**Most definitely not a Scaling Framework – Something from Spotify**

After all, if it works for Spotify, why not for your company? First of all, by all intents, this is not the way Spotify work today. For a more detailed, and somewhat opinionated assessment, read here.

But crucially, Spotify did not choose their way of working in isolation. They evolved an approach to scaling that not only matched their culture of empowerment and trust, but was also compatible with their software architecture. Spotify was born as a digital company in Sweden in 2006. Its product is streaming digital music, which is delivered globally through software. So the “Spotify Model” was a way how Spotify applied agile principles to meet a given business need in a given stage of their development of the company. It certainly wasn’t intended to become a template to be adopted far and wide by companies large and small. Simply relabeling your existing product structures into squads, tribes and chapters is not only lazy, but it will also be unlikely to work.

5. Portfolio Management

The final consideration when seeking to harness the effort of a large development organisation is the question of “what” to do and “where” to invest. I cannot even think of scratching the surface of this topic in a couple of paragraphs. This is the realm of business strategy, product roadmaps, investment planning, budgeting, and resource planning, all grouped together into some form of portfolio management discipline.

Portfolio management is often the most overlooked part of many agile transformations as it is often conflated with visions of Gantt Chart-wielding project managers and large programme offices. However, the task of ensuring that the resources invested by a company are put to good use and deliver the intended business benefits do not go away simply because you’ve ‘gone agile’. Indeed for any organisation to truly consider itself agile requires very high levels of transparency and visibility. For how can you steer course and change direction if you don’t even know where you are?

The approach that has gained most traction recently is that of Lean Portfolio Management, a discipline aimed at applying lean management principles to link business strategy with execution. There are several different flavours as to how this (see here for a good summary of the key principles), but all have the following characteristics in common:

Transparency – Make work and initiatives as visible as possible, creating as much transparency as possible
Business Value – Prioritise work ruthlessly on business value
Maximise Throughput – Reduce the work in progress in order to maximise throughput
Maintain Agility – Embrace the fact that things change, and adapt according to what works and what doesn’t

Some tips on easing scaling pains

So given all this, how can one avoid all the pitfalls in making an organisation more agile?

1. Be clear about the expected business benefits

First, unless you simply want to slavishly follow the latest management fad, ‘becoming agile’ is not a means in itself. Determine what your business objectives are and whether they would benefit from greater business agility. By example, you may define your objectives as making better use of online and mobile channels as part of a digital transformation, reducing your time to market, reducing risk by improving business agility or simply improving the efficiency of your development organisations. The starting point for any agile adoption is being crystal clear on these objectives. This is the only way you’ll be able to determine if you’ve been successful or not.

2. Determine your ‘agile’ success metrics. How agile do you want to be?

Once you’ve settled on your business goals, it is necessary to figure out whether adopting agile practices in your technology or IT departments will indeed help you meet those goals. In doing so, it is worthwhile establishing some hypotheses. For example, if your main goal is to improve your time-to-market, this may be based on a hypothesis that this will ensure an improved competitive position or reduced development costs.

In this example it is essential to put in place metrics that measure characteristics such as velocity, development predictability, rate of business value generation, release frequency etc. This will allow you to determine the effectiveness of your agile operation. This on its own does not guarantee success. This will depend on whether your hypotheses on the impact ‘going agile’ will have on your business was indeed correct.

The key point is that at the outset of an agile journey you have two unknowns. The first is whether you will be at all able to improve your development effectiveness at all. The second is that even if you do, will it have any meaningful impact on your business performance? In other words, does agility matter in your context.

3. Be honest as to the level of team empowerment you are comfortable with

As mentioned above, one of the principal sources of tension in any agile transformation is that between the centralising and control bias of most leaders and the decentralised, self-empowered ethos that underpins agile. Depending on the nature of your business, you may wish to confer different extents of ownership and accountability to your agile teams on areas such as your feature roadmap, your architectural design, your go-to-market strategy, budget ownership etc. There are often good and valid reasons why certain decisions need to be taken away from the agile teams. Being clear and honest about these can help reduce tensions within your teams. The levels of decentralisation that are appropriate for a Software-as-a-Service company built on scalable and decoupled cloud technology will be very different to achieve in a safety-critical, highly co-dependent aerospace company.

4. Understand the limits to ‘agile practices’

It is important to be crystal clear on what is meant by ‘Agile.’ In this context, I am referring to teams working together, iteratively to some recognised agile methodology, such as Scrum. The reality is that while you can apply many agile principles to other functions, such as finance, legal and compliance, customer service, procurement, logistics etc, their operational reality is that you will be unlikely to apply Scrum or any other recognised agile approach. Rather, for those parts of the business where predictability, efficiency and repeatability are sought after, then approaches such as Six Sigma are probably more appropriate.

Having defined the boundaries of the Agile organisation, attention should then shift to all the other functions and operations to identify what is causing waiting times. Typically, these will include annual and quarterly budgeting cycles, procurement and legal processes, marketing and go-to-market strategies, supplier lead times and so on. The key point is that if lead times in all the ‘non-agile’ parts of the business are not addressed, it is irrelevant how efficient or nimble your scaled agile implementation is. You would have simply optimised a sub-system and had minimal impact on the system as a whole.

Final Thoughts

In this post, I have sought to organise my thoughts on what I believe is required when running agile transformations beyond a few teams. There is no such things as ‘true’ or ‘fake’ agile. Likewise, the endless debates about which scaling framework are not only tedious, but they miss the nub of the problem. There are simply successful transformations that meet business goals, and unsuccessful ones which don’t. Nothing else matters. Hopefully, this post has provided a few pointers that will improve the odds of being in the right category.

Why Scaling Agile is not Easy