Why Agile?

In case you missed it, I’m pivoting into product management. As a part of the pivot I have to learn new things. And because writing is a core activity for me, I’ll be documenting my questions and explorations along the way.

The first question I have comes in response to the Agile Manifesto, which is as follows:

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools.
Working software over comprehensive documentation.
Customer collaboration over contract negotiation.
Responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.”

My question is a simple one: Why?

From recent conversations and readings it seems that the Agile Manifesto was developed for a specific reason: To better enable the research, development, release, sustaining and renewal of complicated products in complex environments.

First point: complicated is not the same as complex. A prosthetic hand is complicated; a human hand is complex. Second, what is it that makes a system complex? After some digging (here, here, here and here), I’ll I’ll narrow it to three things:

  1. A complex system is more than the sum of its parts.
  2. A complex system is unpredictably sensitive to any variation in input, structure or scale.
  3. A complex system’s behaviour cannot be predicted, only modelled probabilistically.

Next question: for a unit to survive–let alone thrive–in a complex system, what does it take? It needs to have just the right amount of size, smarts and speed. The respective spectrums are as follows: small-large, dumb-smart and slow-fast.

For the size spectrum, I like the breakdown I first saw in this Ribbonfarm post, Unflattening Hobbes. The units are individual, pack, troop, tribe and imagined community. (Note: the breakdown was introduced in Pack Experience, and the post includes a good discussion of the differences which I won’t quote here.) The general consensus, in terms of robustness to reality, seems to be:

Pack > Individual / Troop / Tribe / Imagined Community.

Phrased differently: the latter four unit sizes have jockeyed for superiority throughout history, but only the pack has maintained its position throughout.

The smarts spectrum is counter-intuitive. The obvious inclination would be to be as far towards smart and as far from dumb as possible. However, the reality is more nuanced than that.

Being dumb can certainly aide survival. A rock, for example, survives in the complex system of nature for a long time. The problem is that it has no agency. I don’t know about you, but I have no desire to be a rock.

Being super smart can work too. I’ve read about some super-genius humans and their exalted level of intelligence certainly allows them to excel. However, it doesn’t seem to result in a quality of life that is many times superior to the average. Better to know a genius, perhaps, than to be one.

Super computers are another angle here. Computationally speaking, they’re smart. But they also rely on a surprisingly complicated technological stack for existence–of course, such dependencies mean net fragility.

The best compromise for a unit is to think smart and act dumb. In the context of an individual playing 4D chess: “Simplify your actions, increase your randomness, complicate the game for the adversary, and try to stay alive.” In the context of a pack navigating in a complex environment: consider diverse perspectives, allow for known and unknown constraints, then act in a co-ordinated, coherent manner on a short timescale.

The speed spectrum is also counter-intuitive. When I refer to speed I mean the speed of straight-line movement as well as the ability to change direction. This results in three qualities: acceleration, deceleration, and re-orientation. Now, it seems plausible to think that maximising these three qualities is desirable. But consider the maxim, Slow is Smooth, Smooth is Fast. Or consider my own maxim, the faster you go, the less you see.

Another way to look at this is via the tech-adoption curve. Units that don’t adapt to change at all, or do so very slowly, don’t tend to do that well. Those who over-adapt to change also don’t do that well–premature optimisation is the root of all evil, move too fast and things break you, etc. So in terms of speed, it seems the best bet is to opt for a tempo that is marginally faster than the mediocre. In Formula One terms: think point-scoring position as opposed to taking pole.

After that brief tour, I think it’s become apparent why the Agile Manifesto came about. Its authors reasoned that to survive and thrive in a complex environment, a unit has to be small (but not too small), smart (but not too smart) and move fast (but not too fast). Thus, they came to value:

“Individuals and interactions over processes and tools.
Working software over comprehensive documentation.
Customer collaboration over contract negotiation.
Responding to change over following a plan.”