On Iterative Software Development

Nowadays, almost everyone doing any type of software development wants to be very "agile", which for some reason usually means doing sprints and iterations. The idea being sold here is that it is almost impossible to know all the requirements in advance. Therefore, it's better to do something, release it, learn from it and iterate in short steps so that the customer gets what they want in the end, while we waste as little effort as possible. The thing is, more often than not, the effort savings seem to be on the side of the project managers and customers, while the developers and other engineers bear the unfair burden of this setup.

What am I trying to say? See, with this iterative approach, the customer doesn't need to know what colour of the car they want, we can just give them a red one. The project manager doesn't have to sit down with the customer and talk them through the various options and their trade-offs, getting them to sign off on something. We simply decide in favour of the red because we all know that red colour makes cars go faster. If the customer is happy, good, but if they decide they want a blue one, we will paint it blue in the next sprint. And if they get green spoiler and realise that the rest of the car should also be green, then we will paint it again in the next sprint. The project manager just has to nod in agreement and offload the work onto the engineers, who are stuck in a repetitive paint cycle that is adding layer upon layer of pain(t) to the job, and probably reducing aerodynamics of the car each time. It gets worse. When the customer realises they want an aeroplane, that is where developers really start to suffer. See the project was supposed to be a car. The position of the engine, the strength of the cabin, everything is designed for a road vehicle. Aeroplanes need a different design right from the start. But building an aeroplane is still squeezed into a sprint, or perhaps split into a couple of sprints and called an epic.

What usually happens in this case is that developers basically try to do what is required of them. They put in reinforcements. They add some transfer mechanism for the propeller and they make it work. Such an aircraft is of course poorly designed, inefficient and prone to breakage. Work on it soon slows down to a crawl and suddenly every single change requires a lot of work and research because we are always at the limit of what our "aeroplane", which still looks and works like a car, can do. Since productivity is so low, the pressure is put on the developers. We have an aeroplane, so we should be able to equip it with a powerful jet engine, right? That's what aeroplanes do. Why aren't we able to deliver it? And then we end up in long sprint retrospectives about what went wrong during development and how we ended with up with so much tech debt.

We don't build houses the same way because we understand that we have to pay attention to load-bearing walls and foundations and other criteria. We expect to have complete plans long before the actual construction. Yes, there is still the possibility for changes at later stages. A window can be moved. Or simply not be there. The furniture can be rearranged, but the kitchen and bathroom are usually non-negotiable because of the installations. We can change the colour of the house, but if a client wants to make the house 5 floors higher, they will be told no. And if they insist, the whole construction of the house is stopped and new plans are made and the architects try to find out if anything can be salvaged from the current construction or if it is better to just start from scratch, with a deeper and stronger foundation. And none of that happens in 2 weeks. And I would argue that software development should be treated the same.

Imagine you are some kind of software architect or maybe just a lead developer who is at the beginning of a new project. Your job is to select technologies, tools and architectural patterns for the software so that the team can start working on it. You need to think about how the data will flow through the future system, and you need to decide on the right levels of abstraction pretty much from the start if you want to achieve a lean and efficient design. All of these decisions involve trade-offs, most of which are known in advance. Do we want a modular, feature-orientated architecture that focuses on the user interface, or do we want lamba architecture for the data flow? Do we want a low-level programming language that can be optimised to the highest degree, or should we rather focus on fast delivery? All these decisions need to be made on day one, but because you are "agile" you only get the requirements for a quick and dirty PoC, with no guarantee that it is representative of the final product. You as an engineer have to blindly make decisions so that the requirements setters don't have to make any. The responsibility is pushed on you. Can't you see how unfair that is? Can't you see how this will always end up the same?

I cannot offer a solution. In many cases, the power imbalance between the requirements setters and the engineers means that there is no recourse. You can't stop the endless sprint cycle, scrap half the application and spend two months redesigning it when the client asks you to remove a load-bearing wall. You have two weeks to create another deliverable. So all I can say is: I see you, you are my people, it is not your fault.