No Silver Bullet—Essence and Accident in Software Engineering by Frederick P. Brooks, Jr. (1986) (read in 2019)
Published by marco on
Disclaimer: these are notes I took while reading this book. They include citations I found interesting or enlightening or particularly well-written. In some cases, I’ve pointed out which of these applies to which citation; in others, I have not. Any benefit you gain from reading these notes is purely incidental to the purpose they serve of reminding me what I once read. Please see Wikipedia for a summary if I’ve failed to provide one sufficient for your purposes. If my notes serve to trigger an interest in this book, then I’m happy for you.
This is a relatively short but important essay in the world of software engineering theory. It’s not really for programmers who’ve stumbled over from scripting Photoshop or who’ve decided that there’s good money in copy/pasting code that they don’t understand from StackOverflow. The audience is more self-selecting. If you’re likely to read an essay with this name, then you’ll likely be receptive to its ideas. That is, if you don’t already agree with the premises in the document, you’re unlikely to be convinced by it.
Brooks is one of the inventors of OS/360 for IBM mainframes and, more famously, the author of the book The Mythical Man-Month. He draws on a lot of experience when he writes that the difficult bit of software is not writing it so that it works—it’s figuring out what you want to write in the first place.
“I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation.”
He argues in this paper that where we often go wrong is when we attempt to “abstract away […] complexity” but end up “abstract[ing] away its essence”. Abstractions are a good thing. The only way to represent any process in software is to create a model of it. But we haven’t historically been careful enough that what we ignore—the corners we cut, the values we round—aren’t part of the essence of what we’re building.
This leads Brooks to the next conclusion, that, “No facilitation of expression can give more than marginal gains.” If you want to write software more quickly, changing languages or runtimes or patterns will offer, at best, marginal gains relative to improving the process you use to design that software and gather and define its parameters and requirements.
This is a tragic conclusion for most programmers, a vast majority of whom are much more interested in trying out new techniques and languages and IDE tools and extensions and just tweaking the hell out of the implementation side of things. This is all very interesting and can lead to fruitful gains. Brooks acknowledges as much,
“The gap between the best software engineering practice and the average practice is very wide—perhaps wider than in any other engineering discipline. A tool that disseminates good practice would be important.”
But once you’re testing properly and have a good framework and a good editor and you’ve got CI and maybe even CD, there’s no more room for quantum leaps in improvement. At that point, you can only get significantly better and faster by writing only the code that you actually need. For that, you need to optimize defining your requirements.
Grasping and defining requirements is by no means an easy thing, as any non-trivial software involves a state machine with exponentially increasing combinations of states.
“From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program, and from that comes the unreliability.[1]”
Where does this complexity come from? Often, from outside of the system, at the edges, where the software must interface with other systems. Perhaps these are legacy systems; they are almost certainly less flexible than the software being designed and written right now. The new software is, by definition, more malleable than software or processes already in production. As Brooks puts it,
“[…] all cases, much complexity comes from conformation to other interfaces; this cannot be simplified out by any redesign of the software alone.”
All of this necessary/essential complexity “makes personnel turnover a disaster” once you’ve trained someone to understand it. This is a lesson that the software world—with its focus on exchangeable resources that just provide hours of work—has never learned or, more generously, forgotten.
Perhaps this is due to the prevalence of so many layers of management. They generally don’t know how to do anything special and are generally highly interchangeable with other managers who also aren’t very special. Their ego depends on their worldview considering all other people to be the same. This includes the highly trained and skilled staff who they manage.
Brooks ends with an eloquently and succinctly stated summary that jibes 100% with my experience over the last 25 years of designing and building software.
“Therefore the most important function that software builders do for their clients is the iterative extraction and refinement of the product requirements. For the truth is, the clients do not know what they want. They usually do not know what questions must be answered, and they almost never have thought of the problem in the detail that must be specified. (Emphasis added.)”
This is what we have to work with: the domain specialist (the client) doesn’t have the know-how to even know how to describe the domain. Sometimes there is no real domain specialist; there is just someone with money and a vague idea or, even worse, someone who thinks that they are a domain expert. The job of a software designer is to become (enough of) a domain specialist to bridge the gap. If nobody bridges the gap, then the software will fail.
[1]
This reminds me a bit of an article I read called Text Editing Hates You Too, which included the following:
“The necessary complexity here is immense, and this post only scratches the very surface of it. If anything, it’s a miracle of the simplicity of modern programming that we’re able to just slap down a <textarea> on a web page and instantly provide a text input for every internet user around the globe.”
Here, too, the author discusses necessary complexity—perhaps better described as immanent complexity.
↩Citations
“The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation.”
Position 38-42
“The complexity of software is [a]n essential property, not an accidental one. Hence descriptions of a software entity that abstract away its complexity often abstract away its essence. Mathematics and the physical sciences made great strides for three centuries by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties experimentally. This worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.”
Position 51-55
“From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program, and from that comes the unreliability.”
Position 57-58
“[This complexity] creates the tremendous learning and understanding burden that makes personnel turnover a disaster.”
Position 62-62
“In many cases the software must conform because it has most recently come to the scene. In others it must conform because it is perceived as the most conformable. But in all cases, much complexity comes from conformation to other interfaces; this cannot be simplified out by any redesign of the software alone.”
Position 68-70
“All successful software gets changed. Two processes are at work. As a software product is found to be useful, people try it in new cases at the edge of, or beyond, the original domain. The pressures for extended function come chiefly from users who like the basic function and invent new uses for it.”
Position 76-78
“As soon as we attempt to diagram software structure, we find it to constitute not one, but several, general directed graphs, superimposed one upon another. The several graphs may represent the flow of control, the flow of data, patterns of dependency, time sequence, name-space relationships. These are usually not even planar, much less hierarchical. Indeed, one of the ways of establishing conceptual control over such structure is to enforce link cutting until one or more of the graphs becomes hierarchical.”
Position 86-89
“The hard thing about building software is deciding what to say, not saying it. No facilitation of expression can give more than marginal gains.”
Position 161-162
“Even more difficult and important is the twofold task of knowledge acquisition: finding articulate, self-analytical experts who know why they do things; and developing efficient techniques for extracting what they know and distilling it into rule bases. The essential prerequisite for building an expert system is to have an expert.”
Position 190-193
“The gap between the best software engineering practice and the average practice is very wide—perhaps wider than in any other engineering discipline. A tool that disseminates good practice would be important.”
Position 194-195
“The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification.”
Position 235-236
“The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is so difficult as establishing the detailed technical requirements, including all the interfaces to people, to machines, and to other software systems. No other part of the work so cripples the resulting system if done wrong. No other part is more difficult [t]o rectify later.”
Position 282-285
“Therefore the most important function that software builders do for their clients is the iterative extraction and refinement of the product requirements. For the truth is, the clients do not know what they want. They usually do not know what questions must be answered, and they almost never have thought of the problem in the detail that must be specified. Even the simple answer—”Make the new software system work like our old manual information-processing system”—is in fact too simple. Clients never want exactly that.”
Position 285-288