Reading classic papers is an effective means of learning; this article is a sharing of the understanding of “No Silver Bullet — Essence and Accident in Software Engineering”. There must be omissions, and it is strongly recommended to read the original text;
If you’ve ever been lost in the depths of software development and felt like a tidal wave of problems, Fred Brooks’ classic 1986 paper “No Silver Bullet — Essence and Accident in Software Engineering” may point you in the right direction. This paper digs deep into the complexity of software engineering and offers some practical countermeasures.
The core idea of the article is that the challenges of software development are divided into two types: one is “Essential Difficulties”, which are unavoidable problems determined by the nature of the software itself; the other is “Accidental Difficulties”, These problems are not inevitable, but they do exist and cause trouble for development.
So what are “essential difficulties”? They stem from the essential properties of software, the abstract structure of datasets, data relationships, algorithms, and function calls. No matter how you use various programming languages or algorithms to describe it, the core of these structures is the same. But it is because of this abstraction that software development is complicated. All the details must be precisely and clearly defined to ensure that the software performs as intended. This means that the difficulty of software development lies mainly in the normalization, design and testing of these abstractions, not just coding.
So, while we may make some syntactic mistakes when writing code, the real big problem is conceptual errors in the system. That’s why there’s no magic “silver bullet” that solves all software development problems at once — that’s what “no silver bullet” means.
So, when we face the complexity of software development, we must accept the fact that software development is inherently difficult work. We can alleviate these difficulties by improving technology and management strategies, but we cannot completely eliminate them. This is the lesson that this paper wants to convey to us.
Does software development have to be so difficult?
Next, let’s take a concrete look at the irreducible essential properties of modern software systems: complexity, consistency, variability, and invisibility.
Complexity: In software, complexity is hard, and it’s not as easy to solve as problems in mathematics or physics can be solved with a neat model. For example, imagine the email client side, which has several cumbersome tasks to do, such as receiving and sending emails, managing contacts, and filtering email spam. And, to do that, you need to deal with a lot of complicated details. If we try to simplify too much, like sending only plain text emails, it always succeeds, then we miss a lot of key parts. Therefore, complexity not only increases the difficulty of software development, it also causes a series of problems, such as communication difficulties, product errors, over budget, delays, and so on.
Compliance: Rather than physicists believing in a unified truth, software engineers often have to deal with the additional complexity of various system interface designs. These complexities are not required, but are caused by differences in designers.
Mutability: Software is like a piece of plasticine that can be shaped at any time, mainly because of its functionality and ease of modification. For example, when the software is found useful and used in new scenarios, the requirements will change; or when the hardware is updated, the software needs to change with it. Not only that, but changes in the application environment, user requests, regulatory restrictions, and even the hardware environment can also lead to changes in the software.
Invisibility: Software cannot be visualized like architectural blueprints or diagrams of mechanical parts. We can draw flowcharts that express the structure and function of software, but because software complexity often involves multiple dimensions, such as control flow, data flow, dependencies, time series, etc., our diagrams may not be so simple planes or hierarchies. This complexity, along with the “intangibility” of software, not only makes software design more difficult, but also hinders our communication when discussing software design and functionality.
Progress in solving occasional difficulties
High-level languages: It does make it easier to write programs, and it also improves the quality and simplicity of the software. However, high-level languages are now getting closer to our usual way of thinking, which also makes the details of the language a bit onerous.
Time-sharing system: It allows us to see the big picture of the system at any time and helps us better understand complex systems. But its main benefit is that it makes the system react faster, and there’s no point in speeding up if the reaction time is already very fast.
Unified Programming Environment: This thing makes it much easier to combine programs. For example, it provides integrated libraries, unified file formats, which allow us to easily implement program functions that can theoretically cooperate with each other.
Hope for the silver bullet
Ada and other high-level languages: Languages like Ada have advanced in design and modularity, but it won’t be the “silver bullet” that makes software productivity soar. Its value lies in helping engineers get rid of the complexity of abstracting machine language. In addition, Ada’s greatest benefit to us may be in pushing us to learn more modern software design techniques.
Object oriented programming: The progress brought to us is real, eliminating a lot of complexity and making design expression more advanced. However, this technology can only solve the problem of design expression at most, and the complexity of the real design itself is still there, only the “expression” is changed, and it cannot be improved by an order of magnitude.
Artificial intelligence, expert systems: While AI may make a difference in specific fields, such as automatic speech recognition or image recognition, its impact on general-purpose software development is minimal. As for expert systems, it has the advantage of being able to handle complex problems and provide programmers with solutions and advice, but the challenges of acquiring knowledge and refining the rule base remain.
Automatic programming: In fact, programming in a higher-level language; however, we need to design solutions rather than just describe the problem itself. In some specific scenarios, such as sorting program generators and systems that solve differential-integral equations, automatic programming is effective. But in most ordinary software systems, because we often cannot describe the problem concisely and clearly, at the same time, there may be multiple solutions to the same problem, and there is a lack of clear selection rules, its application is limited.
Graphics programming: It may not be as beautiful as we think. Reasons include: flowcharts cannot effectively reflect the real structure of software; screen pixels of existing display devices limit the display range and resolution of software diagrams; the complexity of software is difficult to fully display, and various visualization methods can only capture part of the information, and it is difficult to get a global overview after stacking. Software systems and chip designs are not directly comparable. The latter is two-dimensional and layered, and the shape reflects the essence, while software system visualization is difficult.
Program validation (testing): It is very important in some areas (such as the secure operating system kernel), but it is also unlikely to save a lot of labor. Also, testing does not guarantee that the program is error-free, as mathematical proofs can also be wrong. Most importantly, testing can only prove that the program meets its specifications, and developing a consistent, complete set of specifications is inherently difficult.
Environment and tools: There are definitely improvements in productivity and reliability. However, big breakthroughs may have been achieved, and future gains may gradually diminish. For example, smart editors can help avoid syntax and simple semantic errors, but this only provides marginal help.
Workstations: Although the capacity and memory capacity of workstations will definitely increase, this may not bring huge gains. Programming and document editing have been supported enough that even with a 10x increase in compilation speed, thinking time is still the main activity of a programmer’s day.
In general, these technologies may bring some improvements, but they cannot fundamentally solve the complexity and difficulty of software development, nor can they bring about a qualitative leap. Therefore, we cannot rely too much on these “silver bullets”, but should pay more attention to the inherent difficulties of software development and find more fundamental solutions.
Essential Complexity Promising Challenges
Buy or Build: Buy off-the-shelf software or build your own. As more and more great software comes out, it’s often more cost-effective to pay for an already made product than to build one yourself from scratch, and use it right away. Because computers are so ubiquitous, we’re all willing to adapt to existing software, like spreadsheets and simple database systems, that allow us to do many applications that previously required customization.
Progressive development, not complete construction: develop gradually, not all at once. First establish a most basic operating framework, even if it does not have any actual function at first, and then gradually increase the function on this framework, and gradually improve and improve. The growth of similar organisms first has a basic skeleton or cellular structure, and then gradually grows and develops on this structure, and finally forms a complete organism.
The advantage of this is that it allows flexible adjustments and changes during the development process, can adapt to complex and changing requirements, and can detect and solve problems in a timely manner. In addition, because results can be seen in a timely manner, it can also motivate the enthusiasm and innovative spirit of the development team.
Requirements Definition, Rapid Prototyping: Deciding on the exact requirements of the software system you’re going to do is the most difficult because customers often don’t know exactly what they want. Take an iterative approach to extracting and refining product requirements, clarifying requirements through constant communication and revision. Make a system prototype that customers can try out so they can experience the main features of the product ahead of time and better understand and clarify their needs.
Great designer:
People are always the most important thing. Good designers can grow through learning and practice, but great designers must have unique creativity and perspectives. Therefore, we need to discover and develop great designers within the organization. It’s like finding and nurturing potential management talents. How to develop great designers?
- Systematically identify top designers early on
- The best designers are not always the most experienced
- Assign a career mentor responsible for the development of potential talent and maintain a careful career profile
- Develop and maintain a career development plan for each potential talent, including carefully selected apprenticeships with top designers, advanced formal education stages, short courses and more
- Provide opportunities for growing designers to communicate and stimulate each other
