An attempt at a unified theory on software engineering

Recently, i have been musing over a thought (a question more correctly) that i have had for a very long time. I have been writing and building software for a while and this question has repeatedly popped up in head on occasion.

Like many of my colleagues, over the years, i have noticed that the more i learn and know about software engineering; the more i realize that i don’t know enough to deal with every new challenge. Is this an inherent property of most software engineering challenges in general or is it a property of my own bias and objectivity (or lack thereof) ?

I have had so many questions!

I have always wondered if the vastness of all you needed to know as a software engineer could be summarized into a single unified theory. One that explains all of what software engineering entails and represents.

All the concepts, the guidelines, the principles found in software engineering.

So, putting it simply:

Does software engineering have a unified theory ?

Why is this question important at all ? Secondly, why should it ever be important to all software engineers ?

For years and decades, the software engineering community across the world has been locked in a debate (or perhaps argument) about principles, best practices, anti-patterns with very little consensus formed.

We argue, we fight and sadly berate each other constantly about our choices and preferences with little eternal justification for such actions.

We have formed diverse opinions on software engineering (deviating from one another) with very little peer-review work done to advance these opinions with proof that they can stand the test of time.

Finally, presentism has taken over as we dismantle and vehemently criticise the work that was done by older, veteran software engineers for not being a 100% perfect (as if anyone or anything can be 100% perfect) in their submissions/opinions on software engineering.

Personally, i have managed to offer my own thoughts on this raging debate in this article.

Also, i have found it very difficult for a long time to link/summarise all the different things i have learnt and continue to learn about software engineering into one coherent theme.

If i found a way to link everything i know into one coherent theme of knowledge, then my brain wouldn’t hurt every time i encountered a new software engineering challenge. Right ?

Hmmm… 🧐

So, my answer to the question above ?

I dare say software engineering does have a unified theory.

Please, here me out before you roll your eyes 🙏🏾

I like to call it the Ifeora theory on software engineering (as follows):

All of software engineering (e.g. SOLID, Design patterns, DDD, Tests, Static Types, Embedded systems, Distributed systems, Infrastructure) can be distilled into 2 simple yet grand points:

- Abiding by a set of meaningful constraints
- Deciding off of a set of reasonable trade-offs

And trying not to overdo any of the two above

I have held this idea in my mind for a long time now albeit as a rough uncollected thought. I even mention it briefly as an intro to one of my earlier article.

As i have continued to write software and improve my experience over the years, i have also continually revisited this uncollected thought of mine and refined it. Now, i am confident and convinced that it is at the heart of everything we do (or should definitely do) as software engineers.

When building/writing software we are constantly making trade-offs on what we are doing, how are doing it, when we do it, where we do it or don’t do it. We are regularly making an Effort vs. Reward choice and/or Cost vs. Benefit choice.

What about trade-offs ?

For example, people who dislike SOLID principles completely say it leads to poor performance and over-engineered solutions. They also say it isn’t explicit enough. However, on closer inspection (with a shift in perspective), and in the grand scheme of things, SOLID principles are simply (in summary) just a set of choices that bother on Cost vs. Benefit.

SOLID principles cannot be thoroughly explicit too because they center around trade-offs and are heavily dependent on context. When you apply a SOLID principle in a wrong context, then you get over-engineered solutions.

What do all coding design patterns have in common ? Well, indirection!

What is the decision to introduce an indirection or not to introduce one called ?

A trade-off.

Remember why senior engineers always begin answering pertinent questions with “It depends…” ? Well, that applies everywhere. This is what makes our jobs as software engineers a highly cerebral one.

What makes SOLID principles useful in my strong opinion (weakly held) is that they are reasonable. Unreasonable trade-offs are not desirable in any way when building software. Yet, you have to be careful with reasonable trade-offs as overdoing them can also lead to very bad outcomes.

When software engineers say things like: “We don’t need more guard rails, we need more safety nets”. What i am hearing is that engineers want to eat their cake and have it.

It means they aren’t interested in working within the confines of a set of constraints (which is the best way to build software in my honest opinion). It means they want to be able to do whatever they like with the codebase as long as its just possible but don’t want to deal with any of the fallout of such decisions.

This is a rather unreasonable approach to building software.

As my people say in Nigeria: “Balance dey for middle!”

When writing software tests; why would you consider using a test double (i.e. a fake or a mock) rather than the real thing ? Is it always feasible and practical to use the real thing in your all tests without exception ?

Well, if you are like me, the answer is:

It depends… 😬

Okay, it depends on what ?

It depends on Effort vs. Reward. It depends on Cost vs. Benefit

What does that even mean 😢 ?

It means that you have a decision to make that will be based on how much cost you are willing to pay and what level of benefit you are able to accept for it.

This decision will also (in addition) be based on how much effort you are willing to put in at the time of writing the test and what level of reward you are willing to accept for it.

Like in life, we cannot win them all. We win some; we lose some. As software engineers we learn to balance feasibility with reality. We learn to be ruthlessly pragmatic (as follows):

• Are you okay with having your tests running for a longer time when you use the real thing or for a shorter time when you use a test double ?
• Are you willing to take a small hit on the reliability of your tests if it means that your tests take less time to run completely when you use a test double ?

See ? You are always making trade-offs.

Just make sure the trade-offs are reasonable and balanced.

Casey Muratori, a very experienced game software engineer published a (now popular) rant on YouTube in 2023 titled: “Clean Code, Horrible Performance”.

Everything he said in that rant video was mostly spot on but it was only 25% of the whole story.

His view is limited to the game development industry. I think this Hacker News user (aforwardslash) put it best in this comment. Performance is not mostly a priority if it brings little ROI to businesses like start-ups. Sometimes, performance only becomes a priority after the business has suffered user apathy for a software product.

If a start-up needs to build out features faster to increase business value, it would prioritise that over improving performance.

Trade-offs, trade-offs, trade-offs.

Also, the true causes of performance issues in software isn’t clean code per say. It’s excessive, bad and/or expensive indirection and heavy cpu-bound tasks.

Virtual-table lookups are one of the top reasons for performance issues in OO (Object-oriented) languages that employ polymorphism. Now, Virtual-table lookups are a very expensive indirection that is needed to make code easier to write and read as well as improve the maintainability of code.

However, the cost is a noticeable performance hit.

In other words, OO languages are making a trade-off on improving maintainability and simultaneously degrading performance. The pertinent question to ask is this:

is this a reasonable trade-off ?

Well…

It depends… 😬

It depends on if performance is of greater business value and whether an investment in improving performance is sufficiently incentivised (although i believe there’s always a business case to be made for improving software performance but it might not be of immediate priority).

It also depends on whether polymorphism is excessively used in parts of the codebase it has no business being used.

This reminds me of the time when Java 1.5 SE/EE made it such that everything had to be a class even when it didn’t need to be. That was such a dark period in history.

Some things are simply meant to be functions not classes. Such choices by programming language designers had on massive impact on the outcomes for performance as is the case with Java 1.5 SE/EE.

Therefore, an effective way to improve performance generally is to cut down on the amount of indirection a piece of software allows and to run long-running heavy tasks in a thread or deamon.

What about constraints ?

The constraints part is what keeps accidental complexity, chaos at scale and emotional frustration that can grow out of hand as we build bigger mission-critical software under control.

Meaningful constraints also help limit abuse. It is very easy to abuse a software tool or process and then turn around to blame the tool for being hard to use.

Yes, there are times when the tool comes with a horrible essential issue/complexity.

For example: CSS (Cascading Style Sheets) has been constantly plagued with the fact that the cascade logic is global and not easily scoped.

Therefore, many attempts have been made over the years to fix that problem (e.g. :where() , :is() , @scope, @layer).

The impedance mismatch issue between how data is structured in databases and how they are used in the core software is a forever problem and doesn’t have a reliable solution yet.

However, we must determine when these essential issues with the tool are the problem and distinguish that from when our misuse/abuse of the tool is the problem.

Have you ever noticed that whenever you are building/writing software with zero constraints (i.e. you do anything you like because its’ possible and because you can); things get out of hand real fast ? Well, i have.

Static types are a very good example of a meaningful constraint. Yes, it can take some getting used to as it inconveniences you in the beginning. But, as time goes on, you come to value the benefits over the discomfort (See 😀, we just circled back to the Cost vs. Benefit point i was making earlier).

Does this mean i should always use static types without exception every time ? Can i always get the same level of benefit from using it (in relation to the cost) in every scenario or situation ?

Well…

It depends… 😬

Oh my gawd! 😭 What does it depend on this time ?

Same as before actually. It depends on Effort vs. Reward. It depends on Cost vs. Benefit

Sometimes, static types can get in the way more than they are helping in a given situation.

Personally, i don’t use static types (with JavaScript) if i am building a POC for a spike or struggling to get something to just work. I immediately avoid TypeScript.

I also prefer not to have static type checks if i am working on a small solo project (OSS or not in its’ alpha version) in JavaScript because i have trained myself to write JavaScript defensively (or offensively) over the years.

Yet, i always welcome static types when working as part of a team irrespective of project size. In a team, the cost of writing say TypeScript is balanced out by the overall benefit because the total cost of defining types and using them is usually shared amongst each team member.

It’s all about Cost vs. Benefit. That’s the most important thing here.

When you come across a software library/framework that describes itself as “opinionated”, it simply means that the library author has created a set of meaningful constraints (sadly sometimes going overboard) that you as a user of the library/framework need to abide by.

On average, the “opinionated” libraries/frameworks are usually better and more useful than the “unopinionated” ones (🙂 tell me in the comments if you disagree and why).

Again, notice i qualified the word “constraint” as meaningful. Why ? well, because constraints that aren’t meaningful or thoughtful hurt you twice as much and hurt the software you are building/writing.

This is why i maintain that the best software engineers aren’t the lazy ones (i do admit that laziness and shortcuts has its’ pride of place in software engineering but it should never be abused — bad things always happen when it is).

The best software engineers in my book are the disciplined ones. This kind of software engineers (when well experienced) can define meaningful constraints for themselves and stick with them even if it causes them great discomfort initially or for a while.

I have a personal rule:

Always be objectively lazy

You have come again! 😠😡 What does this one mean now ?

It simply means that you are lazy only when you have to be.

If there’s a patch that needs to go out to the live software instances for a bug because the bug is a serious blocker for a lot of paying customers who make use of the companys’ software.

At this point, you aren’t interested in doing a thorough job. There’s just no time for it. So, it’s okay to take multiple shortcuts like skipping a failing test or coming up with a hack or workaround (rather than a proper fix).

The most important thing is to push a hotfix out to production as soon as possible. Later on, you can come around and implement a proper fix.

But there are other cases where being lazy from the onset for no immediately useful reason might bite you in the ass in the the future.

Good, Fast or Cheap ?

The problem however, is that we (software practitioners) keep trying to either come up with a general constraint that works for every situation or context or we try to shoehorn a known generic trade-off to strictly fit exactly one context or no context at all.

I call this “the error of our ways” (See tweet/post below):

I cannot emphasize here enough how mistaken Mr. Kelsey is.

I have argued severally that best practices (e.g. TDD, SOLID — i know you don’t agree that these are best practices; i know 😁 & it’s fine) always fit into any context that meet specific pre-conditions. Whenever these pre-conditions no longer hold true (even within a context where they once held true — perhaps because the scale of use or complexity has increased or other parameters changed), that best practice becomes unusable for that context.

Still, it does not change the fact that it is a best practice (yet only for contexts where the pre-conditions hold true). Anti-patterns are a different matter all together. They should always be avoided at all cost. Anti-patterns are (in my opinion) the very likely result of abusing a best practice. Also, Anti-patterns have no way to safely use them ever.

Best practices hold up at any scale as long as pre-conditions hold true. Design patterns are not best practices because they do not hold up at a certain scale irrespective of the pre-conditions holding true.

For instance, multiple inheritance looks like an anti-pattern but it is not. Why ? because a proper anti-pattern has no way to safely use it. Yes, i know it leads to the diamond problem as well as other issues. However, all these issues are man-made (i.e. not caused by the nature of inheritance but by abuse due to the bad habits of programmers) and can be mitigated. Similarly, the only reason why the interface segregation principle exists is because of the bad habits of software engineers who prematurely (i.e. performed before any substantial amount of code is written) create interfaces.

The only thing i believe is that it (multiple inheritance) should be used scarcely and only when absolutely needed per solving specific domain problems.

Also, there should be constraints that guide its’ usage.

I personally have 4 meaningful constraints that guide me when using multiple inheritance :

1. Zero Overrides — Never override anything from a parent class.
2. Only abstracts — Only use an abstract class as a parent class used within and outside the same packages/namespaces.
3. Same namespace — Only extend from a (abstract) parent class within the same packages/namespaces. This means once inheriting from a parent in one package/namespace, you can’t inherit from another one different package/namespace.
4. Final Concretes — All concrete (non-abstract) classes defined within any package/namespace MUST be final and never open for extension.

These 3 constraints ensure that the diamond problem or other issues never ever arise at all ever again. Also, the 3 constraints are in line with proper modular software design. I came up with these constraints by analysing the cost and benefit of each constraint to the work that i was doing some time ago using inheritance. I had to experiment a lot and watch the outcome over time.

For single inheritance, i have 4 meaningful constraints as well that guide me:

1. Only abstracts — Only use an abstract class as a parent class used within and outside the same packages/namespaces.
2. Same namespace — Only extend from a (abstract) parent class within the same package/namespace. This means once inheriting from a parent in one package/namespace, you can’t inherit from another one different package/namespace.
3. Use Final Methods — Use final methods on public parent classes that implement an interface for extension outside of the same packages/namespaces.
4. Final Concretes — All concrete (non-abstract) classes defined within any package/namespace MUST be final and never open for extension.

As you can see, for single inheritance, i allow overrides. However, for single inheritance, i take an extra (defensive programming) precaution of specifying which methods in a parent class can be overridden and which can’t using the final (or sealed for C#) keyword.

Furthermore, the pre-conditions for any best practice are the “meaningful constraints” that ensure said best practice isn’t abused. Once you notice that the pre-conditions no longer exist or hold true in a given context/situation, stop using that best practice at any scale.

Let me give a concrete example with TDD (Test-Driven Development).

TDD should only be used under these 2 pre-conditions being met:

1. You know enough to create the behaviour you need using code logic. This means that there are no gaps in your knowledge about how to create the desired logical behaviour.
2. You are employing outside-in reasoning. This means that you are not pre-occupied with the implementation details at first. You are focused on the public interface and how it interacts with other artefacts/interfaces first before considering implementation details.

In my experience, whenever these 2 pre-conditions weren’t met, TDD instantly became a pain to use. I would find myself putting TDD aside during spikes or when building a POC (proof-of-concept) where i don’t yet fully know how to create the desired behaviour. This meant i won’t write tests

Only after i figured it out, would i pick TDD up again and start coding afresh but using the POC codebase as a guide.

This is the same for every other best practice.

Also, there’s always a cost to using a best practice. There’s always a trade-off you are making!

I (as a software engineer) believe that instead of spending time trying to wax philosophical about what software engineering fails me at, that i spend that time getting better at creating more meaningful constraints and reasonable trade-offs (per context/situation).

That to me is a better use of my time.

The way to get better is by becoming very good at observing yourself as you work and taking lots of measurements while/after you work. These measurements will form the basis for how you make significant decisions about building software and make your intuition better.

Essentially, get better at analysing cost and benefit as well as effort and reward. This is even much more crucial for deciding when and where to allow coupling/cohesion in software code and where to disallow it.

Conclusion

I think that the software industry as a whole needs to have consensus on better ways of building software rather than complaining about its’ current dismal state. It would be a tough endeavour (no doubt) but it will be worth it.

I hope this theory of mine makes you reflect on how software should be built. Soon, maybe soon, we will have that much needed consensus.

Now, after many years as a software engineer, i see that all of the (seemingly distinct and isolated) sub-topics and topics (or knowledge) about software engineering isn’t so different or isolated after all.

Nothing is ever black or white. Greys exist out there so instead of stating categorically that some practices are bad and other are good. it’s better to try to learn the Pros & Cons of each practice.

It’s all connected. Actually, more connected than you might think.

Hence, the need for consensus and yes — a unified theory!

Cheers 👍🏾