It’s a sunny day. You are in the middle of the road leading to your village, where your friends, your family, and you favorite dog are living. What a great day!
Suddenly, you hear a terrible and monstrous scream scattering the peace of the countryside. A massive and hideous Hydra is rushing to you, willing to destroy your whole village. You’re the only one on the road to stop this madness!
You grab your sword (of course you have a sword!), trying to defend the ones you love. But you have a slight problem: the monster has many heads and, when you cut one of them, it grows back! This is a battle you can’t win.
You can see software entropy as this Hydra: it’s an enemy which will make you sweat, but it won’t be possible to get totally rid of it. You need to fight it anyway, to keep your codebase, your colleagues, and yourself, healthy and sane.
As you might not have guessed while reading this suspenseful introduction, this article is about software entropy. We will cover:
- What is software entropy and how to spot it in your code.
- The possible causes of software entropy and solutions to keep it low.
Enough mumbling: let’s dive in!
Know Your Enemy
What Means Entropy And Why Is It In My Software?
It’s sometimes useful to look to the roots of the words we use, especially when they are pretty specific, like entropy. In Greek, it means transformation, change. Something you should be accustomed to, as a software developer.
Entropy also refers to the second law of thermodynamic in Physics. It states that, for a closed, independent system, the amount of disorder doesn’t decrease overtime. It can stay stable or increase. The idea of software entropy, first coined by the book Object-Oriented Software Engineering, was influenced by the scientific definition. Basically, more a software system change, more its disorder, its entropy, increases.
We all need to understand, us, poor developers, this tragic truth. We can fight the amount of entropy in our software, but we can’t entirely remove it. Take the best team (with you in it, of course!), the best environment, the best management, the best company culture, the best project. What will happen, overtime?
- You will sing while taking your shower “this greenfield project is the best tralala” every morning.
- You and your team will add more and more stuff: features, code, packages, microservices. Since you’re all the best of the bests, everything will look fine for a while.
- After many months, nobody will want to work on the project anymore. Even if it was well-designed, the technologies used get old, the project is too big, it’s difficult to understand, ti’s time-consuming, and it’s boring. It’s too complex to have a good mental model of it.
How do you create complexity? You just need to scale your project. More elements often bring more coupling, the bedrock of complexity. I wrote already a detailed article about this idea.
If you explain that to Dave, your colleague developer, his answer will be the following:
“That’s wrong! You fool! Chaos and disorder don’t have to plague my beautiful software! My reputation will never be tarnish by some nonsense! I’m the master of my destiny! If I never modify my software, complexity will never grow, and, with this profound proof by contradiction, I declare you’re wrong and I’m the best!”.
You can explain to Dave with calm, verve, and determination, that even if you don’t modify your software, the world around it will change. If you have third party libraries which are never updated, you will have security issues. If you update them, you will open the door to changes, complexity, and entropy.
Additionally, if you need to change something in your software after enough time, you won’t remember everything about it. Even the best documentation won’t help you. The world changed, and what was true yesterday might not be so true anymore, on the technological level as well as on the business level.
These changes will introduce a lot of entropy at once.
The real question is not about removing entropy, but slowing it down. That’s the real fight we are thrown into, as developers.
Detecting Software Entropy
Software Quality
How do you know your application has a high level of entropy? Code quality is a good indicator. It begs the question: what’s software quality, and how to measure it? According to the book Accelerate: The Science Behind Devops:
- If the change / fail rate is increasing, the quality is decreasing. Track your changes, track the bugs and failures popping up, link the two. If each of your change brings more and more bugs or failures, your software entropy is increasing.
- The developers’ subjective perception on the project is important too. If everybody has the feeling it’s more and more difficult to scale the application without breaking it, the entropy is likely to be high. It has to be a shared feeling however, not only Dave, your colleague developers, complaining about complexity because spaces are used instead of tabs.
- If your team spends a high percentage of the time fixing bugs or recovering from failures, it’s likely that your software as a high level of entropy.
If you could get some data about change / fail rate and time spend to fix bugs overtime, you can come up with nice graph to convince your management that some refactoring is necessary.
The Demon of Complexity
Even if the quality of your code is at its top, complexity can ramp up easily when it comes from the features themselves. There is some necessary complexity you can’t get rid of in your code, that is the complexity which comes from the business itself.
With too much business complexity, developers will have a hard time to get a good mental model of the company’s business. Therefore, they will make mistakes while translating the business requirements in their code.
Let’s speak a bit more about business complexity: do we really need all the complex features the management often throw at their developers?
The Causes of Entropy and How to Fight It
Waterfall of Entropy
The Problem
Many seem to think that software entropy is mainly caused by the developers themselves. They are lazy, they lack skills, especially Benny, the annoying developer you work with. He’s asking so many questions!
In my experience, the main root cause of software entropy is the management, especially when you’re working in a company adept to the Waterfall Style™.
As Gerald Weinberg pointed out in The Secret of Consulting:
Even when it’s really a technical problem, it can always be traced back to management action or inaction.
I know what you might think: “You’re wrong! The management is not always responsible of everything! This is easy for developers to always blame the management!”
I don’t say that everything is the management’s fault, but they always have some degree of responsibility: a developer do a crappy work? You need to review your hiring process. The specifications were not well implemented? Maybe some people at the top didn’t know what they really wanted, therefore the specifications don’t really make sense.
That’s why being a manager is so hard! Higher you are in the hierarchy, harder it is.
How much, as a developer, do you influence the decisions? If it’s 100%, congratulation, everything is you’re fault; if it’s 0%, nothing is. Between, there is a spectrum of influence. Waterfall won’t give you a lot of it.
You’re using Scrumm, Kanban and you do planning poker? Of course, you’re not doing Waterfall! You’re doing Agile as much as a baby can build a house with a screwdriver.
Tools are fancy for sure, but they are far less important than the mindset you need to have to correctly develop software. Let me quote the Agile Manifesto:
Individuals and interactions over processes and tools
Doing Scrum doesn’t imply you’re doing “Agile”, especially if you lack the mindset the agile methodology tried to teach.
In the waterfall management, some people on the top decide, then on the layer above some other people take more decisions, and so on and so forth.
At the last levels of the hierarchy, we find the followers, the employees which are condemned to follow whatever was decided above. Their task? Producing, like a manual worker is assembling different pieces of a car on a chain. They should not think, they should produce a car.
Bad news: as a developer, you will be on the last levels of the hierarchy.
If some people on top decide about the software features and you have almost no power to bring your expertise in the decision process, welcome to the Waterfall!
Unfortunately, what the top management fails to see, most of the time, is that the production phase for a software doesn’t exist. Developers do system design, which is far from assembling a car on a chain.
Why? Because, contrary to a car, a software is always changing! You need to design it correctly to make it work, to answer the needs of your users, to have acceptable performances, and to make it scalable, robust against changes, while keeping the entropy low. To achieve all of that, you need to make many decisions how to codify the business knowledge in your code.
If you have no impact on the complexity coming directly from your managers, you will have a hard time to manage it in your code. This will be considered as “necessary” complexity: the kind which can’t be avoided.
The possible result? A software entropy level which can grow very, very quickly.
The Solution
Decisions and responsibilities need to be shared in a company, as much as possible. If you recognize your company when I describe the Waterfall model, you need to speak to your deciders:
- Bring data and strong arguments why and how X or Y feature can be simplified. Explain the possible cost now and in the future of developing the complex feature.
- Explain that the agile way of developing software means that people need to work together. Managers, designers, developers, and everybody else should be part of the decision process concerning the business requirements.
- If your management automatically reject everything you propose, you need to understand why. Ask questions.
- Speak about what deciders understand the best: money and time. Show them that having an agile mindset (and not only the tools) can save some of both.
It’s difficult however to try to solve problems your managers never asked you to solve. Many of them out there are very happy about the Waterfall method, often without even knowing that they follow it. You need a good dose of diplomacy and tact.
That being said, if you feel that too much complexity pour on your application because of the business decisions and you can’t do anything about it, even after trying, I would advise you to find another job. It’s tiring to deal with high level of entropy all day long, and a bloated product is never nice to build… or to use. This might give you a clue about the success the product will have.
A last thing: something which might look complex on the business level can be simplified if you know the business enough. This is really, really important. That’s why gathering business knowledge about your company, what they are doing, and what they want to do, business wise, will help you to find simplifications for the features and for your codebase.
As we express our design, we are explaining something to the computer. And to do that, we need to really understand what we are explaining.
Code Quality and Technical Debt
The Problem
Apart for the “necessary” complexity brought by the business (and, by extension, the messy real world), software entropy is tightly linked to the quality of your code and its technical debt, as we saw above.
What’s technical debt? Simply put, they are the shortcuts (hacks) you write to save time, knowing that it might come back to haunt you later. Especially if some other features begin to rely on it: you will have to fix it, with interests, like a real debt, in the form of difficulties and headaches.
Depending on the scope of your software, technical debt is more or less acceptable. After all, if your software has a planned lifetime of two months, you don’t have to take care of technical debt, entropy or even software quality. Just hack together some piece of code and… voila!
This can be a temporary solution before writing a more complex software from scratch, for example. It could be a prototype quickly written to show if your idea serves your target market, before rewriting everything entirely, or throwing the whole concept away.
On another side, more often than not, companies want to develop software which last longer. It makes sense: it can be very costly to rewrite a software (especially the big and complex ones), without any guarantee that its level of entropy will be lower. Rewriting is a very dangerous gamble.
In these cases, you need to be very careful with your level of technical debt and the quality of your code. It’s a big part of your job, as a developer.
The famous book The Pragmatic Programmer (the one which coined the DRY principle, among many other things) has an interesting analogy about technical debt.
The book describes a study which correlate the decay of buildings in cities with whether they have… a broken window. The broken window convey a sense of abandon, giving creative ideas to every hooligan around. In short: buildings with broken windows are vandalized faster than the one without.
A technical debt, a shortcut or a hack in your code, is a broken window. When another developer, or even your future self, will notice it, he will be tempted to add even more technical debt because “it’s already some bad code anyway, so why bother?”.
An example: you wrote a workaround for a difficult bug because you have tight deadlines. Another developer (or even your future self) might code on top of it to take care of another issue. Very quickly, you’ll have layers of workaround nobody, not even you, will understand anymore.
The Solution
First, how to know that a piece of code has technical debts?
Ask yourself:
- Can you explain simply and logically what the code is doing?
- Is the naming correct? Does it help explaining what the code is doing? What part of the business is codified?
- Is your name long, with “and”, like “deleteUserAndShutDown”? It might be a good sign you need to break your function, method or whatever construct.
- Do you feel that some code was added on top instead of modifying existing code? Does it hurts the logic and the understandability? Does it makes the system bigger and more complex?
The more your knowledge and your experience will increase, the more you will be curious and have healthy debates with other developers, the more you’ll see these technical debt patterns, these code smells.
If you find these patterns, please, don’t take it as a green light to add even more the mess:
- Increasing the technical debt will create more problems. They will come back haunting you (and your team) even harder.
- When you find some technical debt, refactor it. This is the best case scenario. If you don’t have the time or the energy, simply add a comment
//TODO: to refactor because this and that. Signal the problem, don’t let it going free and hidden in your codebase.
Don’t forget that refactoring should be done while developing other features, for these new features to fit the existing design. Similarly, fixing technical debts should not be a task by itself but an ongoing process which can be unrelated with your current task. In short, when you develop a feature, give yourself the right to fix the technical debts you encounter.
Refactor small chunks of code at a time and run your tests often, to see if your system still behave as expected
Finally, you might need good arguments when you’re refactoring something:
- First, for yourself. It’s easy to think that your coworker don’t know how to code and that you know better. But if you don’t see any benefits in refactoring, don’t do it. It might only be your ego which wants to prove that yes, you’re worth something.
- If the refactoring is about code style, it means that it hasn’t been clearly defined in your team. Set up a meeting where you will decide, altogether, what code style you want to adopt. You need to write it somewhere for everybody to have easy access to it. Modify it overtime, depending on your needs, as a team. Otherwise, your code reviews will be cluttered with “we use tabs!!! NO SPACES!!11!1!!!”. This is noise without value.
- If somebody asks you why you changed his wonderful code, you have real arguments to explain your decision.
Good arguments always have a positive impact in the future. For example, wrapping a third party library and keeping it to leak everywhere in your codebase is a good argument. If you want to know more about leaky abstractions, I wrote about the topic here.
Technical debt show the duality between the easy path and the right path. The first is fast and attractive, the second is the one which will make your project scalable and maintainable. You need some discipline to take the second one as often as possible, or your future will be full of pain and headaches.
If you feel too tired, too irritated, or if you have any other thoughts, emotions or moods decreasing your motivation to take the right path instead of the easy one, just stop coding for a while. Go for a walk, talk with your colleagues, take a break. Come back to your codebase with a fresh mind.
The most common technical debts you’ll make is while debugging. If you fix your bug by adding a workaround, you didn’t fix your bug. It’s still there, it’s simply more difficult to trigger and, incidentally, to find. You should fix your bug by changing or deleting your code. Then you need to test your code to be sure this bug will never pop up again.
A last thing: blaming your colleagues for their mistakes won’t help anybody. Talking badly or sarcastically about your codebase won’t solve your problems either; quite the contrary. Don’t destroy the spirit of your team. You need to take action to lower your project’s entropy instead of verbally trashing it.
Automated Tests
The Problem
There’s something which fascinates me in the magical software development world: I saw many people who still don’t want to write automated tests.
Why is it fascinating? Every other engineering field tests what they do, following a rigorous process. When you build a rocket, you seriously need to test many things or you might see your rocket crashing on the ground. The same goes for cars, bridges, planes, and whatnot.
It saves money to test.
It will be difficult to find managers or developers against test, but it’s easy to find some against automated tests. These folks will prefer spending time to tests manually for each change they make, instead of writing and running automated tests.
I’m no innocent here. I did exactly that, for years.
We can all agree, I hope, that computing is all about automating things for us, human, to save time and increase our efficiency. Why on earth tests should not be automated, too? It doesn’t make sense to me. Testing, like refactoring, takes some time to write (you know, like everything) but save you time, energy, and sanity on the medium and long term.
Here’s a very easy indicator to guess if a software has a high level of entropy: if it has good automated tests, the level of entropy can be low. If it doesn’t, the level of entropy is high.
Again, bigger the project is, stronger the last statement will be.
The Solution
If your colleagues or your deciders don’t want to hear about testing, you will need some strong arguments.
What about a concrete example? In a bank, 1.3 billions of customer records were lost because of insufficient testing. You don’t need to be a bank to have bugs and data corruption. You just have to say “no” to automated tests.
The argument against tests you’ll see often is related to velocity:
- “Testing slow down everybody / everything.”
- “We are a startup, we don’t have time to test.”
- “Testing is for chimpanzee. I know what I’m doing ‘cause I’ve a big brain.”
Again, even if you lose some time writing them (which I even doubt, actually), you will gain a lot of time and energy in the future:
- Tests will show you the bugs you missed. Make a list of them to show to your deciders which problems the company could have without tests.
- Everybody will be more confident to change the application. Developers will spend less time asking themselves if the change they make will break other things.
- No need to test manually everything each time you change something. Manual testing is a real, painful waste of time you should try to reduce as much as you can.
- It will show you if your code has some design problems. If your tests are hard to write because of too many dependencies, for example. In that case, fix your design!
- Tests can be useful to understand how your code should work. Think about your colleagues!
Testing will give you a feeling of control on your code and, therefore, will lower the stress of everybody.
I don’t say to test everything with unit tests. I don’t say that your test coverage should be 100%. I don’t say that unit testing is better than any other kind of test.
I’m saying that you need to test your code where it makes sense. Here’s a good talk of Sandy Metz exactly about that.
Testing when you fix bugs, for the bugs not to reappear, is useful too. You can then show the test to every test haters out there, telling them that the bug would have never existed if somebody had written a test at the first place.
Now, if your team still don’t want to write tests after all these arguments, it’s because they might not know how to do it. Instead of insulting them in your nightmares, you can:
- Do some pair programming with your colleagues, to show them why tests are useful and how easy it is to write them.
- Organizing a meeting with your team to propose some basic testing workflow you can then adapt overtime, depending on your team’s needs.
If you have beginner developers in your team, you should tech them the art of testing. I often read that testing should not be taught to beginner; this is wrong. Tests should be considered as part of your code, as part of the features you develop, not something optional, on side.
Tests are the glue which makes everything work together smoothly.
Testing leads to failure, and failure leads to understanding.
Sharing Knowledge To Avoid Mistakes
Sharing your knowledge is very important in a team:
- Constantly learning will make your work more interesting and challenging.
- It will increase your knowledge and, as a result, your value as a developer.
- If everybody shares his knowledge, nothing will be lost if one of your colleague is leaving.
This will lower the entropy of the projects you’re working on. Your whole team will be more motivated to do a good job, everybody will learn from each other to produce a better, cleaner code.
How can you succeed in this endeavor?
Pair Programming
Pair programming is an exchange. Somebody is coding and show to another developer its workflow. In exchange, the other developer point out mistakes, give advice and possible improvements.
It’s as simple as that! With interested developers, it’s a fun exercise.
Now, unfortunately, pair programming is often perceived as a waste of time.
Dave, your colleague developer, has a pretty common opinion about pair programming. “Two developers coding on two different features are faster than two developers on one feature, right? Plus, in pair programming, one developer develop, the other one is looking at a screen. What a lazy man! What a waste of time!”
You can explain to him the following: we want code quality, not code quantity. Code is not our friend; it’s a source of bug. We should fight entropy, not writing code as fast as possible.
If your deciders can’t understand that, find another job. If you don’t, you will end up with a pile of legacy applications impossible to maintain. Welcome to hell!
Code Review
Another good way to share your knowledge is via code review. It can help you:
- Teaching you better ways to handle problems.
- Teach the others what you know.
- Improve the communication in your team.
Everybody should review code and have his code reviewed. Experienced and beginner developers alike. It will allow you to catch bugs and, more importantly, increase the skills of your entire team.
You’re complaining of Arnold, the beginner developer you’re working with, because he doesn’t have any clue about the Single Responsibility Principle? His code is a mess?
Teaching is always better than blaming. Give him good advice, point out good resources, give him your knowledge and your passion.
Here are some advice how to do good code reviews:
- Try not using comments full of negation(s). “Do you think this solution is better?” is better than “Don’t do this! You fool! Here’s the good and only answer!”.
- Ask questions instead of cluttering your comments with affirmations. It shows that you are not always right, that you are doubtful like everybody, and that you don’t have a crazy sized ego. “Do you think this code could be enhanced if we do this and that?” instead of “You have to do that and this, ‘cause I’m the boss and I’m working for 289 years”.
These methods can as well create interesting discussions and healthy debates to bring even more value to your team. Obviously, debating needs to be done in a respectful and constructive manner!
Sharing Knowledge
Finally, you should share interesting articles (like this one! wink wink) to your teammates via Slack, ICQ, AIM or whatever way of communicating you’re using.
Experimenting with your team is important too. If you think a new technology could solve one of your problem, begin some pair programming and explore the possibilities.
Don’t forget, sharing is caring, and improving the overall knowledge and experience of the team you’re working with is a good way to write better and cleaner code. This will keep the level of entropy as low as possible!
Keep the Hydra of Entropy Far Away From Your Software!
For those who still think that laziness is the first cause of software entropy, I would argue that it’s never the root cause. Why are your colleagues lazy? Is it due to a bad hiring process? Are they too stressed because of short deadlines? Do they find their job fulfilling?
Speak with your team. Ask them if they like to work with you. Be genuinely interested by who they are, as human beings. Listen to what they have to say, to their feedbacks, even if you don’t share their opinions.
You will need a good dose of empathy, honesty, and open communication to work effectively with people. Again, blaming or criticizing won’t bring anything to anybody. Pointing out people as lazy is blaming.
Alright. What did we learn with this article?
- Entropy is the level of disorder in your system. It’s impossible to lower it. Keeping it stables needs actions more than complaints.
- Code quality decreasing is a good indicator that the entropy is increasing.
- Developers are not the only one responsible for a high level of entropy; complex and/or unclear features can bring the entropy level very high too.
- Code quality needs to be actively take care of, by pointing out the technical debts, or, even better, fixing them.
- Automatic testing is a no-brainer if you want to decrease the codebase’s level of entropy. Actually, a feature should only be considered done when it has automated tests.
- Try to create an environment with your colleagues where the knowledge is easily shared, via pair programming, code reviews or simply sharing interesting resources (articles, books…).
At the end, we must accept that the perfect software doesn’t exist; software entropy will be always part of our work. It doesn’t mean that it’s a fight we already lost. It means that lowering entropy anywhere in our codebases is already a victory!
I covered a lot of ground with this article. Don’t hesitate to let a comment if you want a deep dive in one of the topic I speak about here.