Why do projects always end up with tech debt?

As a developer, I have both explained and heard others explain to less technical people about how the team need to slow down adding new features/functionalities and take care of some tech debt first. Often, the management asks why. Sometimes it is hard to communicate the idea across to some people. In fact, it is quite nature for tech debt to occur and it is no one's fault that this happens. It has always bothered me when tech debt become a point of tension between developers and management. Because I think it happens so often that everyone, developers and management, should understand why projects end up with tech debts, what they are and how to manage them.

Tech debts is always talked about in development teams. It is a common questions in job interviews and it is often the hot topic of debate in team discussion. It is always hard to decide if time should be spend on building new features or reducing the amount of tech debts within a given project.

When a team gets stuck in a loop of "why should we reduce tech debts" and "why shouldn't we", it is extremely frustrating for everyone. This has always bothered me, because I don't think tech debt should become a point of tension in any project. It is completely natural to end up with tech debts in any project, as long as everyone has a good understanding of it, what matters the most is how to address it. I'm not saying this kind of conversation isn't health, in fact, quite the opposite it is very health for the Product Owner (PO) to challenge the devs and vice versa. What isn't healthy to going around in circles, and debating about it week after week.

This blog will explain what tech debts are from a developer's perspective since it is a technical problem at the end of the day. But I'll also explain why it might also be useful for devs to bear in mind of some the business concerns too instead of just pushing for reducing tech debts.

Why do we have tech debts?

Let's use library as an example, to illustrate why projects might end up tech debts.

Imagine there's are a librarian, who works hard to keep 1,000 books tidy and organised in the right order on all of the book shelves. The above is what maintaining a project looks like. Sounds easy enough right?

Now, let's say customers comes to the library everyday and sometimes books gets misplaced, books can also be borrowed and returned. On top of all that, each week 100 new books are also added to the library. That is a bit more work now right? As time goes on, it gets increasingly more challenging to achieve the original goal of keeping books tidy and organised. Simply because of how many books there are in the library. This is how it feels for a developer when they have to deliver value in an agile methodology such as scrum. There are a lot to balance out, e.g. time vs quality, off-the-shelf approach or use own implementation, performance, security and etc.

As the number of books increase, sometimes books gets misplaced by customers but the librarian has too much to do and doesn't notice it. Books might be lost because something went wrong with borrowing or returning processes. Librarian might not have enough time to shelf all the books properly, and they just do their best and only put the books in the rough areas. All the these problems can occur in a project. As the project grows, same "amount" of work will take longer because there are more things to consider when delivering a new feature. It is like playing a game of Jenga, even with the same number of pieces, as the game goes on it comes harder to stay in the game. Not just because there are more variables to consider, but all the previous actions has a massive impact on the future actions, as well as limits what can and cannot be done.

In some projects, it may have been worse. Assume, now the CEO of company decided they want to track the performance of each library by measure how many new books they can add on to the shelf each week. It seems crazy and illogic here right? But it does happen and maybe happens too often. This forces libraries to import more and more books to add on to the shelfs, and has a knock on effect on all of the problems mentioned above. This is why scrum recommends teams to maintain a constant velocity instead of strike for a higher number each sprint.

What are the tech debts and how to address them?

Tech debts comes in all sorts of shapes and forms. Here are just some of the common tech debts that I have seen first hand.

Duck tapped business logic

Now, we all work using some kind of agile methodology to ensure the product development can quickly pivot if and when the business requirement changes. This means requirements are not fed through until when they need to be pulled in. This means development teams are often unaware of business logics that's further than a few sprints away. This is where people having to build on top of any existing business logic. The lack of big picture at a given sprint and the fact these logic could have been the worked on by different people means that, the end result might look messy and is difficult to understand. Naturally, it will make any future work in this area more difficult, because people need to first understand what is going on before implementing anything new. In bad code bases, it can feel like walking around in a massive maze, where many routes will end up to be an dead end. It's a very time consuming and frustrating process.

Not mentioning the fact, unless the existing code is fully understood, it is very easy to introduce new bugs. Unfortunately, the areas where this kind of duck tapping happens is also where the core business logic sits. So this could be financial calculations, complex user journey mapper and etc. Basically, things you don't want tripping over on production environment. This is why it is worth tackling them ASAP.

When dealing with these issues, the core focus is to rewrite the code so that the logic is easier to understand. This kind of actions could massively speed up future development, because rather than looking at something like this:

face North
turn Left
turn Left
face South
turn Right

It will be as simple as:

face West

Pretty straight forward to see which is easier to understand.

Code cleaning

Code cleaning doesn't sounds like a necessity, but it can be hugely helpful. Unclean code makes it difficult to read and understand existing code base. Often a small amount of messy code and spiral out of control very quickly as the team loses grip on the "right way" of doing things.

The common tasks of code cleaning includes:

  • Remove redundant code
  • Remove badly formatted/organised code
  • Remove inconsistent coding styles
  • Adopting better coding practices

All of the above should be fairly simple to understand. If you are still unclear, then think of it like tidying up a box of tens of thousands of Lego brick. The best way is to separate each type into its own compartment, then organise them in a logical way.

Performance improvements

When features are implementing quickly, most of the time it is probably NOT done in the most efficient way. Going back to previous point about the lack of big picture, which can definitely be challenging on one person is focusing on a very small part of a product or system. It might be that one aspect of the performance was missed during development. It could even be that, a while later someone has found a better way of doing the same thing.

Depending on the situation, the performance improvements will be different. But a simple example might be that, instead of serving a default sized high quality images on a website, the team could export different sizes of the same image, so we can server different image sizes depending on the user's internet connection and device.

Other tech debts

Improving security: this is probably a bit rare for a product, since most likely everything has been thought of at the early stages of product live cycle. Since nowadays most products uses open source software and the open source nature means it is easy for hackers to find security flaws. Thus it is important for the team to review the libraries and frameworks used constantly, and update to the latest version ASAP when bugs/security flaws has been revealed.

Improving SEO performance: overtime changes made to the code base might have negatively impacted the website/app ranking on search engines.

Improving accessibility: accessibility is often missed by development teams, but we should aim to build products that can be used easily by everyone.

Better practices: as a team and product grows, it should be encouraged for the team to explore for better tools and practices to improve the ways of working, code quality, efficiency and etc.

Why might teams want to delay fixing tech debts?

For the business, tech debts is this vague and mysterious work that only the developers knows about. It is not hard to understand why management gets worried when the development teams says they need a month to work on tech debts without much justification. Unless people are being difficult and is being unreasonable, as long as they get the justification the management needs there is no reason why they would say no.

Understand priorities within the project

Tech debts is horrible, and teams should aim to reduce it as soon as possible. But it is worth to be mindful about the priorities in the current project. For example, any of the following would be good reasons to postpone tech debt for while, unless the tech debts is a massive blocker when it comes to delivering those goals.

  • A big release is about to go out
  • High priority feature addition/changes
  • High priority bug fixes

Basically, use the best judgement to decided if it is the most appropriate time to be working on tech debts. For example, between releasing the next version of the product which contain important bug fixes and fixing some code formatting, it would be wise to focus on the release first then come back to the code format issues post release.

Summary

Hope this blog post has helped to clear up any questions about tech debts, and in some cases how to fix them as well. If you have any question feel free to get in touch!

Buy Me A Coffee