Why do software projects fail?

...and with "fail" I mean "not perform as well as they could".

I think we can split the fails in three different categories.

1. Management problem
2. Engineering problem
3. Culture problem

It's hard to say if one is bigger than the others or if one is more common than the rest.

Lack of vision and power to execute

Some years back Finland's tallest apartment building was built in my neighbourhood. (It's not tall at all compared to any tall building in any major city.) The rendering on left is the vision that won the architectural competition and next to it is the actual result.

The vision is not that great, there's nothing special about it but the result is way worse. The same thing happens with software all the time. We start from mediocre idea and end up with horrible product.

Bad start

Many if not most products are not made because someone had an idea and really wanted to see that happen. Instead they are made because someone thought that someone else might need it. Market research tells that customers want something or everybody else has it. So let's do it.

What then happens is the next manager in line is picked to be in charge of the project. That person does not love the product. He will not be passionate about it.

When the product owner does not feel strongly about the product, he will not have the energy to demand excellent quality. If the team implements the ideas poorly, he will OK it. If long term goals threaten short term goals (and the next bonus incentive), he will go with quick and dirty.

When this happens the result will have all or most of the functionalities that were planned, but everything is bit off. Nothing feels right. Nobody loves the product. These are the Nokia phones and Windows Vistas. Nobody loves to use them.

Weak managers

Software development, especially at the management level, contains a lot of politics. Sometimes upper management will not let product owners to hunt their dreams.

It is really common to not let newer product to eat the market share of older one. Upper management does not accept market cannibalization. When this goes long enough, all we have is crippled products.

And it is not always upper management who cripples new products. Sometimes it is other product owners. If it looks like the new product will eat some sales from the existing ones, the product owners of existing products will fight back. That behaviour is built in their incentives; sales go down, bonuses go down.

Weak product owner = weak product

If PO does not feel strongly about the product, it will be weak. If PO does not have the corporate muscles to hunt for the dream, the result will be weak. If PO is weak against other PO's, the result will be weak.

Bad, incomplete and unsuitable code is knowingly released

if ((err = SSLFreeBuffer(&hashCtx)) != 0)
    goto fail;
if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)
    goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)
    goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
    goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
    goto fail;
    goto fail; /* THIS LINE SHOULD NOT EXIST */
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
    goto fail;

The code above is real code (except for the comment). It's the reason for infamous Apple's "goto fail" bug. Because of it all iOS devices had vulnerabilities in their SSL implementation.

The thing about it is that the style of the code is just bad. It's bad in such way that any static analysis tool would have caught it. Any code review process should have caught it.

I've never seen a programming style guide that allows if statements without enclosing the block with curly brackets or what ever the language uses. Leaving those characters out is prohibited precisely because without those these kind of bugs will happen. And it is dead simple to enforce.

Developers do not care

It is really common not to have any static analysis tool in use and it is also really common not to have any code review process. The lack of any sanity and quality checks at the code quality level means that we happily assume that every member of the team produces quality code all the time. That assumption is of course insane.

In practice most teams have one developer who can not actually program. It is really common to have one person in team who can not solve problems by programming and not using Google all the time. Eventually they will come up with some solution but personally I consider that they can not produce code.

And even if all team members could actually program, we all have bad days, weak spots in our knowledge and even the best ones make mistakes.

Not using static analysis tools and not doing code reviews means that the only quality checks are at the product quality level done by test engineers. The worst spaghetti code with misleading comments and insane API decisions can pass all the unit, integration and system tests. But the code quality is still awful.

Everybody wants to be the nice guy

Even if we do have code review process in place, we still let horrible code go by. We do this because we are humans, too.

Usually most of the mistakes, bad decisions and misunderstandings are made by the same person. The one who can not really write code without Google. If we correct him every time he makes a mistake, even the smallest one, we will look like bullies. At least if we do not communicate clearly and nicely. Doing that every day all day is really hard. So we loosen up our standards. We let bad code go by.

Even if everyone in our team is a good programmer, there is always the most experienced one. If he keeps the bar for others as high as he keeps it for himself, all he can do is spot errors in others designs and algorithms. So he lets less than ideal code go by. And the same goes for the rest of us also. Usually we keep higher standards for ourselves and let others' bad code go by.

We do this because demanding more from our peers is wearing. We feel like it is not our task to demand from others. But it is. There isn't anybody else who has the knowledge to see what is bad code and what is not.

Prototypes as product 1.0

If implementing some new feature takes too long, sometimes managers asks us how long it would take to make a prototype, proof of concept or something similar. That takes way less because we get to cut all the corners. We skip all validations, we write huge functions and fake API's and what-not.

Then we demo our prototype when the sprint is over and suddenly the prototype is in the release branch because it seems to work and the managers wants it there. And nobody says a thing. Nobody says a thing because it is nice to get things done. It's history, no need to worry about that thing any more.

We need to realize that prototype code needs to be thrown away and we need to make others know that, too.

Anything but open

Windows 8 is what happens when you do not listen what your customers say. You end up making a product that the market does not want or need. No matter how well it is made, it is just wrong product.

Not listening

Most teams do not make it easy to try their product and give feedback about it.

Writing installation and configuration instructions is almost never considered its own task. Which results in bad instructions so nobody outside the development team can try the product. Release notes are either missing or way too technical (git log or similar). So even if someone did manage to install the product, they have no clue what's new in this release so they can't focus their testing efforts.

And when was the last time you created a separate feedback channel for dogfooders and others who might try out the product but are not in the team? Yup, it almost never happens. The only way to give feedback is to file a bug or talk to the team members. Some people do not have access to the bug tracker, using bug trackers is not natural to all folks and sometimes the problem is not specific at all, the product just feels wrong, but nobody wants to file bugs like that. And the team might be located in different continent on a different time zone and not everybody knows them.

Not asking

Most teams do not actively ask for others opinion.

Pretty much everybody agrees that hallway usability testing is cheap and effective. Still it is really rare to see it happen. Demo days are too often held only with team members. Which means new ideas are really uncommon. And when somebody talks about dogfooding, we nod agreeingly, but then nothing happens.

Even though distributing software costs almost nothing, the development is usually done in isolation. When developing physical products it is understandable that due to the cost not everybody in the company can be a dogfooder, beta tester or something similar. But distributing software is pretty much free. Still non-team members are not used as a test crowd.

Summary

We got mediocre ideas, weak leaders, developers who do not care enough and for some reason we think that we do not need to ask help or advice.