Complexity Killed the Design

Software exists to solve problems, some of these problems are born from tedium, a need for speed or a requirement for computation.

Something all these problems share is a degree of complexity both in understanding there nature and in formulating a solution.

In 1986 Fred Brooks wrote a paper entitled "No Silver Bullet — Essence and Accidents of Software Engineering". In it he argues that complexity falls into two distinct categories, essential and accidental.

Essential complexity is caused simply by the fact that the problem at hand is complex, accidental complexity is caused by software engineers themselves in trying to devise an answer to the problem.

Complexity in any design activity should be avoided so if were going to produce good software we need to have strategies for reducing both of these potential sources.

Minimum Required Complexity

Practitioners of agile are familiar with the concept of a Minimum Viable Product (MVP), a product that balances return versus risk, getting us to market quickly to start getting feedback from users.

An engineering benefit of having an MVP is that it reduces the scale of the problem were trying to solve, this in turn will reduce the amount of essential complexity inherent in trying to solve it.

A project burdened with too much essential complexity can very quickly become stagnant, struggling to get off the drawing board or floundering trying to produce a trouble free product that works.

A well thought through MVP will also focus the team on solving the right complexity, it may be that a certain feature would be great for users to have but will derail the team from solving the complexity that will actually make the product viable.

Don't Design By Accident

It may well be impossible for us to eliminate the production of accidental complexity. Software engineering is a difficult activity, mistakes will be made, sub-optimal choices will be chosen.

The best element of accidental complexity is that we have it within our power to fix it.  

A phrase I hate to hear developers utter when answering questions about something they've written is "but it works".

The essence of good software engineering is not that something works, even badly written software normally works, its in how it works. This should lead us to rank working alongside, or maybe even underneath, qualities such as efficiency, scalability, testability and maintainability.

Never stop when something works for the first time, instead ask yourself is there a better move, can this be done better. Its crucial at this stage to know that by better we don't mean by adding new functionality, instead we mean can I achieve the same functionality and increase any of the qualities we've just outlined.

Good design signals intent, it is never the result of an accident. You should never be happy with code that works that you or others don't understand, this means your unlikely to be able to test it effectively or maintain it when new functionality is required or God forbid it ever stops working.

As software engineers we are essentially employed to deal with complexity, we need to make sure we do this in a fashion that ensures there is less of it when we've completed an iteration of a product then when we started.

This applies not only to the user but to the engineers that will come after us, we shouldn't be trying to impress them with our grasp of complicated principles and practices applied to situations where they really weren't required.

Instead we want those engineers to look at our code, nod knowingly, and say "I understand whats happening here".   

Extreme Delivery

Many of the principles and practices that we now consider to be part of agile development were first presented in 1996 by Kent Beck under the name Extreme Programming (XP).

Many of these best practices were already known, the view that Beck put forward was that these things were so beneficial there use should be taken to extreme levels.

If doing a little of something yields some benefit then doing a lot of it should lead to large benefits. 

Extreme Activities

The approach to XP can be divided into four categories, coding testing, listening and designing.

The goal of software engineering to provide code that is a solution to a problem, therefore working code is the most important output of the team, all other aspects of how the team works should be aiming to ensure this code is efficiently produced and is the solution the customer needs.

The key description of the code is working, this can only be determined by testing. We take testing to extremes in terms quantity, frequency and tenacity in ensuring all errors are dealt with. We realise that the only way to achieve this is via automation.

So now we have an efficient pipeline for producing working code we need to ensure this code fixes the correct problem. We do this by listening to users, whether existing or potential, we realise that they are the only ones that define the problem that they need solving and only the will ultimately rubber stamp our solution as effective.

The transition between the knowledge we gain from listening to the output of working code is delivered via design. We do value design, we also realise that a law of diminishing returns applies to its effectiveness.

Extreme Design

The goal of our extreme design is not to formulate a solution to the whole problem, it is instead to provide an effective solution to a certain scaled down version of the problem with enough flexibility to provide a roadmap for fixing the bigger versions of the problem to come.

In the design of our code we value communication, simplicity and feedback.

The most effective form of communication between developers is code, it breaks down barriers and communicates ideas and principles. But it can only do this when its written to be clear and concise, we design our code to communicate its purpose via skim reading not via having to tackle a large unwieldy novel.

Problems in software are always related to complexity, sometimes this complexity is essential, the problem we are trying to solve is by its nature complex. But when we have it within our control we should prefer simplicity in solving the problem at hand at not choose to introduce complexity to solve problems that we don't yet know need solutions.

This concentration on simplicity means we must have effective channels for feedback, we will need to gradually increase the level of complexity to continue to offer value to the user. We also must design our code such that it can provide feedback on its current state of health, we do this by making our code testable and capable of testing itself. We aren't satisfied that the solution appears to work we want to ensure that under the hood it isn't by luck or magic but by design.

Extreme Delivery

All of these extreme practices are aimed towards delivering on promises. But this extreme approach is related to the speed and frequency at which we deliver, we aren't attempting to solve an extremely large problem in one fell swoop, we are attempting to solve an ever increasing number of small problems at an extreme rate.

This is only achieved via flexibility, our code must be designed to expect to be continually re-factored, re-purposed and re-used.

We also need to be flexible in our approach to our users, feedback from them may not always be coherent or easy to work back into the product. We need to realise the responsibility for this lies with us and not the user, it is our job to coax out of them what problem they need solving not to tell them what will make there lives better.

As with most things in life problems can occur when things are taken to extremes, but the majority of teams are nowhere near this barrier and will nearly always have something to gain from trying to turn the dial up to 11.  

The Cathedral and the Bazaar

In 1997 Eric S. Raymond wrote an essay (later turned into a book) entitled "The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary".

In the essay Raymond compares and contrasts two different open source models and proposes the idea that one of the reasons open source is a beneficial activity is because "given enough eyeballs, all bugs are shallow".

Also presented are 19 lessons gained from various experiences in software development about good practices in software development.

I'm not going to go through all 19 lessons but I have picked out a small number that particularly resonant with me.

Good programmers know what to write. Great ones know what to rewrite (and reuse)

When we first become developers (I'm assuming that anyone taking the time to read this is a good one) we want to write code to solve every problem were presented with.

While this enthusiasm for plying our trade never leaves us experienced developers do develop a healthy lazy-ness for not wanting to write code to solve problems that were very effectively solved long ago.

This laziness will give rise to two effective instincts, recognising when someone else must have already written code to do this, and recognising when code your writing has the potential to be applicable in many situations.

To paraphrase a famous writer "Talent imitates, Genius Steals".

Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away

The first solution we provide to a problem will almost always be comprised of more code than is necessary to achieve the result.

Because of this we should not stop thinking about the problem when we arrive at this first solution, instead we should set ourselves the goal of trying to get to the same outcome with less code.

There is obviously a balance to be drawn, developers are often obsessive people who could very easily never be happy to say something is finished. 

We don't necessarily have to achieve the optimal solution first time round, what we do have to realise is that if were adding more code we are adding more potential defects, we are probably adding more features not refining the ones we already have.

The goal of re-factoring should always be to end up with less code than when we started, the solution should be come more readable, more understandable and simpler.

Any tool should be useful in the expected way, but a truly great tool lends itself to uses you never expected

If code is going to be re-usable it first of all must solve a problem that exists elsewhere in the code base. Secondly it must be written in such a way as to make it loosely coupled from the code it is serving and solving the problem for.

In many ways software development is a repetitive activity, many projects your involved in will require solutions to the same or similar problems to the ones you've faced in the past.

Its also true that its not always easy to predict when your faced with a problem whether or not its the kind of problem you may face again in the future.

If you write loosely coupled cohesive code I guarantee that you will end up using this code in a situation you couldn't have predicated at the time when you wrote it.

Many people will be amazed when an experienced developer quickly provides a solution for a new feature, is it that this developer is able to write code at a fantastic rate? Or is it another example of healthy laziness where instead this developer is reaching into the toolkit of code he or she has amassed over the years?

All of these points are pointing towards the fact that as developers our job isn't to just produce more and more code. We are not in an industry where producing more and more of our raw product is the aim of the game.

We are actually employed because were smart people who can solve problems, and the best solutions to those problems are born from a desire to not have to write so much code.

Be smart, be lazy.   

Dealing with Defects

Simplicity is a beautiful quality for code to have but its very difficult to avoid complexity entirely.

Because of this software has had, and always will have defects. As an engineer you don't need to beat yourself up about this, you have a difficult job and occasionally mistakes will be made.

So given that even if were striving for perfection we won't achieve it we need to know how to deal with defects when they inevitably are uncovered.

When Is A Defect Not A Defect

Preceding a defect being unearthed there will usually be someone heard to say "Its not supposed to do that", but we shouldn't always assume that every time unplanned or undesired behaviour is observed that this is the result of a defect.

The surfacing of technical debt will very often look like a defect but its important to recognise when this isn't simply a mistake but part of the design of the system.

These situations need to be given special importance, quite often but not always defects are static in nature in the sense that there impact on the code base and the user doesn't change.

Technical debt is dynamic in nature, it generally gets worse over time both for the user and the engineer.

This dynamism will usually also apply to the solution, once you start to feel the affect of technical debt it must be nipped in the bud, it isn't just a mistake its in-built in your code.

Not All Defects Are Equal

Defects will be uncovered in a random order, you won't necessarily find the worst ones first and there isn't always a correlation with the impact of a defect and the amount of time its going to take to fix.

All this means you need a triaging process, the chances are you aren't going to have the time to ship defect free so you need to make sure your making the best use of your time and concentrating on what will most affect the user.

What annoys the user and what annoys the engineer are not always the same, the engineer might be very annoyed by a slight glitch in the UI because he or she knows it it could be better, the user may be blissfully unaware of this potential enhancement.

Once you've accepted your not going to be able to fix everything you need to resist the urge to jump on a defect as soon as its discovered and instead document it and regularly review and asses which will have the biggest impact on the user.

The user will probably put up with or be unaware that the button isn't quite rendering properly, what they won't forgive is nothing happening when they press it.

Attacking with the Toolbox

Because on the whole defects are caused by people making honest mistakes they will very often fall into the similar categories. This similarity can help us first of all detect them but also formulate a plan of attack to fix them.

Before any effort can be made to fix a defect our energies should first be concentrated on finding the correct way to reproduce it.

A defect that can't be reproduced is difficult to repair and even harder to verify as fixed.

Once you've been burnt by the defect fixing process a few times learn to trust the intuition it will nurture, trust that voice in your heard that says "I bet this is because..." and recognise the patterns that certain defects produce in code.

First and foremost we want to fix the defect so we can ship but we also need to give some though to ensuring that we don't have to spend anytime fixing it again in the future. Always give some though to how this could be prevented from happening in the future and share this knowledge with everyone in the team.

Its pointless to try and pretend that you can get to a level where your code won't have defects, no-one is perfect and mistakes will always be made, what you can strive for is developing a strategy for dealing with defect to ensure that there impact is minimal both on you and your users.   

Engineering without Fear

I am a believer that there is a lot to gain from viewing software development as an engineering discipline.

We can undoubtedly apply principles such as SOLID to contrive and devise systems from our raw building material of code.

But having worked in other forms of engineering I'm also aware that there are differences between the processes and patterns we apply in software engineering and our brethren practicing in other industries.

Far from these differences detracting from the validity of software development as engineering we should embrace these differences, indeed they are the reason I decided to make the switch and become a coder.

Price of Mistakes

No-body wants there software to have defects and we put a lot of effort into the prevention and eradication of them from our code.

But actually compared to other forms of engineering we pay a relatively low price for these mistakes at least in the development phase if not in production. 

Some defects can be fixed in minutes, more persistent ones might take a day, but imagine the life of an electronics engineer where defects might take weeks to fix as PCBs have to be re-laid out, fabricated and populated with components.

In some forms of engineering defects simply cannot be tolerated at all, a structural engineer will be in much more trouble if a mistake causes a bridge to collapse then we might be in if an app crashes.

Our relatively rapid fix and deploy cycle mean we have a much greater capacity to improve our output and drive increases in quality, this coupled with out ability to update software for our users means we are in a unique position to improve things for them on a continual basis.

Speed of Production

We all spend a significant amount of time thinking about and building code, if we work on large code bases it may take several minutes for us to compile and link all our code into an app or web site.

In most other forms of engineering it isn't possible for an engineer to see the fruits of his or hers labour so quickly.

The pace at which we can build and produce software is actually comparably high, when a decent coder gets into a groove significant progress can be shown over the course of a day or two and not just plans or simulations but actual working code.

Creative Thinking

Software development is a very creative exercise.

We allow users to directly interact with what we produce in the form of a UI, whilst also getting to shape this contact via careful application of UX.

This allows us to not just be cranking the handle and churning out code but indulge our more artistic side and build something captivating and engaging.

This very direct contact with our users is not something all engineers have the pleasure of experiencing.

Let Go of Your Fear

So reading this post you may be getting the impression that as software engineers we have any easy life, obviously this isn't true.

Writing software can be a very frustrating as well as rewarding experience, it requires intelligence and a dedication in applying your talent.

What we should take away from these differences is that as software engineers we shouldn't be afraid to try things.

Software development on both a personal and professional level is elevated and progressed by people having new ideas, the speed at which we are able to try these ideas out and put them into practice gives a unique opportunity to advance our industry.

Not every idea you have will be a winner, some may even turn out to be dumb but this should never stop you from trying.

The worst that might happen is you've spent a day or so finding out that this isn't quite going to work as you hoped, but knowing what doesn't work can be just as informative as knowing what does work.

But when you do have a winning idea, that time spent implementing it will not only improve both you and your team as engineers but also your users lives that benefit from your engineering skill and your willingness to try something new.