User Trials and Errors

The lives of a software engineering team would be made a lot easier if they were always feed with exact requirements from the people who will use their software that if delivered would guarantee success.

There are many areas of software development that make it an inexact science, one of these is how to identify and meet users wants and needs. Certain aspects are obvious, no users want software to crash or error, they want software to be secure, reliable and fulfil its main purpose. Beyond this which features users want to see and how they want these feature to work is difficult to predict.

There are certain techniques that we can use to try and smooth this journey towards having happy and content users, but the premise behind adopting these strategies needs to be that we don't have the power to simply predict the right path.

There is no crystal ball, only controlled trial and error.

Analysing Failure

The majority of modern software applications will employ some form of analytics, collecting data about how software is being used for future analysis. Unfortunately in almost equal proportion this task is approached with a certain bias.

Frequently we decide on the data points we will collect with a view, either conscious or subconscious, to prove we are right in our assumptions. We look through a lens focused on how we expect and\or want people to use our software.

This will very often lead to a situation where when the data doesn't prove conclusive we are hampered in further analysis because we aren't collecting the data that would unlock our understanding.

When deciding what data to collect a general rule should be to record everything. Obviously this needs to consider user privacy, anonymity and consent. In this context we are talking about recording how users interacted with the software not their personal details or data.

We also need to set expectations on what our analysis will reveal, analytics will generally raise questions rather than answers. It will point out where we are failing but not the solution. However the value of these observations is that they aren't guesses, they are facts based on real data relating to real users.

Open for Criticism 

Research with users is also common practice with many teams, however this can be equally tainted with the desire to be proven right. We can lean towards wanting to show users our new ideas and gain acceptance for them, however this can be undermined by the natural human reaction to being asked if you want more.

User are often likely to be positive towards the possibility of new features but this doesn't mean the current lack of this feature is their number one gripe with the software.

This kind of research can be much more useful if we are prepared to open ourselves up for a bit of criticism. The insights we can gain from effective analytics will have already highlighted problem areas, engaging with users to ask them exactly why these problems present themselves can yield further insight into what is wrong.

Openly asking people to criticise you is not an easy step to take, and sometime this criticism may not be presented in a useful or constructive manner but if you engage and try and structure the conversation you will learn a lot about the problems that if you can solve will make a big difference to users views on your software.

Trial and Error

A theme that has been running through this post is that you can't necessarily trust what users say or what they do in isolation if it's under controlled circumstances. The only real source of truth is what they do in large numbers when they are using your software in the real world.

This affects not only your analysis of issues but also the effectiveness of your solutions, nothing is certain until you've tried it with real users in real situations. This implies that you have to take a big leap into the unknown whenever you think you have something to improve your users experience but thankfully tools exists to allow a more experimental approach.

By applying A\B testing you can trial your ideas either on a random or targeted proportion of your user base and allow analytics to determine if it has changed user behaviour or has influenced outcomes. This enables you to perform trails on real users in real situations whilst also reducing the impact when you are inevitable on occasion proved wrong.

If we except that perfecting features first time every time is not a realistic ambition, adopting a strategy of trialing data driven experiments in the wild is the next best option.

Much of adopting an agile mindset is in accepting a lack of control over the process of delivering effective software. The techniques it promotes are geared around making the best of the situation given that limitation. Many that want to be agile struggle with letting go of the illusion of control, it isn't always a comfortable situation to be in but that doesn't make any the less true.

Learning to use analytics and techniques like A\B testing will help with the acceptance of this situation and you can even learn to love the excitement of whether this idea will be proven to be the one that makes a big difference, just don't lose heart when it turns out it isn't quite there yet.

Iterate to Accumulate

When assessing your agile maturity it can be easy to fall into the trap of only considering your adherence to the ceremonies and procedures of the particular flavour that you've decided to adopt. Whilst this is obviously important to ensure your working within a framework that provides structure to your teams work, it doesn't provide a measure of how much you've adopted agile values

Describing the values that define agile is a potentially lengthy discussion and I won't attempt to cover them all in a single post, however one value that I think is often overlooked is the need for an iterative approach to all aspects of the delivery lifecycle.

Again the importance of an iterative approach is not born from the process itself but from an understanding of why it represents a better way to solve problems than its waterfall counterpart.

Iterate Understanding

If we define an iteration to be a sequence of requirement gathering followed by execution that is repeated multiple time then we can see that each iteration represents an opportunity to increase our understanding of the problem, possible solutions and the merits of each.

Sometimes an iteration will not make it to end users, we will realise before this that something is wrong with our approach and decide to re-think, or it maybe that we need to go further in what we are trying to deliver. Sometimes it will reach users and based on their feedback or our use of analytics we will realise we can improve. It is also possible that the data derived from users using the software in the wild will indicate a diminished return in iterating further as what we've delivered is achieving its stated aim. 

In all these scenarios we have given ourselves the opportunity to benefit from feedback loops that give us the time and freedom to re-think while the iron is hot. A waterfall approach robs us of that opportunity by increasing the price we pay for mistakes in our understanding, the longer the time period between iterations the less likely you are to be able to quickly act on new information and the more reliant you are on having mastery over all the moves your users could make. If this kind of mastery was possible software development would be a very easy activity and a solved problem.

Iterate Solution

The importance of all the SOLID development principles comes from how they influence the approach to change in a code base. A natural driver to embrace these principles is likely to be fostered within a team that expects to iterate.

An iterative approach will fail if the execution phase always represents large refactoring or is weighed down by technical debt. A reluctance to end up in this situation along with a desire to enable the accumulation of understanding that we've previously discussed will create an opportunity for an architecture to organically grow in the right direction.

A need to ship frequently will also promote the adoption of other methodologies to promote delivery such as increased test automation and effective continuous delivery pipelines. 

Software development teams by their nature are resourceful people who have the skills to be able to shape their world in the way they want to work. Promoting the right values to these teams will nudge them towards developing software in a certain way, an emphasis on being able to effectively and efficiently implement multiple iterations of change will be the mother of invention to bring new ways to shorten the feedback loop.

Iterate Failure

A fundamental lesson in agile development is to learn that you will encounter failure, rather than exhaust energy in trying to avoid the inevitable, failure should be embraced as just another feedback loop.

Failure in this context is not bugs and defects, albeit they are equally a fact of life in software development, failure in this context is misjudging the needs of your users or the effectiveness of your solution in meeting those needs.

In this sense failure isn't something that can be, or indeed should be, hidden in the dark. It should happen in full view worts and all, this is only way that you will get real feedback on what users think of your software, how they use it and what its shortcomings might be. As much as we may think we can head-off this failure by talking to users or asking them what they want and need nothing speaks the truth like putting working software in the hands of large numbers of people to use in their everyday lives.

Techniques to limit the impact of this failure do exist, by using techniques like A/B testing or phased releases we can observe interaction with the software in a targeted way with an ever increasing audience size. 

Only the most trivial problems in software engineering have ever been solved in a one and done manner. Perfection in any feature is probably unobtainable but by paying attention to how your users are interacting with your deliveries you will both identify areas for improvement as well as the time when continued effort is unlikely to bring a proportional increase in user satisfaction.

Agile's promotion of tight and frequent feedback loops provided by continuous releases places this lesson at its heart.  

Pop Goes the Tests

Behaviour Driven Development (BDD) is about so much more than automated tests but the reason these things are so frequently conflated is because an effective implementation of BDD principles presents the opportunity to write tests that proves your systems behaviour.

A frequent mistake when implementing these automated tests is to concentrate too much on the presented User Interface (UI) over the actual behaviour of the software. The purpose of these tests first and foremost should be to validate that things work, that isn't to say that defects in the UI aren't important but they are likely to be less impactful than broken functionality.

Alongside this a frequent criticism you will hear from teams that implemented automated tests is that they are flaky or can't be relied upon, if when tests fail the issue can be with the tests themselves then there becomes little point in running them. The source of this fragility again is often an overemphasis on the UI when implementing the tests.

UI is often an area of frequent change and tests that are too closely tied to a particular implementation of look and feel can become brittle. The Page Object Pattern (POP) is a technique for implementing automated tests that can address this fragility by encapsulating UI details into a single place whilst leaving the rest of the elements that make up the tests to concentrate on function using a business domain language.

Feature Files

Although not expressly part of POP the approach can be undermined when feature files contain too many UI related details. The language used in feature files should reflect the operations users want to complete using concise phrases that both business stakeholders and users would likely use to describe what they are doing.

As an example two different ways of describing a login process might be:

"I'm entering my username into the first text box, I'm entering my password into the second text box, I'm clicking the terms and conditions checkbox, I'm clicking the login button" 

"I'm logging into the site"

The first describes in great detail the interaction with the various UI components on the screen as well as the details of what is happening under the hood, the second is a much more concise explanation of the process the user is trying to complete. The first would need to be changed should the implementation of the login process change, the second would be unlikely to change unless there is a significant change in the flow of the site.

Those that are familiar with the site have an implicit understanding of how the second description is realised and it therefore isn't necessary to go beyond this level of detail.

Page Object Models

Keeping the feature file language simple and concise allows for the code that will underpin the process (usually called step definitions) to be written in such a way as to avoid the need for frequent refactoring when the UI of the experience may change. Obviously since the interactions with the UI need to be automated within the tests this detail does need to be dealt with somewhere, this is where Page Object Models become part of the mix.

Page Object Models provide an encapsulation of the implementation detail of a page and present a business focussed implementation. To return to our previous example the Page Object Model for out login page would define a login method that internally automates the detailed steps listed in the first description, this leaves the test themselves to use the simplified language of the second description.

Should the details of the login process change the the implementation of the Page Object Model would need to be updated but the tests themselves can remain unchanged. This isolation of change promotes stability in the tests allowing them to concentrate on describing and testing function over form.

SOLID Tests

POP and the use of Page Object Models is an example of applying the same engineering discipline to writing tests as to writing the software they are testing. This frequently isn't the case, tests are often the most untidy part of a code base where they are simply churned out rather than thought through and executed with the same care and attention.

Whilst tests may not represent code that is deployed into production they are the mechanism, or at least should be, that gives the confidence to action deployments. Can this confidence be forthcoming if they are sloppily implemented or can't be relied upon to adequately and reliably verify the operation of the system?

Sometimes the reluctance of developers to implement tests is also driven by the fact that the code in this area is difficult to work with and time consuming to change. Putting the implementation of the tests under proper engineering scrutiny can remove this barrier.

If code is worth writing, whether it be implementation of functionality of tests to verify it, then it's worth writing well.

Strangling Tech Debt

Technical Debt will always get you in end, like a circus performer spinning plates you might be able to find ways to keep things intact and maybe even get some more plates spinning, but eventually everything will come crashing down.

You will have reached this point when it is no longer practical or feasible to continue to bolt on extra features or to refactor your way out of trouble. You are left with a code base that whilst it may continue to perform its intended function can no longer evolve in its present form.

This leaves you facing the proposition of a re-write, whilst this may light up the eyes of many developers, this potential new green pasture is not without its downsides. Starting again with a code base also means starting again in terms of credibility in the eyes of the business the system serves. You may see the foundation of sand that the functionality is built upon, but the business sees a system fulfilling its purpose.

Strangler Application

A different approach is that of a Strangle Application, Martin Fowler when defining a Strangler Application draw parallels to the strangler vines of Northern Australia:

"One of the natural wonders of this area are the huge strangler vines. They seed in the upper branches of a fig tree and gradually work their way down the tree until they root in the soil. Over many years they grow into fantastic and beautiful shapes, meanwhile strangling and killing the tree that was their host." 

He uses this as a metaphor for a technique where a new approach to a code base is implemented alongside an existing application and not instead of. The exact mechanism for doing this will depend upon the architecture of the system you are trying to strangle and the technical debt that has stopped it in its tracks.

But imagine as an example instigating a proxy in front of a set of APIs that enables certain calls to gradually be moved over to a new implementation, or an event based system gradually migrating the code that handles certain events over to a new architecture.

This is sometimes referred to as Asset Capture where gradually assets that the system deals with, whether data with in it or interactions from other systems, are migrated to the new way of thinking. This enables new and old code to live alongside each other, giving the new code a chance to prove itself while the old code continues to serve the business in a way they are happy with.

Over time as confidence in the new approach grows and it gains credibility it can assume more responsibility and capture more of the assets of the old system. Ultimately if this proves successful all remnants of the old system will dissolve leaving you with the refactored code base you dreamed of while never having to have faced the blank sheet of paper of a re-write and the risks that entails.

False Dawns

The reduced risk that this approach delivers can, under controlled circumstances, also allow for a degree of experimentation.

If the implementation of the Strangler Application approach allows for responsibility to be passed between new and old in either direction then if a false step is taken with a new implementation approach responsibility can be passed back to the old approach whilst the drawing board is re-visited.

This isn't to suggest that your environments become a playground, but as much as a new idea may look foolproof on paper nothing proves a line of thinking more than it being battle tested in a system fulfilling a real user need.

Any wrong step taken once a re-write has commenced can be extremely costly and much more visible to stakeholders, all of which can damage the prospect of the re-written system gaining traction with the business or enthusiasm for its arrival.

Writing Tomorrows Tech Debt

Whenever we are writing a new piece of code we are sure that this time we are on the right track, this time we have the right design and the right approach. Sometimes this will turn about to be the case, but on a not insignificant number of occasions eventually flaws in our thinking will begin to emerge, exposing cracks in our architecture or implementation approach.

This combined with the benefits we've discussed of a Strangler Application means we need to write our code to at some point in the future be strangled itself, to allow itself to eventually fade away and not to be dragged kicking and screaming.

To a large extent this can achieved by proper adherence to SOLID principles, by properly assigning and segregating responsibilities. It is the loose coupling that this way of thinking promotes that will eventually allow and area of code to be quietly and efficiently replaced with a better idea.

Any time you get the feeling that an area of code would be difficult to refactor out of a system this should serve as warning that you are taking away the opportunity to at some point replace that implementation without having to contemplate a complete re-write.

Whilst starting from scratch on a purely technical level is often quite an attractive option, on a practical level it frequently isn't a realistic prospect. As pragmatic engineers we need to develop strategies that allow us to correct our mistakes without having to take on the risk of asking the business to return to zero lines of working code.

A code base that promotes this approach allows for both the patterns of implementation and the architecture to be evolving and improving as our understanding of the right solution increases and develops.              

The Art of Imperfection

If you've spent any amount of time with a development team you will recognise the obsessive, compulsive and perfectionist behaviour that whilst possibly a stereotype also has a bucket full of truth.

This strive for perfection can be a teams downfall, leading to stagnation caused by a reluctance to commit and release. Clearly this is a difficult juggling act, swing too far the other way and pragmatism gives way to simply bad software.

The ability to apply well reasoned pragmatism to everyday development situations is a core skill that will generally come with experience as idealism is complemented by the experience of shipping, or indeed not shipping, software.

This isn't an exact science and this post is by no means a definitive guide on how to achieve it. You'll have to find your own way to achieve this in your world but hopefully these pointers will help you.

Imperfections in Architecture

If any architect tells you they are 100% happy with their architecture I would question how much they understand about how it has been implemented. We all strive to follow a ratchet mechanism where every change we implement improves our technical capability and moves our architecture forward. However, software engineering is a far too complicated discipline to never take a wrong step.

You will at various points define an architecture that you think will provide a structure for never ending success and be proven wrong when the implementation goes in a direction you didn't anticipate and is less than ideal.

Whilst a good architect will hopefully keep these situations to a minimum they won't be able to eliminate them entirely. Instead they can be judged on how quickly they recognise these situations, how they contain their impact and how they formulate a plan to reverse and try a different direction.

Adherence to SOLID design principles will help you in these situations, they will naturally ensure that loose coupling avoids the transmission of technical debt across a code base. They will also allow for a cookie cutter approach to fixing the issue where the problem area can be cleanly removed and improved upon.

Imperfections in Code

All developers will recognise the situation where you can't let go of the feeling that a piece of code can be improved. Either adding functionality or trying to achieve the same functionality with less code are common goals that lead to a reluctance to declare something as done.

Both of these goals are to be applauded but they must be approached with due to regard to the YAGNI principle (Your Aren't Going to Need It). Coding for a situation that will never arise is a fairly good definition of over-engineering, along with coding for a level of performance that you will never need to obtain.

The major benefit of following SOLID principles is how they allow for effective future refactoring. They allow for simple extensions to functionality along with the ability to change the implementation of existing functionality.

The majority of code must be accepted as a continual work in progress, only the most trivial areas of a code base are ever "finished". It's fine for code to be good enough for now if this is coupled with a clear plan of how it would be changed in the future should requirements change.

Imperfections in Features

We write software to implement features and our willingness to declare a features as ready is equally as problematic as our willingness to declare code as done. What we must realise is that it is the vacuum caused by a lack of user feedback that leads to this gun shy nature.

Obviously the initial inception of an idea requires reasoned thought as to the best way to address a user need, however before long this idea will have been implemented well enough that the overhead of continued refinement will not be matched by addtional user positivity.

Any time we place ourselves in the shoes of our users we are in danger of over thinking based on incorrect assumptions. Any refinement cycle that isn't driven by observations of users using released software has reduced odds of hitting the mark.

Getting ourselves to be comfortable with releasing what might be deemed good enough is an important mindset for a team to attain. Create something functional, prove that users will engage it with and look for where this engagement could be streamlined, increased or otherwise improved.

Perfection in software development is all about the journey and not the destination for the simple fact that perfection will never be achieved. Those that excel are able to optimise in any given moment whilst also managing the areas that need improvement in the longer term.

Whilst shipping code that doesn't work is never going to benefit anybody this aside failing to ship is likely to do you more damage in the long term than shipping something that might not be 100% perfect.        

Unit of Work

The number of individual work items required to achieve a software engineering outcome can be staggering. If the execution of these tasks is to result in success they need to be organised into a structure to avoid chaos.

Within agile teams we have developed a structure containing many different types of work items, features, themes, epics, stories, tasks and possibly many others are an every day part of teams lexicon and operation.

Whilst their definition can sometimes seem arbitrary or open to interpretation they shouldn't be seen as a collection of things to do. They must be written and grouped to achieve a purpose and have a defined outcome. Whilst this may seem obvious it can be surprising how often we can lose sight of this.

Defined Value

The purpose of properly segregating what work needs to be completed isn't simply to ensure a team is kept occupied. It must also serve to help planning and act as documentation for what is going to be built, what has been built and why.

This means each item must explain its value to the business. Providing functionality for users is clearly valuable to any business but constraining work items to only being described in these terms is limiting and doesn't acknowledge that there are other reasons to write code that benefits the business.

This needs to be equally true when we group together work items. As an example it can be tempting to treat items such as epics simply as large collections of user stories, but such a loose definition can lead to the collection being superficial. No matter what the scale completing an item of work should bring a self contained benefit, it shouldn't imply a benefit that isn't actually achievable until more work is done.

Do No Harm

If a team is to follow a DevOps mentality, to ship continuously on an automated basis, then each work item must result in shippable code. This doesn't mean it needs to implement an entire feature, it maybe that nothing changes the user can see or interact with whilst still achieving a business benefit of moving towards something bigger.

However it shouldn't result in the code base being unstable or incomplete unless further items are completed. If this is the case then it indicates that the original division of work was wrong or that a different implementation approach was required.

Sometimes a dose of realism is required to accept that the required functionality requires a large amount of effort, dividing the work up into too smaller chunks to try and change that situation won't result in achieving the goal any quicker.

A Plan Not An Idea

Work items shouldn't document ideas they should document plans to achieve a statable and desirable outcome.

Generally if you review an untidy backlog many of the work items that have lingered for sometime will be ideas that haven't made the journey from initial inception to work the team can and should do. There are many ways to document ideas but they must be refined and pass through a filtering process before they should be considered for execution by the team.

This shouldn't be seen as stifling creativity, teams absolutely should be free to spit ball ideas, but when it comes to spending some of the teams valuable effort and expertise on delivering one of them then it needs to be undertaken with a clear plan of what they will do and why it is worthwhile them doing it.

Nothing is hard and fast with agile, whilst we may have taken a decision to push back against large upfront planning that doesn't mean we have adopted chaos. Agile does involve planning but with a dose of realism on what can be controlled and what can't. It isn't possible to do this planning without understanding the work that will get you there. The effort of your team is a currency that you must plan to spend wisely and not to squander on a series of unrelated tasks with no clear purpose.

Drowning Under The Waterfall

Whilst the software engineering fraternity has widely embraced agile methodologies it has also struggled to let go of its waterfall roots. The behaviours that waterfall encourages have not been entirely let go and continue to creep into the ways that teams approach delivering code.

It is understandable why this is a difficult break-up, the benefits that waterfall claims to offer are intoxicating. It claims to offer certainty and the ability to plan an exact strategy for success, but ultimately these are hollow promises that it cannot deliver.

It cannot deliver because the world it is intended to operate in doesn't exist.

Crystal Ball

A vital piece of equipment required to implement an effective waterfall strategy is a crystal ball. It requires stakeholders to have certainty over their requirements with no gaps or uncertainties, it also requires developers to understand all the problems that will arise when trying to implement that vision.

The reason for this is that waterfall creates elongated feedback loops. It requires initial up front design, development and finally test. If at any point stakeholders are found to have missed or misunderstood a requirement, or developers to have not anticipated an implementation problem, the price paid in elapsed time is heightened.

Software development is fair too complicated to implement a strategy based on certainty. This complication exists both in its implementation and in defining the required outputs. Effective development strategies shorten feedback loops and encourage collaboration in an attempt to combat rethinking and lessen its impact.

Excluding the User

The big bang release approach encouraged by waterfall has the unintended consequence of sidelining end users. Stakeholders act as a proxy for the user, as they do in most development methodologies, but the issue caused by waterfall is it takes away the possibility for them to be wrong.

A plan based on perfectly predicating users wants and needs is high risk and likely to fail. A more realistic approach to accept the possibility of being wrong and to structure releases to reduce the size and impact of any possible mis-step.

Every release should be a learning experience whilst equally allowing time and resource to implement those learnings. If each release is a wager on your understanding of your users how much of your development teams effort and hard work will you risk?

Linear World

One of the most alluring aspects of waterfall is its ability to predict outcomes over long periods of time allowing a teams planning horizon to be far into the future. The issue with this is it implies a linear and predictable understanding of all the factors that can influence the outcome of a project.

Very little about human behaviour is linear, this applies equally to people charged with delivering a project as well as the intended users once the project delivers.

The interesting aspect of this flaw is that the illusion of certainty in predications is so attractive that we fail to be to able to let it go despite contradictory and frequent evidence to the contrary. It has almost become a cliché for an IT project to be late and over budget, this is despite the fact that all these projects would have started with a carefully crafted plan that was supposed to mean this time it would be different.

No-one wants to hear the answer "we don't know" or "we're not sure" but if uncertainty exists this answer invites the possibility of a contingency to be planned, "we got it wrong" offers no such hope.

No-body can claim that any methodology has all the answers, but an advantage of agile is its willingness to embrace the things we don't know and structure our approach accordingly. As attractive as it may be to be seduced by a methodology that claims to hold all the cards its important to remember that we develop code in the real world and we are therefore subject to its foibles and unpredictability, learn to live with that and you'll start making better decisions.  

The Art of Software Engineering

Software development is rightly categorised as an engineering discipline, there are many well established principles and practices that deliver more robust and scalable code. But compared to its contemporaries such as structural or mechanical engineering it is still a relative young practice. 

This combined with the fact that coding can be a blank canvas with many possible ways to achieve an outcome mean it can also exhibit traits more akin with an art form.

Non-Scientific Measures

Developers will often describe code as being good or bad, this is very rarely a judgement based on measurement and opinion on the quality of code may not even always be universal. These opinions can drive new proposals for solutions to common scenarios, providing new takes on old problems.

Whilst I'm sure this can happen in other engineering disciplines it feels like software is unique in the way it can revisit old problems that already have solutions. To put this in context, structural engineers probably no longer debate how to build a wall, a correct solution has been found and we've moved onto other areas.

Correctness in software engineering can be more opaque, it is weighted against the opinions and beliefs of the engineer making the assessment. 

This isn't necessarily a negative, a willingness to re-evaluate and even criticise can be a strength if it's used as a driver to ensure that all possible avenues have been explored and no ideas discounted.

Valued Characteristics

One of the things that can drive a change of view of the correctness of code can be a change in the characteristics that we most value. Most languages and frameworks have many options for tooling that can be used to quantify these measures and different philosophies may choose to put more emphasis on one or the other.

This might be testability, reusability, speed or efficiency. Whilst when questioned engineers would realise the benefits of all these things they may choose to sacrifice one for the other because they feel it's a better measure of correctness.

To continue our analogy, the valued characteristics of a wall are unlikely to change. It needs to be a strong and durable, dissenting voices amongst engineers are likely to be few and far between and unlikely to gain traction.

An advantage of this can be that it develops strategies and practices to optimise certain aspects of our code. Even if you decide that this optimisation isn't the sole focus of your engineering effort their existence is still an advantage when you feel that particular aspect of your software could be improved.

No Raw Material

Where software engineering's uniqueness is harder to question is in its lack of a raw material. Most other engineering disciplines take raw materials, whether it be bricks and mortar or steel and plastic, and turn them into the desired structures and assemblies.

The raw material of software engineering is the creativity, ingenuity and effort of the developers that practice it. The advantage of this is the freedom it gives to evolve quickly and achieve impressive things on minimal budgets, the disadvantage comes when mindsets assume closer alignment with more traditional engineering.

Scaling software development is a difficult, and possibly still unsolved problem. If you need to build a wall then clearly having more people laying bricks will speed up the process or enable you to build a bigger wall. Taking the same approach to painting a portrait is likely to lead to an incoherent and unsatisfying outcome.

Realistically software engineering exists somewhere between those two extremes. In the early days of a project adding more engineers can increase output whilst maintaining quality, but this relationship is not linear and will very quickly breakdown if the differences to traditional engineering aren't realised.

It maybe that these aspects of software engineering are related to its relatively young age, after another hundred years of software development maybe we will see less flux. Or it could be the case that software engineering has qualities that make it inherently unique in its approach. Rather than something to be feared we should embrace the freedom our discipline gives us to try new things, evolve our thinking and change the shape of the code we build. 

Code Recycling

If you observe a group of software engineers discussing an area of code for a long enough period you will hear the phrase "reuse". We rightly hold the benefits of code reuse in high esteem, it can reduce development time, provide increased levels of reliability and increase the adoption of desired practices.

But is this the whole story? Are there disadvantages to reuse? What different forms can reuse take?

This is potentially a large area of discussion with almost philosophical overtones but one potential approach is to analyse what is being reused and identify the pro's and con's of the approach.

Reuse of Effort

The simplest form of reuse that is often considered sinful is copy and paste, this is where code is simply duplicated, and potentially modified, but results in the same code existing in multiple places in a code base.

This allows the developer doing the copying to take advantage of the thought and effort expended by another to solve a shared problem. The downside comes from the duplication of code. Code is not an asset to an organisation because it needs to be managed and maintained, duplicating an area of code creates the potential to duplicate bugs and the effort of fixing them.

So code should never be copied?

Well the answer to that question is almost always going to be yes but as with many things it doesn't hurt to apply some pragmatism on occasions. Copying a small number of lines of code when the effort of defining an abstraction and creating a dependency for other areas of code may not always be the wrong thing to do.

The analysis of whether or not this is the wrong thing to do needs to take into the account the number of times the code has been copied. Copying it once when defining an abstraction might be problematic might be able to be justified, copying it any more times than that is indicating that the opportunity for reuse, and its associated benefits, is presenting itself.

Reuse of Code

When we've made the assessment that copying the code would be wrong then we look to reuse the code itself. This can take many forms whether it be creating a library or having shared source code.

This is where the benefits of reuse are realised. Reusing code decreases development time by providing ready made solutions to common problems, reusing the code in multiple places allows it to be battle hardened and honed overtime.

So code reuse is always a good thing?

Well the answer to this is usually going to be yes but it isn't always an absolute. Reusing code creates dependencies and this doesn't come without its own potential headaches. A large web of dependencies can stifle the ability for a code base to be refactored, maintained and grow. It is no surprise that many of the SOLID principles relate to reducing the scope of individual areas of code this will therefore reduce the scope of the dependency on it.

This shouldn't be taken as an argument against reuse but as a word of caution to think it through. Reuse code when the abstraction will hold and the opportunity for reuse is clear and unambiguous.

Reuse of Function

Ultimately we aren't actually trying to reuse code but functionality and we have developed ways to achieve this that can go beyond code reuse.

Software as a Service (SaaS) is common place among cloud providers but the same approach can be taken within your own code base. Divide and conquer has long been an effective strategy for engineering well defined code bases and techniques such as microservices have this at their heart.

A library doesn't have to be the only way to reuse functionality, using REST APIs can also allow functionality to be reused whilst also providing a really clear interface with the potential to evolve over time as requirements change.

This won't always be the correct approach and will come down to the scale of functionality that needs to be reused. Having a microservice to perform operations on strings would rightly be deemed ridiculous, but caching or complex data manipulation maybe more likely candidates.

Identifying and effectively implementing code reuse is a core aspect of software engineering. We have many tools available to us to achieve this and whilst it will nearly always be the right thing to do the potential pitfalls must also be considered and factored into the strategy.   

Language Lessons

Developers will very often define themselves by the programming language they are using to apply their skills. Based on the perceived strengths and weaknesses of different languages they will also very often judge their counterparts along similar lines.

To a large extent this view point is misguided. The differences between languages in the same paradigm are often superficial and the majority of developers will use multiple languages during their career.

The reason developers will utilise many different languages is because they are an evolving tool, this evolution can be tracked through various generations as new concepts have been developed and different approaches postulated.

First Generation (1GL)

1GL languages, also known as machine language, is code that can be directly stored and executed by the CPU, in other words 1's and 0's.

Programming in a 1GL language would often involve the manipulation of switches or other hardware and is almost unrecognisable from modern programming techniques.

Second Generation (2GL)

2GL languages start to introduce constructs to allow developers to read and make sense of the code. They are formed of combing instructions that are executed directly by the CPU being targeted. However they provide little to no abstraction from the steps taken to run the program, developers operating at this level are manipulating registers via simple mathematical and logical operations.

Coding at this level has the potential to be very fast and very efficient, but equally is extremely complex, error prone and difficult to debug.

Code written at this level is also not portable, it will only run on the generation of processors it has been written for. While most developers will not operate at this level the code they write using higher level languages is often compiled down to a 2GL program with the subsequent binary file that is deployed representing a 1GL language. 

Third Generation (3GL)

3GL languages introduced many of the basic programming constructs that all modern day developers are familiar with. This includes logical constructs such as if, else-if and looping constructs such as for, while, do-while.

Whilst still describing what needs to happen when the program runs it describes this behaviour in a more algorithmic fashion as opposed to describing how the CPU should manipulate bits and bytes. 

A certain division of 3GL languages also gave rise to the concept of Object Oriented (OO) languages which attempted to represent programs in terms of collections of data and functionality that interact in a manner designed to model the problem space being solved.

Some 3GL languages also attempted to make code portable by being compiled to a 2GL language that runs on a virtual machine and therefore not tied to a particular physical CPU.

Fourth Generation (4GL)

4GL languages attempt to improve upon 3GL by operating upon larger data constructs. Sometimes this distinction can be subtle but 3GL languages often operate on relatively simple and low level data structures such has strings, booleans and value types.

Operating with higher level data constructs can make these languages less general purpose and often leads to 4GL languages being specialised around a particular purpose such as graphics or database manipulation.

Fifth Generation (5GL)

5GL languages attempt to shift the view point of code from how an outcome should be achieved to describing what that outcome should be. Rather than a programmer coding an algorithm they describe the constraints the program should operate in and the desired outcome.

The boundary between 5GL and 4GL languages is often blurred. Some 4GL languages also try to operate along the lines of what as opposed to how and are sometimes miscategorised as 5GL languages.

It would be wrong to view this evolution of languages as each subsequent generation being better than its predecessor. Mostly the changes between generations are focussed around making the programmers life easier and allowing them to describe algorithms in higher level terms, making code more readable to someone that understands the problem that is being solved.

Many coders still operate at 2GL or lower 3GL level because of the potential to achieve higher performance and efficiency. A good example of this is the prevalence that languages such as C and C++ still have in fields such as graphics and gaming.

There is no language to rule them all, eventually you may be required to solve a problem that requires a different approach, this requires you to be comfortable to go to the toolbox and select the correct tool for the job in hand no matter what generation that may be.

Whatever generation of language your are using don't define yourself by it. A programming language is a tool and you would likely be as adept at applying your coding skill in many other languages.