Scaling Out Delivery

As agile becomes more widely adopted by organisations both large and small inevitably questions are asked as to how this can work at scale, we have seen the birth of this phenomena with the advent of various frameworks built around the concept of agile at scale.

I believe the thinking around several of these frameworks is flawed and moves us further away from the pillars of agile that attracted us to it in the first place.

Segmenting the Orange

The thirst to adopt agile at scale is mostly fuelled by the results that are seen when using it at a smaller scale, the key thing to keep in mind when trying to grow that success is recognising what it is we are actually trying to scale.

It is very easy here to fall into the trap of assuming benefits of scale in software development that simply don't exist, the purpose of trying to scale out agile delivery shouldn't be about trying to justify the addition of more developers.

Without proper segmentation of the deliveries they are working on having multiple teams of a handful of members is no different to having one large team.

The reason to create a new team shouldn't be to justify adding more developers to a project whilst paying lip service to the idea that a efficient team has a maximum number of participants.

The reason to create a new team is because we have identified an independent delivery of software that can be worked on in isolation, divide and conquer is an old adage but its applications are many.

Obviously in the long term teams will need to consume the output of what the others are working on but the aim should be to maximise the number of different areas that can be under active development without creating inter-dependency between teams to deliver over the short term.

We are trying to grow the number of distinct outputs of teams not the amount of output in any one particular area.

The Grand Plan

As the scale of development increases there will be an increasing need for some co-ordination, which one could call planning, to be in place to ensure everyone moves towards a common target.

When trying to scale agile it can be very tempting for the waterfall illusion of control to creep back into our thinking.

The co-ordination thats established between teams has to be of a loose nature that has the ability to flex with changing circumstances.

This always carries negative connotations assuming this means expect things to be late but actually it can be positive, an agile mindset is very much to ship when something is shippable.

Better segregation between the deliveries of teams can enable a multitude of different cadences of iteration to be in place where opportunities to ship present themselves more frequently not less, not all will be visible or obvious to the user but all will move the product forward.

Philosophy not Frameworks

Needing to find ways to work at scale is an inevitable reality but its important to distinguish mechanics from mindsets.

Agile is not a framework for software development it is a philosophy, a mindset, a way of thinking.

Quite often some of the pillars of agile such as face to face communication or the preference for working software over documentation are the first things to suffer when we are trying supersize our organisation.

Its therefore important that whatever the mechanism we use to scale out delivery we recognise when we are betraying the principles that as agile practitioners we are supposed to hold dear.

One of the best ways to adhere to this is to make sure practices don't become rigid and that we continue to garner feedback and input from the those at the coal face.

We must avoid the temptation of thinking that to operate in an agile way at scale requires a different approach to adopting agile at any other time.

We don't need to re-invent the wheel or go back to the drawing board, the principles of the agile manifesto still hold and our job is to ensure they aren't eroded as the number of participants grows and the allure of falling back into a waterfall pattern become attractive.

Open Source Musings

Developers have a long history of sharing the fruits of their labours with each other, as software development became a commercial activity and not just for hobbyists this concept of sharing was eroded.

In the later 90's several organisations were founded to promote the concept of open source software to not only share code but to commit to developing it in the open for everyone to see and contribute too.

Open source is now a core part of the software development ecosystem with everybody from backroom coders to large enterprises consuming it and contributing to it.

So is using open source now a no-brainer, a moral obligation? As with most things it depends, there are benefits and disadvantages.

Free at the Point of Need

The most obvious benefit to open source is that it is generally free to consume. Aside from the obvious, this removal of a barrier to entry can foster increased innovation and enable the faster development of new ideas.

However its important to remember that free doesn't mean unlicensed, whether it be MIT, LGPL or BSD the majority of open source projects will come with licensing requirements before they can be used and its important to pay attention as to whether or not this fits with the project your working on.

The absence of a license is also not necessarily a good thing as it means a lack of an agreement between you and the writer of the code over acceptable use.

Finally being free means you have to adjust your expectations on some of the services you might expect when dealing with proprietary software, once of which is support.

While many major open source projects have a thriving community if your project becomes over reliant on a small open source library it may not be reasonable to expect round the clock support, you have to be prepared to take responsibility and dive into the code base yourself.

Flexibility to Break

A major benefit of open source is the freedom it allows to take something that is close to what you need and adapt it or add that killer piece of functionality that you think would really increase its usefulness.

The ability to see under the hood can provide inspiration to see many different ways something could be improved as well as enabling you see the solution to that bug that has been causing you a headache.

However in software development where there is change there is always the potential for instability and the threat of complication being introduced.

When assessing an open source project take the time to look into the community behind it and look to see how the project has evolved.

Does the project have a history of uncontrolled change causing instability or is there clear leadership within the community to steer the ship in the right direction?

An open source project is more than just the bits and bytes of the code itself its about the community of people who drive this code base, if the project becomes important to you these people will be as much your colleagues as anyone within your own organisation.

A Change of Perspective

In a traditional commercial approach to software development the code itself is a means to an end, part of an overall package being sold to the end users who require the features it implements.

In an open source project this view point changes, the code becomes very much centre stage, when your users are developers it brings a different mindset.

Developers are details people, they are more likely to concentrate efforts on an optimisation or an improvement in quality.

Developers are also generally opinionated people who have a well defined perspective on how they apply their craft.

Whereas the commercial development of a software based product is likely to be governed by the drive for new features for end users, the development of an open source project is much more focused on the needs and wants of the developers that intend to use the publicly available code base.

This change of focus doesn't necessarily have to be labelled good or bad it is simply different and what attracts many to open source projects. As with any venture that is collaborative it means that not everything will go in the direction you want it to.

Open source has many advantages and its introduction into the software development ecosystem is to be welcomed but the choice to use it should be taken based on more than just the fact that the code being made available can fix your problem.

It should also be based on an intention to engage with the community that supports it and to realise that although the software is free it comes with certain responsibilities to play a role within that community.

The collaborative nature of software engineers is admirable and we are fortunate that sharing the code that represents our currency can be made so straight forward but whenever you embark on a new journey it is advisable to look before you leap.

Functional Lessons

Software engineering is made up of many different paradigms each one with its advocates and its detractors.

Which one is currently en vogue can be subject to trends in thinking or technology.

Functional programming is one such paradigm that has grown in popularity with its application being championed in various different arenas.

I have written before that the application of a technique in all situations regardless of merit is foolhardy and that Object Oriented Programming (OOP) is being written off too readily, however that is not to say that there aren't things we can learn from functional programming to improve our approach to OOP.

Avoiding Side Effects

A key tenant of functional programming is referential transparency, this is that a function or method should be interchangeable with its return value.

This behaviour implies that a method is pure, has no side effects and contains no state.

Writing methods that are this pure is difficult, despite our best efforts objects very often do have state.

A more practical approach is to avoid side effects that cannot be anticipated based on the name of the method or the name of the class in which it resides.

This avoids what is sometimes referred to as action at a distance where operations in one area of code have an unintended and unpredictable effect elsewhere.

This is a very good definition of a bug and shows how side effects can set the caller of your code up to unwittingly create problems in the code base.

Immutability

A consequence of referential transparency is that objects and values once created are immutable and cannot be changed.

Whilst this also plays a part in avoiding side effects it also has more practical benefits in providing efficiency in certain situations and going a long way to ensuring thread safety.

At the heart of every threading problem is usually data or state being mutated in an uncontrolled manner.

While again pure immutability can be difficult to achieve the change of any important piece of data or state in an object should always be implemented in a controlled manner.

Where at all possible changes should be surfaced by creating new objects or data rather than modifying anything pre-existing.

Declarative Style

Functional programming is said to have a declarative style where code expresses what must be accomplished in contrast to an imperative style that describes how something should be achieved.

This implies that code describes the results of computation without exposing detail of the control flow of how this was achieved.

As complex as this may sound good naming of classes and methods will allow code to be written that doesn't focus on the detail of what is happening and instead allows the caller to only be concerned with the results.

The details of computation are often subject to change, the more coupling that has been created between this detail and other areas of the code base the more impactful this change will be.

Something as simple as using a well named abstraction to hide this detail can loosen this coupling and allow code to be written using a more declarative style.

What we should take from all this is that the pillars and principles of writing good code are actually universal, different paradigms often choose to place more emphasis on certain aspects but we will observe a certain amount of cross pollination.

Trying to take any one of these principles to its ultimate conclusion will very often require you to put more effort into coding within defined boundaries as opposed to taking a practical approach of trying to ship software.

However as we've demonstrated many of the benefits of these principle can be achieved by relatively simple means when you have a sensible and practical outlook.

Team Under Construction

Its a normal part of business practice to on occasion need to hire new people.

The process can be stressful for both sides, employer and employee, and both sides would do well to realise that it isn't a precise science and ultimately both sides will go with their gut feeling about whether this seems like the right thing to do.

So if we can't have a full proof strategy for hiring the right people how can we maximise the likelihood that we make the right decision?

Understanding over Knowledge

In the early stages the interview process can often resemble a pop-quiz, asking the candidate questions relating to the language, paradigm or area of programming that the candidate will be expected to work in.

Whilst this is useful for weeding out candidates who can't demonstrate the required knowledge we should also be looking to allow the candidate to talk about the subject at hand and their approach to their chosen discipline.

There is much more to being a good developer than simply a technical understanding of syntax or the infrastructure involved.

The reason we value experience is because it breeds a deeper understanding of the subject matter, it helps develop a philosophy towards software development that elevates a candidate beyond just the technicalities of coding.

By allowing the candidate to talk you allow the possibility for a warm feeling to develop that the person in front you is an engineer who not just understands the individual cogs but appreciates the intricacies of fitting them together into a machine.

Team Game

There is another question that needs to be answered during the hiring process that is equally important as whether or not the candidate is capable, can we work with this person?

Development is not an individual pursuit, teams write software and whoever joins that team needs to fit the dynamic that has already been established.

No matter the technical expertise of an individual if they won't gel with the other members of team they will ultimately not have a positive impact on the teams output.

It is also important to assess if this individual brings something to the team that it doesn't already have, this may be a skill set that is currently missing or a personality trait that will add to the overall make-up of the team.

Why not introduce candidates to their potential team mates as part of the process? Have them write some code together? The feedback you get from an activity like this will give a good indication of the effectiveness of adding this person to the team.

Opportunity Knocks

But this process isn't all about the employer, as a potential employee the hiring process is also a chance for you to gain the knowledge you need to decide if you want to make the commitment to be part of this team.

We write software to achieve business aims this will be a lot more enjoyable process if you find the nature of these business aims interesting and something you want to help achieve.

Don't see engineering as just a technical exercise also try and gain insight into what this team is trying to build and the functionality and experience they are trying to deliver.

Also ask yourself if working in this team will stretch you, no-one wants to feel out of their depth but ultimately we will only grow as engineers and individuals if we attempt things we haven't done before or try and do them to a level we haven't achieved before.

Looking for opportunities like these is what will give you the experience and confidence to become the well rounded and mature candidate that can demonstrate an understanding of their discipline beyond the explanation of syntax or buzz words.

Hiring people is often an error strewn process and there needs to be an acceptance that all we can do is attempt to find people that give us confidence and demonstrate a personality that we think will fit our team.

Only time will tell if we make the right decision because we aren't just hiring someone to churn out code all day we are hiring an individual who we can incorporate into a team to make it more than the sum of its parts.

Breaking Not Bad

You'll will often hear developers lament at the fact unit tests are broken and need fixing.

While this can sometimes feel like needless extra overhead in implementing a feature the breaking of these tests is an important part of the ebb and flow of development and has value in its own right.

Many may be scoffing at that sentiment, so what is so great about failure?

Change is Good

We can learn a lot about the changes we have just made to the code by the failure it introduces into our unit tests.

Firstly it demonstrates that we have changed the function of the code we have been working on, I would be much more concerned if no unit tests were broken, that would imply our changes have had no impact on the code or our unit testing is inadequate.

This time the change was intentional next time it might not be and surely we would want the tests to fail in this scenario?

But its not just a case of seeing failing tests, its also a matter of seeing which ones have failed.

Do the tests that are failing make sense within the context of the change that was made or are they actually demonstrating an unintended side effect or problem with the engineering of the code? Ask yourself should these test be broken? 

Don't just blindly fix the tests that are failing also make sure that this all adds up. When it comes to pushing these changes the tests you had to fix will also serve as documentation for the reviewer of your work so they can do the same thing.

Walking on Egg Shells

Whilst extolling the virtues of failure its important that unit tests failing is always in relation to change in the code base and that its scale is in proportion to changes being made.

Unit tests shouldn't fail when no-one touched anything, running the tests over and over again should always produce the same results. Race conditions and brittleness are not qualities we want in our code bases's unit tests.

When measuring the scale of failure we should consider several factors.

Its natural to just look at the number of broken tests but this can be misleading, we also need to consider the amount of time needed to fix the tests even if a relatively small number have been broken.

If we change an interface or a method signature then its natural that a proportionally large number of tests may break, if however only a handful of tests fail but serious thought needs to be put into how the functionality can now be effectively tested then this is indicative of wider problems. 

Taking the Red with the Green

Ultimately we have to ask ourselves why did we write the tests if we didn't expect or even want them to fail occasionally? Are tests that never break actually worthwhile?

Unit tests are a comfort blanket that gives a nice warm feeling that our code is still fit for purpose and still does what it says on the tin. Its difficult to derive this warmth that they truly can be relied upon if we never see them fail, especially if we know the code base is changing.

When we are practicing TDD we take satisfaction from seeing a set of unit tests gradually change from red to green, we do this because it proves the code we have just written does what we wanted it to.

We can take similar satisfaction from seeing the same transition from red to green when a change we make breaks some tests. It provides demonstrable evidence that the code is still doing what we want it to.

We need to realise that the value of unit tests increases over time, they have value when the code is first written to help us engineer interfaces and behaviour and to validate new code works. But they have even more value as time goes by to highlight when code is broken and act as documentation for how code should be used.

Failure is a naturally part of this process and is simply part of the information unit tests convey.

Next time you break some tests don't sigh, instead smile because your unit tests once again have proven their worth.