Adolescent Behaviour

The start-up holds an almost mythological place in the mindset of the technology industry, often perceived as a zen like state with all elements in perfect balancing producing perfect software.

This article is not meant to be an attack on the notion of a start-up, many large organisations can benefit from the principles and practices they observe.

Instead it presents some views of behaviours often observed in a start-up culture, not necessarily good or bad but simply a consequence of being new on the block with limited resources and a strong desire to be successful quickly.

Bend the Truth

Start-ups are often in a constant state of seeking funding, whether directly via venture capitalists or investors, or indirectly via trying to attract customers to provide an income stream. This means they are very often pitching, both formally and informally, they are trying to get their message across and advertise their capabilities and ambitions.

This may seem a controversial statement but sometimes they lie, and actually this is ok.

Especially in the very early days if a start-up was honest about their current capabilities they would have very little to pitch so it is natural that this gets embellished to present future goals as already being in the here and now.

The critical factor is that everyone needs to recognise that the truth is being bent.

Those being pitched to need to understand the nature of a start-up in this position and judge them more on whether they think they can get to the destination they are pitching and not whether they believe they have already arrived.

In turn the start-up should not present whats required to its developers as if this was always supposed to be the case and acknowledge hard work and savvy will be required to meet expectations.

It is possible to sell a dream as reality and hit deadlines but all parties must acknowledge the nature of the pitch and agree to work together to make the lie true.

Building Front to Back

Another consequence of the constant need to be pitching is the hunt for the wow factor. No matter the level of engineering genius it is difficult to get people excited about an effective and scalable infrastructure design or testable and adaptable software architecture.

This leads to a natural concentration on the frontend over the backend, to developing an awe inspiring UI\UX that is teasing at the functionality and possible ingenuity underneath.

In time if the venture is to be a success an equally effective backend will be joined with this snazzy and flashy front end but initially there will be growing pains as the first iteration creaks under the weight of the frontend ambitions.

Again the important factor here is an acknowledgement of reality, such that we don't over estimate what is achievable in this initial phase of the organisations development, along with the fact that effort will need to be expended on building the capability that was hinted at.

Make Noise

The goal of many start-ups is not to scale their original idea into a Goliath, starting from zero and scaling out to infinity is an extremely difficult undertaking. Thats why often many are looking for a shortcut that often involves becoming part of a larger organisation.

A tactic for achieving this is to make noise about the possibilities your technology points at, to wet the appetite of other organisations and tempt them into taking a chance that this could be the next big thing.

This tends to lead to effort being put into the core technological aspects of a proposition and not necessarily on scaling it out. Scale is put off to another day when the resources available to the organisation may have grown significantly and quickly, both financial and in terms of experience of taking things to the next level.

As with the previous points this approach isn't a problem providing all involved are in on it and honest about the objective of the initial short and medium term goals.

Not all the points made in this article are relevant or accurate for all start-ups. So called unicorns who start life or soon achieve vast resources are often not constrained by the issues mentioned here.

But this is a very different situation to be in compared to a small band of people trying to build success from humble beginnings, this presents a unique challenge without a guaranteed blue print for success. This leads to certain unique behaviours that while at first may seem questionable is understandable given the circumstances and can lead to success despite the initial seemingly overwhelming odds.

Advancing the Art

Software Engineers have an intimate relationship with programming languages. Vigorous Arguments and spirited discussions will ensue whenever the advantages of one over the other is debated.

Sometimes these arguments while enjoyable revolve around technical nuance. Whilst different paradigms exist, functional, procedural or object oriented being examples, languages within each are often exhibit similar features and functionality.

Most languages continue to evolve, introducing new ideas to try and stay relevant to developers and the industry. Sometimes these new features have a fundamental impact on the approaches that are taken to writing code. In this article I'll present what I feel to be some such features, they aren't necessarily recent but there impact cannot be overestimated.

Managed Runtimes

The differences between managed an unmanaged code lies less in the constructs of the language itself and more in the target they are compiled to.

Unmanaged code compiles to the instruction set of a physical machine, whereas managed code compiles to be run on an idealised virtual machine. Many examples of this exist but two of the most common are the Java Virtual Machine (JVM) and the .NET Common Language Runtime (CLR).

The original intention of this approach was to make code more portable, the so called "Compile once run anywhere" methodology, but it has also brought many additional benefits.

These benefits have been related to both performance and security but chief among them is the advent of garbage collection. Because of this a whole generation of developers now don't  know the pain of managing memory, trying to match-up allocation and deallocations, pointer arithmetic and heap exhaustion. Whilst it is still clearly possible to generate memory leaks many tools now exist to help with this.

The benefits of a managed runtime are continuing to be felt, emerging technologies such as Web Assembly are attempting to bring more languages choices to web development, the implications of these approaches and the impact they will have on software are probably still yet to be realised.

Asynchronous Programming

Phil Karlton is often credited with the quote: 

"There are only two hard things in Computer Science: cache invalidation and naming things"

Many have suggested adding to that list, one that would get my vote would be concurrency. As code evolves beyond the trivial there almost inevitably comes a point where we need to be able to do multiple things at once, equally inevitable are the problems we get ourselves into when we reach this point.

There was a time when developers were directly exposed to thread scheduling and expected to manage this complex and error prone area. Thankfully modern languages have provided abstractions to protect us from ourselves and allow concurrency to be approached in a safe manner.

These abstractions are usually hiding a large amount of complexity and they do not mean that concurrency is always completely safe and bug free, but as with managed memory they do allow developers to not expose themselves directly to this complexity and thus give them a fighting chance of getting things right.

Lambda Expressions

In the past all functionality had to be bound to an identifier, this made it difficult for functionality to be passed between entities. Its true that languages like C have function pointers but even in this scenario the functionality itself is not unbound and can be a difficult concept to use effectively.

The emergence of lambda expressions has enabled variables relating to functionality to be defined that are truly first class citizens, this can lead to very expressive code that provides extremely readable solutions to what were once difficult problems.

A good example of such a technique is the .NET Language Integrated Query (LINQ) feature. Using this feature operations on large datasets that would have traditionally required potentially large blocks of complicated code can now be very simply expressed.

The ability to pass functionality around as easily as we do data opens up new techniques and approaches that previously wouldn't have been impossible or at least would have made for inelegant and buggy code.

Reflection

It used to be that code was mostly geared around operating on various forms of data structures, with the invention of reflection it was possible for code to become meta, code could be written to operate on other pieces of code.

This can be a double edged sword, on occasion reflection can be used to get developers out of corners that they are responsible for painting themselves into. Reflection is a powerful technique that is sometimes used as a nuclear option when more efficient solutions could have been found.

Despite this the addition of reflection to the toolbox was a big step forward for the art of software engineering and opened up whole new ways of thinking about problems and there solutions.

Many people reading this will have other language features that they feel should be on this list. My views on this are likely to be influenced by my own experience in development but this speaks to the diversity that exists within software engineering.

We should consider ourselves lucky to work in an industry where the raw materials available to us are constantly evolving and growing. These advances allow previous scars to heal, whether these be related to memory leaks, deadlocking or pointer confusion, the pain of dealing with these problems inspires us to solve them once and for all so that future generations of engineers don't have to experience the same frustrations.                                    

Rears on Seats

An adage that could be applied to many software projects would be something along the lines of from small beginnings large inefficient teams grow.

In the early days of a project a well focused, dedicated and small team can appear to achieve miracles in terms of the scale of what can be built and the time it takes to build it. When faced with such tremendous success it is a common and natural reaction to assume that if we grow this well performing team then we will be able to achieve even more in even less time.

Unfortunately this thinking is flawed but despite this its a lesson the software industry has failed to learn almost since its inception.

Software as a Commodity

Much of this flawed thinking comes from viewing software as a commodity, more engineers means more software and more software means more functionality? The problem with this approach is it views software as a raw material with intrinsic value, this in turn equates software development with a production line.

Engineers aren't simply producing chunks of software to package and ship to production, they are in fact trying to work as a team to craft and refine a single piece of software.

To continue the production line analogy, software engineers aren't all building individual cars, they are a race team trying to perfect a single race car. When viewed like this it should be clear that more craftsmen won't help complete the job any quicker or to any greater degree of quality.

Software has no intrinsic value, we aren't employing engineers to produce more of it, we are employing them to produce just enough for us to extract value, and to try and increase the amount of value we can squeeze out of any giving quantity of code.

Scaling Capability Not Code

As an organisation engineering software its important to understand within the context of scaling what is it your are trying to scale?

If this is simply the amount of code you can produce then increasing the number of engineers in your organisation will achieve this, if however you want to scale your capability to deliver reliable and robust functionality then the solution maybe more subtle.

Software engineering resource is more akin to a sponge than a brick. It can be counter productive to try and line them all up to build a wall, instead you need to concentrate on squeezing more out of the resource you already have.

This doesn't mean working them harder, it means having faith in their abilities and asking them how they think they could be more efficient or more productive. At a certain point the answer will be that more engineers are required, reaching this point prematurely will ultimately have a much bigger negative impact.

Your engineers are experts in the development of software both in general and with regard to the software you currently have, they will also be experts on how your current processes can be improved and refined.

People Bring Overhead

Software engineers do not exist or work in isolation, software is developed by teams. As both the number of people in a team and the number of teams themselves grow this comes with an increase in certain overheads.

Effective communication between engineers will decline, more formal processes will grow and dynamism will take a back seat.

Of course this isn't inevitable, many organisations have large and successful teams. But effort is required to think about how to structure these teams to ensure that barriers don't grow between them.

Ineffective communication between teams is quite possibly the number one reason why large numbers of engineers can fail to be more than or even equal to the sum of their parts.

There are no hard and fast rules as to how to accomplish this but trying to resist the temptation to throw resource at a problem will at least to delay these headaches until the point where there is more certainty that a team needs to grow.

As with many things an important step when trying to scale software development is acceptance that you'll probably get it wrong. Recognising the signs that will highlight this will ensure that it isn't your wrongness that scales. The next step is in embracing the fact that imposing a scaling strategy is likely to be ineffective, instead, placing it in the hands of those that are being asked to scale has much more potential to produce the right answer.

Divide and Deploy

Certain patterns and practices can come and go within software engineering, either becoming defunct, disproven or otherwise going out of fashion. However, divide and conquer as epitomised by principles such as Single Responsibility has always been a strategy to deliver scaleable and robust solutions.

It is the application of this mindset that lead to the advent of microservices. This technique divides an application into a group of loosely coupled services each offering specific, cohesive and a relatively narrow set of functionalities.

Any architecture or pattern when described succinctly can sound overly simplfied, the description in the previous paragraph doesn't enlighten the reader to the devil in the detail when it comes to the implementation and application of a pattern such as microservices.

This post also won't cover these topics in enough detail to fully arm the reader but it will attempt to point in the right direction.

What is Micro?

The effectiveness of any microservices approach will be reliant on developing the correct segregation of duties for each service, this provides the modularity that is the basis for the success of the pattern. The reason this aspect of the approach is non-trivial is because it is possible for microservices to be too small.

This can create increased cognitive load on those tasked with working on the system as well as creating a ripple effect when changes are required as each service becomes less independently deployable due to its increased dependencies on other services to achieve an outcome.

A good starting point can be to align your initial design to the agents of change within your organisation, this might be areas such as ordering, billing, customer or product discovery. This can enhance the benefit of the scope of change being contained whilst also allowing engineers and the rest of the business to have a common view of the world.

Another more technical analysis can be to asses which is more time consuming, to re-write or to re-factor? A sweet spot for a microservice and an indication of its scope is whether it would be quicker to re-write or to re-factor when a wide reaching change is required.

This isn't to say that it is optimal to be continually re-writing services but the freedom offered by an architecture where this is the case can be a great aid to innovation and the effective implementation of change.

Micro-Communication

Once we have a suite of microservices how can they be drawn together to form a coherent and competent system whilst remaining loosely coupled? Two possible solutions to this conundrum are REST APIs and an Event Bus.

These two different approaches to communication between services fulfil the differences between explicit communication and a loose transmission of information.

On occasion two microservices will need to explicitly communicate, as an example an ordering service may need to communicate with a payment service and require an immediate response. 

In these situations services can present a RESTful API surface to allow this kind of interaction. Intrinsically linked to this approach is the concept of service discovery, there are various approaches that can be taken but essentially this involves microservices registering their capabilities and the APIs they offer to clients to allow a system to form dynamically as new services are introduced.

This kind of communication exists where interactions between services are necessarily pre-defined but sometimes it is desirable and advantageous to allow these interactions to be looser.

An event bus offers this capability by allowing services to broadcast the occurrence of certain events and allowing others to react to those events without having to be aware of the source.

Following a similar example the same ordering service on successful completion of an order might broadcast this as an event so that services that handle communication with customers can take the appropriate action. The loosely coupled nature of this communication allows behaviour to be easily changed without the need for refactoring in the service handling the initial action.       

Micro-Deployment

So now we have our collection of properly sized and loosely coupled services how do we manage their deployment into our environment?

A key element of microservice deployment is independence, a sure sign that a microservices architecture is flawed is when it is necessary to deploy a number of services concurrently. This speaks to strong coupling caused either by a tightly coupled domain or an incorrect segmentation of responsibilities. 

A lack of independence also negates the intrinsic promotion of a CI\CD approach that microservices when applied correctly can offer.

Independence of deployment when combined with a load balanced blue\green approach also creates an environment where changes can be gradually introduced and quickly rolled back if problems present themselves. This is in contrast to a monolithic architecture where a small problem in one area may result in working changes in other areas having to be rolled back.

Microservices is a pattern that comes more with a serving suggestion than a recipe. Its proper application cannot be applied via a cookie cutter methodology. There are certain aspects that are universal but a good architecture will only be designed by understanding how they can be applied to your particular situation whilst keeping an eye on the successful realisation of its benefits. Many of these benefits will enable you to make mistakes by giving you the freedom to reconfigure, refactor and realign your architecture without causing widespread disruption.

The Miracle of HTTP

The web is now all pervasive in the majority of peoples lives, sometimes its explicit when we are surfing the web and sometimes is implicit for example in relation to the rise of the Internet of Things.

The complexity of the implementation of the web is on a scale that can cause you to wonder how this thing always keeps working. The perceived speed of advances in the capability of the web can lead you to think that the technology is constantly evolving and refining.

Whilst this may be true for the technologies that enable us to implement things on the web, many of the underlying technologies that underpin the web and keep it working have remained surprisingly static since its inception.

Once of these technologies is the Hypertext Transfer Protocol (HTTP).

What Did HTTP Ever Do For Me?

HTTP is a request\response protocol that allows clients, for example your desktop browser, to request data from a server, for example the server allowing you to read this blog.

It acts within the application layer of the Internet Protocol suite and handles the transfer of resources, or hypermedia, on top of a transport layer connection provided by protocols such as the Transmission Control Protocol (TCP).

It is a descriptive protocol that allows requests to describe what the user wants to do with the indicated resource, to this end each request uses an HTTP verb such as GET, POST, PUT and DELETE and each response uses status codes such as OK, NOT FOUND, FORBIDDEN to form an exchange that describes the interactions during a session.

Each request and response also contains certain Header Fields that can be used to further describe both the contents of the request\response and the parties sending them.

If you were to see an exchange of HTTP requests between your browser and a web site you would be surprised at how much it is human readable, understandable and intuitive. This speaks volumes for the initial design of the protocol and it's elegant simplicity.

Without HTTP we wouldn't have a way to allow the enormous amount of data on the internet to be queried for, retrieved, uploaded or managed.

The Brief History of HTTP

Given the importance of HTTP you may think that it has been under constant revision and refinement as the use and the size of the web grows.

Actually this is not the case, the first published version of the HTTP protocol (v0.9) was published in 1991. This was followed by v1.0 in 1996 and v1.1 in 1997, with v1.1 of the protocol still being the prevalent driving force behind the majority of web usage.

Over the intervening years there have been clarifications and refinements to v1.1 and the built in extensibility designed into the protocol has allowed improvements to be made, but still the protocol delivering the majority of the benefits we derive from the web has remained largely unchanged since what most would consider the birth of the public use of the internet.

This really does go to show the benefit of designing for simplicity, if you develop robust and descriptive technology this can be the mother of invention when creative people pick up the tool you've provided them and combine it with their imagination.

What The Future Holds

So does all this mean that HTTP is done and dusted with all problems solved? Well no, our use of the web and the nature of the data we channel through it have changed since the 90's and while HTTP is able to service our demands there is scope for improvement.

HTTP/2, derived from Google's experiments with the SPDY protocol, is a major release and update to the HTTP protocol. It focuses on performance and efficiency improvements to adapt HTTP to the ways in which we now use the web.

Officially standardised in 2015 the backwards compatibility of HTTP/2 with HTTP v1.1 allows the adoption of the new protocol to spread without fundamentally breaking the web.

As of 2018 around 30% of the top 10 million web sites support HTTP/2 with most major browsers also providing support for the new version of the protocol.

The substantial effort involved in being able to successfully advance a fundamental protocol such as HTTP whilst not breaking the web speaks to the relative slow progression of new versions of the protocol.

Its easy to take the web for granted without admiring the engineering that originally built the basis for all the future innovation that has subsequently changed all our lives. Anyone that has worked in an technological industry will to speak to the difficulty in getting things right and the normality of deciding this isn't going to work and being forced to start-over.

Whilst it would be misleading to assume this never happened during the development of HTTP, the small number of published versions of the protocol combined with the explosive and transformative impact of the web truly is something to behold and admire.

Next time your using the web, or an app, or talking to your smart speaker tip your hat to the engineering that not only made it all possible but that also provided the stability for the evolution that got us to this point.

The Start-Up Delusion

Many organisations of different sizes have a software engineering function, from a small band of developers to a large team of people covering many specialisms.

As organisations grow they can often pine for the days when they were smaller, looking back on these days and determining this is when they were most efficient, productive and successful.

Sometimes this nostalgia can be through rose tinted glasses, we lose sight of the cause of our previous success and this means we fail in trying to reproduce it.

What form do these delusions take? What mistakes do we make in analysing why things seemed better then?

Need for Speed

When we look back on our days as a smaller team we often refer to the speed that we feel we used to work and produce results. We castigate ourselves for not being able to produce results as quickly as we used to.

But to look at this in terms of speed is to misunderstand the cause of the productivity we now long for. The time taken to produce the code for any given feature does not change between a small team and a large team, the difference comes from the scope of the feature we are trying to implement.

The amount of scope a start-up will include in a feature is often dramatically less than will be attempted by a larger team. Start-ups want, and need, to get code into production quickly, to achieve this edge cases are often dismissed, scaleability is reduced to the next milestone without over reaching for something that is well over the horizon.

Start-ups do not distort the mathematics of what is possible in a certain period of time, instead they embrace the certainty that if time is fixed the only lever that can be adjusted is scope. They are masters in pragmatism, doing just enough to get over the line, as organisations grow adjustments in scope become harder because you have a user base that expects more from you and will have a reduced tolerance for missing functionality.

Different or Better

Disruption is a quality we tend to hold in high regard, this leads to frustration when we feel we held back from being disruptive by our size. I believe a cause of this frustration is a misunderstanding of what it means to be disruptive.

Disruption is often equated with being different, but being different can simply be a pseudonym for being wrong. Disruption isn't just a function of a software engineering department, it isn't about features, its about re-defining a marketplace and creating new ways to deliver value to users.

When your disruption is successful it will naturally start to transition towards normal, others will cotton onto your foresight and begin to copy.

In this situation continuing to be disruptive will become harder, instead you should look to iterate on your initial innovation and extract the rewards of being first.

There is only so many times you can change a market place before you slide into just trying to be different, but being different without an understanding of why that approach or new feature will be better, both in terms of user satisfaction but also in terms of your ability to extract value, will eventually lead to mistakes and you giving up your position as the leader in the market you created.

Successful start-ups just need one good idea that they can run with, there goal is to make noise and attract attention. As an organisation grows focus will naturally change to wanting to realise the potential of all that hard work.

Barriers to Communication

As organisations grow they naturally start to build up processes and communication channels start to become more formal.

In 1967 Melvin Conway coined Conway's Law:

"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."

If organisations do wish to return to their roots as a start-up they would do well to heed the message of Conway's Law.

Whilst each individual new process thats introduced may come from good intentions they have a cumulative effect that degrades both the frequency and quality of the communication between teams.

To return to the topic of speed, this lack of communication in organisations that consist of multiple development teams will put the brakes on the output like nothing else.

This is also a different process to reverse, to have the approach of a start-up isn't just about the engineers mindset your whole organisation must be structured with that goal.

Its tempting to think that start-ups have access to knowledge or magic that enable them to achieve the impossible or work to a different set of rules.

In fact they are a demonstration of factors that we are all aware of and have it under our control to master.

If an organisation truly wants to achieve this hallowed state then it isn't enough to simply tell your employees to think that way, your organisation from top to bottom must embrace these factors and not shy away from them.

Securing the Machine

Security is now a front and centre consideration when designing any software solution, the explosion in the ability of technology to help achieve goals applies equally to attackers as to the good guys.

We have also come to realise that not all threats are external, we have in recent times seen the emergence of the concept of a malicious insider, someone who plays a role in delivering software or a user of it who abuses their access to steal data or otherwise cause mischief.

This means as much consideration needs to be given to the security of the machine producing the software as to the software itself.

Privilege and Roles

For large systems the amount of infrastructure involved in building, testing and hosting it can be substantial, many team members will need various degrees of access to this infrastructure to perform a variety of tasks ranging from the mundane to the complex and far reaching.

Its easy in this situation to end up with super users who have access to do almost anything.

Aside from malicious intent this level of access also places a higher potential cost on any mistakes that may be made.

A role based approach that promotes segregation of duties provides much more protection against any user abusing their access, this can also ensures that multiple people need to be involved to complete more potentially dangerous or impactful tasks.

This approach also makes it easier to compose the correct level of access for a user without having to resort to granting super user status or granting high level access for one particular use case.

The lack of super users is also an advantage if an outside attacker manages to compromise the system or any of its users.

Secrets

Software involves many secrets, service credentials, connection strings and encryption keys to name a few.

An effective infrastructure needs to include a mechanism for managing these secrets, this needs to be both within the development process, so not every developer knows production secrets for example, and also in production to prevent exposure following any attack.

Many solutions are available for these situations, the majority of which rely on the secrets being defined within the environment in which the software is running as opposed to being in the source code itself.

This then allows access to this area of the environment to be strictly controlled and where possible be a one way transaction for users i.e. users enter secrets but only the software that needs them can ever get them out again.

This can even be taking a stage further where secrets are randomly generated inside the environment so no user ever needs to know them.

Audit Trail

Despite all the measures that may be put in place it won't be possible to reduce to zero the possibility of a destructive change, whether that be deliberate or not.

When this happens it is paramount to answer two questions, exactly what has been compromised? And what needs to be done to revert the change?

The first can be answered by having automated audit logs that ensure all changes are automatically recorded in terms of what was changed and by whom. We are used to this with source control systems like Git and this approach is becoming increasingly adopted for infrastructure changes with tooling promoting infrastructure as code.

A second advantage of this kind of tooling is the ease at which changes can be reverted, rather than having to discover and unpick changes they can be reversed much like a commit to a code base might be.

To return to our discussion about privilege and roles, the best defence against any destructive change, deliberate or not, is to ensure that major impactful changes cannot be instigated by individuals and instead require sign-off and validation from others.

An effective audit strategy also need to be able to asses impact, this may be by recording data flowing in and out of a system, recording the errors being generated or unusual system activity that would not be deemed normal.

The purpose of this article isn't to try and make you see potential threats all around your development team, while malicious insiders can be very real you much more likely to be threatened by unintentional incompetence. People make mistakes, limiting the potential security implications of these mistakes is equally as important is defending yourself against someone that intends your system harm.

A sensible approach promoting least privilege, emphasising the importance of secrets and keeping an accurate system of record for changes will all help in both scenarios. Don't see your teams roles as just writing good, and secure, software also see it as developing an equally good, and secure, machine that moves this code through to production.         

Defining Software Architecture

The role of software architecture can sometimes be difficult to define, it can be easy to demonstrate its absence when a project or a system is saddled with technical debt but the impact when its being done well can sometimes be missed because the apparent simplicity it fosters.

However software architecture should not be an isolated pursuit, it should be a tool to achieve the aims of an organisation, when this is applied well the goals of the architect and the goals of the organisation are symbiotic.

It can sometimes be trite to attempt to simplify complex subjects into bullet points or pearls of wisdom, the points made here are not an attempt to do this, they are simply trying to help in providing an explanation for the role of software architecture.

Modularity and Patterns

The creating of software is an expensive pursuit, individuals skilled in its production are an expensive resource. This means that every last drop of value needs to be squeezed from their efforts, in short there is an economic benefit to code re-use. 

Every time a piece of code is re-used the value returned on the original effort to produce it is increased.

Code re-use is only viable when the opportunity for re-use can be seen and the code itself can be cleanly extracted. Good architecture provides structure and patterns for implementation that promote both of these aspects and make it more likely that code, and the value it provides, can be re-used.

Good architecture also provides a framework for code, both old and new, to be stitched together to form a coherent system that can meet objectives. Bad architecture breeds re-writes and duplication where the achievement of value always comes at an expensive cost.

Flexibility

The priorities of a business are subject to change, the predictability of the frequency and direction of that change can vary.

Good architecture can adapt to these shifting sands and demonstrate a flexibility in the things it can achieve.

In part this is accomplished by the same modularity we've previously talked about, when a system is the sum of its parts then these parts can be re-worked and replaced without having to contemplate starting from scratch or undertake major surgery on a code base.

The second, equally important, aspect to flexibility is that this change can be quickly deployed.

It is fast becoming a fact that the effectiveness of a software engineering function can be measured by the frequency in which it deploys code. Against this backdrop an architecture that can't deliver a fast pace and unrestricted frequency of deployments will be deemed ineffective and lacking.

Good architecture wouldn't be overly simplified by being categorised as the smooth implementation of business change. 

Containing Complexity

Writing software is a complicated business, it is so incredibly easy to do it wrong. Coupled with this the businesses and domains that it is attempting to model and facilitate are often equally complex.

Without a good architecture to provide a path through this complexity software can become fragile, ineffective and ultimately a burden on the organisation it is designed to serve.

A good architecture recognises this tendency towards complexity and the problems that can bring. To counteract this it tries to simplify where possible and contain complexity where this isn't possible.

Understanding where complexity lies in a system allows care to be taken when modifying these areas while allowing the speed of development in areas freed from complexity to be increased.

A bad architecture breeds complexity and allows it to put the brakes onto development activity, it makes it more likely that change will bring unintended consequences and acts as a multiplier for these effects having an even bigger impact in the future.

As mentioned at the start of this post software architecture can't be summed up in so few words as we have here, it is an activity that is very difficult to become proficient at and potentially cannot be mastered.

However, having certain goals in mind and being able to explain its benefits can help to garner an appreciation for the discipline and an understanding for the pitfalls of not giving it proper consideration.

Good architecture can be undermined when it takes a complex situation and makes it appear straight forward, sometimes its presence is only felt when bad architecture lays out this complexity for all to see.          

RESTful Experience

Many different technologies have been devised to provide structure to the transfer of data from here to there and back again. These have ranged from the heavyweight to the lightweight.

REST, or Representational State Transfer to give it it's full name, is now the prevalent mechanism when dealing with an API to retrieve or send data, with these APIs being described as RESTful.

What does it mean for an API to be RESTful? Is it simply the sending and receiving of JSON via HTTP requests?

Although REST does have some requirements around the plumbing of how requests are made and received there are also philosophies that go deeper than this traffic management aspect.

Stateless

All the information necessary to process a RESTful request should be contained in the request itself. This is to say that the server receiving the request should not need to use state about the current session in order to process the request.

This puts the power with the client to decide which APIs to call in whatever order, it ensures the API surface is unambiguous with no required knowledge of the order APIs should be called in or any possible side effects.

This statelessness should extend to authentication and authorisation which each request containing the necessary information for both those important factors to be fulfilled.

Its important to realise that this only applies to the state of the session and the processing of the requests, the state of the resources and data being accessed is of course subject to change between requests and will have the concept of state.

Uniform Interface

REST APIs deal in the currency of resources, a resource can be almost any data item, this could represent a customer, a shopping basket, a book or a social media post.

Resources should be uniquely identifiable and have a representation in the system that is descriptive and processable.

Operations on these resources should be via standard HTTP verbs that describe the operation that is taking place.

• GET: Read.
• POST: Create.
• PUT: Update\Replace.
• DELETE: Remove.

The HTTP response codes returned from using any of these requests should also be related to the state of the resource, for example:

• 404: Resource with that unique identifier cannot be found.
• 405: Not allowed, such as when a resource cannot be deleted or modified.
• 409: Conflict, when a resource with that unique identifier already exists.
• And so on...

The paths used when accessing these resources should also be self explanatory and naturally readable.

• GET /api/v1/customer/ - Return all customers.
• GET /api/v1/customer/866823e5 - Return a specific customer.
• GET /api/v1/customer/?lastname=smith - Return all customers with a last name of smith.

All of this structure allows an API to be self discoverable by a user familiar with the resources being represented.

The path can also be used to impose a versioning system, ensuring that when breaking changes must be made to the how the API behaves or the representation of the data being returned that this is non-impactful for existing consumers of the API.

Cacheable and Layered

Much of what we've discussed allows REST APIs to implement certain characteristics to increase performance such as caching.

GET requests within a REST API should be cacheable by default with standard HTTP mechanisms being used to control the life time of the cached data. This helps reduce latency whilst also reducing the strain being placed on the backend sources of the data.

Segregating an API based on resources allows for a certain hierarchy and layered architecture to be put in place.

This lends itself to the micro-services model allowing systems to be composed by narrowly focused components dealing with a particular element of the domain being modelled.

REST has come to dominate the API landscape because the application of a few simple rules greatly simplifies the process of implementing an API as well as reducing the barriers to entry for a consumer to get an up and running with the API.

On occasion it may be difficult to always adhere to the rules we've laid out and it may be the case that an API being RESTful is a metric rather than an absolute. But these situations should be rare and once you have acquired a RESTful eye you will soon become adept at modelling your world according to its guidelines.

Black Swans and Antifragility

All of us who work in software development will have the scars caused by a disastrous update, outage or product launch. These experiences shape the way we approach our roles in the future, we are more attuned to possible catastrophe, planning strategies to deal with things going wrong and positively expect them too.

The Black Swan Theory, postulated by Nassim Nicholas Taleb, deals with the nature of unexpected events, although this is in a wider context then technology there are none the less parallels to the kinds of events as IT professionals we have to react to.

Within this theory a black swan is an event with the following properties:

• It is an outlier, not expected, with past experience not pointing to its possibility.
• It carries an extreme impact.
• Human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.

In software engineering events that could be considered black swans might be sudden increases in the load placed on a system, catastrophic hardware failure or a breach in security.

So are we doomed to the consequences of these events or can we architect our systems to try and cope with the aftermath.

Modularity and Weak Links

Black swan events have an increased, or at least more sustained, impact on complex systems.

Complexity breeds mistakes and causes solutions to become harder to envisage, therefore a strategy for combating complexity can help combat the cause and effect of disastrous events.

Composition by breaking down a systems problems space into multiple simplified chunks can work at many levels, from individual blocks of code to whole sub-systems.

Allowing these blocks to be swappable and having the ability to re-configure and re-organise them not only allows functionality to be easily changed it also allows a non-functioning areas of a system to be quickly fixed or replaced.

The links between modules can transmit stress and failure as well as functionality, the weaker they are the easier they can be broken when necessary.

Redundancy and Diversity

Black swan events related to hardware or service failure become more catastrophic when no alternative is available.

Redundancy and diversity are strategies for ensuring that alternatives do exist to ensure continuity of service. Discussions around redundancy and diversity can get caught up in slightly pedantic arguments, for the purposes of this discussion lets try and simplify.

Redundancy can be viewed as having more than one of a particular resource while diversity can be thought of as having more than one channel for the functionality.

Using the example of databases, redundancy would be achieved by having back-ups, mirroring or replication whereas diversity might be achieved by using multiple cloud providers for hosting data.

Testing and Probability

No strategy for dealing with failure can be said to be fully implemented unless it has been proven to be effective via testing.

This verification or testing can be as simple as ensuring a database backup is valid and can be restored from , it can also be as sophisticated and automated as the chaos monkey techniques employed by companies such as Netflix.

There have been many instances of companies who believe they have plans to cover every eventuality being left floundering once disaster strikes.

If we were to list all the possible disasters that could befall us they would be numerous. each having a probability and a level of impact on our system.

These two factors need to be balanced along with the cost and effort of mitigation. Whenever protecting against the scenario is relatively straightforward this should be implemented regardless of likelihood.

For everything else the impact of the event should be balanced against its probability, in these scenarios don't be too quick to write off an event as improbable, if it would bring your system to its knees then its worth considering having a strategy.

Trying to asses the potential failures in your system is a good way of assessing your architecture and infrastructure, highlighting technical debt or areas for improvement. If at the same time you can develop strategies to try and reduce fragility then you be able to sleep more easily in your bed.