Security Pitfalls

 

Whether or not a software application will be exposed to security threats is mostly a case of when rather than if. We can often be tempted to think that we need to be more concerned by the attention of a sophisticated attacker with a large arsenal of weapons. However it's unfortunately the case that most threats are a result us failing to learn from previous mistakes.

The OWASP top ten, largely seen as the industry standard guide to prevalent security threats, has consistently been dominated by well known attacks. These vulnerabilities have been with us for a long time and have well known mitigations, yet they continue to be seen in countless applications.

In this article I won't go through all of the OWASP top ten, instead I will highlight some of the categories that the threats fall under and the mitigations that should always be at the forefront of your mind.

Don't Trust The Outside World

Still by far the most commonly observed vulnerabilities relate to applications putting too much trust into input received from the outside world. These injection attacks fall broadly into three categories, SQL Injection, Cross Site Scripting (XSS) and Cross Site Request Forgery (XSRF). 

SQL Injection can come in many forms but occurs whenever user input is directly included in SQL statements without being sanitised. This either causes more data to be exposed to the caller than they are entitled to view or allows them to run potentially destructive commands against the database.

XSS occurs when an attacker is able to inject client script into web pages being viewed by other users. XSS scripting attacks are often categorised as Non-Persistent (reflected) and Persistent. An example of a reflected attack might be when an attacker is able to manipulate the query string submitted to a page that is then in some form included in the page contents causing the script to execute within the victims browser. An example of a persistent attack might be if an attacker is able to include client side script into data that is stored by the server and subsequently rendered in multiple victims browsers, this could be via a message forum or some kind of profile page that is viewable my multiple users.

XSRF occurs when an attacker is able to get a victim to submit client side script to a page that trusts the user. This will generally involve trying to get a user to click a link or otherwise submit data to a site they are currently logged into. An example might be sending a user a link to a site that includes potentially damaging query string parameters if the user has administrator privileges.

The protection against all these attacks is to distrust all input from outside your application. Never assume that a user will only ever submit data in the intended format, sanitise all inputs and block where necessary where script or potentially executable code is detected.        

Not Implemented Here

Many well defined and proven techniques for securing applications exist, these range from encryption and authentication to data integrity checks. Often these techniques rely on necessarily complex logic and mathematics.

Although sometimes flaws in this logic can be found it is far more common for attacks to exploit mistakes in their implementation rather than weaknesses in the core algorithm. Many of these techniques can be interesting to implement, and many engineering teams will frequently have a "not implemented here" attitude only wanting to rely on code written internally.  

Security is not an area of coding to have this attitude towards, unless you are an expert in the field it will be very likely that you will make a mistake in its implementation. These mistakes may not be immediately visible and may not relate to the core application of the technique but none the less they may give an attacker the crack in the door that they are looking for.

Trust in the implementations written by experts, even they will have to release patches when small mistakes are uncovered but their expertise and ability to verify their work will put you in a much better position than if you try and go it alone.  

Undefined Behaviour

Many categories of attacks revolve around getting applications to behave in an unexpected way. These range from buffer overflow and arbitrary code execution to memory corruption and insecure deserialisation. 

Quite often these attacks exploit the very low level details of code execution, also thankfully many built in defences such as stack canaries and Address Space Layout Randomisation (ASLR) exist to try and protect you against them being exploited against your application.

Despite this you still need to be alert to when your application is exhibiting unintended behaviour. This is true even if the observed behaviour is not considered a bug. It may be that the apparent side effects are benign but it's important you always understand why your application behaves the way it does.

Attackers will very often begin probing your system by doing unusual or unexpected things to view your systems response. They will look for ways to exploit the behaviour they observe and possibly chain together multiple instances of unexpected behaviour to get the result they want.

Always take the view that if something isn't supposed to happen then the system should be configured to not allow it to happen regardless of whether or not the consequences of it happening seem dangerous. In a similar manner to injection attacks show a healthy distrust for the world outside your application and its adherence to how your application is supposed to be used.

We sometimes assume that all potential attackers are extremely sophisticated. Although such attackers exist there are many that continue to exploit relatively simple and well known attack vectors. Because these attacks are well known and have existed for some time they also have well proven mitigations. Try to ensure that your application doesn't fall prey to these defensible attacks and always be mindful of the OWASP Top Ten. 

Docker Basics

 

In my previous post I covered the explanation of Kubernetes terminology that finally helped me gain an understanding of its purpose and operation. In this post I decided to delve one layer deeper to cover the basics of Docker.

As with my previous post I must add a disclaimer that I am not a Docker expert so the descriptions below aren't meant to be authoritative, but they are the explanations that make sense to me and aided my initial understanding of the topic.

Containers vs Virtual Machines

Prior to the advent of containers, of which Docker is one implementation, the most common deployment mechanism in cloud computing was virtual machines.

Virtual machines provide full hardware virtualisation by the means of a hypervisor, they include a full operating system install along with abstractions of hardware interfaces. Software is installed on the machine as if it was a normal physical server, however since the hypervisor can support multiple virtual machines on a single physical server they enable the available resources to be maximised.

Containers do not try to provide an abstraction of a whole machine, instead they provide access to the host operating system kernel whilst also allowing each individual container to be isolated from each other. Not only is this a more efficient use of resources but it allows software to be packaged in such a way as to include all required dependencies in an immutable format via container images.

Daemon and Clients

Docker follows a client server architecture. The Docker daemon provides an API for interacting with Docker functionality and also manages containers running on the host machine.

The Docker client provides users with a mechanism for interacting with the Docker daemon. It allows users to build, run, start and stop containers as well as many other commands for building and managing docker containers.

A second client called Docker Compose allows users, via a YAML file, to define the make up of a whole system comprising multiple containers. It defines which containers should run along with various configuration information related to issues such as networking or attachment to storage.

Images, Containers and Registries

A Docker image defines an immutable template for how to build a container. A powerful aspect of docker is that it allows images to be based on other images creating a layered approach to their construction. For example you may define an image for your container to start with an image for the operating system you want to work with, then add the image of the web server you want to use followed by your application. These steps are defined in a Docker File that provides the instructions on how each layer should be built up to define the container image.

A container is a running instance of an image. When running a container you define the image you want it to be based on plus any configuration information it might need. The important aspect is that the container contains everything necessary for the application to run. As opposed to deployment to a virtual machine that might rely on certain dependencies already being present a container is self contained and therefore highly portable.

A Docker registry is a means for storing and sharing images, it acts like a library for different container images that can be updated as new versions of the container are defined. When using Docker Compose to define the make up of a system you will often specify the version of a container to run by pointing at a particular version of a container within a registry.

Clearly a technology as complex as Docker has many intricacies and complexities that I haven't covered in this post. However more advanced topics are always easier to approach once you have sound understanding of the basics. Never try to tackle the task of understanding everything about an area of technology, instead see it as a journey and accept it may take some time for the knowledge to take hold. The explanations I've provided in this post helped me on that journey, hopefully they can help you too.

Kubernetes Basics

 

As containerization became a more and more popular deployment choice it was natural that tools would need to be developed to manage systems that may comprise large numbers of containers each focusing on different aspects of functionality.

Kubernetes is one such tool providing an orchestration layer for containers to handle everything from lifecycles and scheduling to networking.

It took me quite some time to get to grips with the concepts behind Kubernetes, I think this was largely because the definitions and explanations online can vary greatly. Presented below are the definitions that finally enabled me to understand what Kubernetes is trying to do and how it goes about achieving it.

I am not a Kubernetes expert so by no means am I presenting these explanations as definitive, all I hope is that they help someone else start their journey towards understanding the purpose and operation of Kubernetes.

Nodes

Nodes are the virtual machines that make up a Kubernetes cluster that can run and manage applications.

One node is designated the master and implements the control plane functionality to maintain and manage the cluster. Other nodes orchestrated by the master run the applications that the cluster is responsible for. Which nodes run which applications will vary depending on the nature of the applications alongside constraints such as CPU and memory usage.

Pods

A pod is the smallest deployment unit of Kubernetes. It can run one or more containers, since Kubernetes treats the pod as a single unit when it is started or stopped then so are all the containers within it.

Whilst in theory a pod could be comprised of multiple container types it is a common pattern for there to be a one to one relationship between a pod and container, for example to provide an API or access to an underlying datastore.

Sometimes addtional container types may be added to a pod to provide cross cutting concerns to the main container. This will quite often follow the sidecar pattern and be related to functionality such as acting as a sink for logging or providing a proxy to a network interface.

Deployments

We said earlier that one of the functions of Kubernetes is to manage lifecycle and scheduling concerns, a deployment is how we indicate to Kubernetes how these things should be dealt with.

A deployment might define:

• A pod and an associated container image.
• That a certain number of instances of the pod should be running at all times.
• CPU and memory requirements for each pod, this may also involve setting limits for the amount of resource pods should be allowed to consume.
• A strategy for how an update to pods should be managed.

Kubernetes will attempt to ensure that the deployment always matches the state described. if your application crashes then an unresponsive pod will be swapped out for a fresh one, if the amount of resource a pod is consuming increases then an existing pod may be moved to a node with more available resource.

When you update your deployment to reference a new version of your container then Kubernetes will also manage the transition from the existing pods to new pods that are running your updated container.

Services

Now with our application running in containers within pods we need a way for other applications in the cluster to be able to take advantage of it.

We wouldn't want pods to have to directly communicate with other pods, not only would this cause problems from a networking point of view since pods can come and go, but also we need a mechanism to ensure load is distributed across all the pods running the application.

Services within Kubernetes act a bit like a load balancer, they sit above a group of pods providing a consistent route to the underlying functionality. When a pod requires functionality implemented by another pod it sends a network request to a DNS entry defined by Kubernetes that represents the service endpoint.

Pods can now be freely added and removed from the service and pods don't need to be aware of each other in order to make use of their functionality.

Ingresses

Services provide an internal route to functionality provided by pods but it's likely that we will want to make some of this functionality available outside the cluster.

An ingress exposes an HTTP endpoint outside of the cluster that points at an internal service. In this sense an ingress acts like a reverse proxy onto the internal load balancer provided by the service allowing applications outside the cluster to invoke the underlying functionality.

An ingress can also provide other functionalities such as path based routing or SSL termination to present a consistent and secure interface to the world outside the cluster.

This has been a whirlwind tour of the basic concepts within Kubernetes, it is by no means exhaustive. I hope it enables you to understand the purpose of Kubernetes to aid your learning of the intricacies of an actual Kubernetes cluster. The devil is always in the detail but understanding of the fundamental concepts provides a solid bed on which to build the rest of your understanding.

Creating Chaos

 

In software development chaos engineering is the process of running experiments against a system in order to build confidence in its ability to withstand unexpected conditions or changes in environment.

First developed in 2011 by Netflix as part of their adoption of cloud infrastructure, it's underlying principles have been applied to many situations but typically experiments include things such as: 

• Deliberately causing infrastructure failures, such as bringing down application servers or databases.
• Introducing less favourable network conditions by introducing increased latency, packet loss or errors in essential services such as DNS.

In an attempt to automate these experiments Netflix developed a tool called Chaos Monkey to deliberately tear down servers within its production environment. The guarantee that they would see these kinds of failures helped foster an engineering culture of resilience and redundancy.

We may not all be brave enough to run these experiments within our production environment but if we choose to experiment in the safety of a test environment then what principles should be following?

Steady State Hypothesis

A secondary advantage to chaos engineering is the promotion of metrics within the system. If you are to run automated experiments against your system then you must be able to measure their impact to determine how the system coped. If the system behaviour was not observed to be ideal and changes are made then metrics also act as validation that the situation has improved.

Before running an experiment you should define an hypothesis around what you consider the  steady state of your system to be. This might involve error rates, throughput of requests or overall latency. As your experiment runs these metrics will indicate if your system is able to maintain this steady state despite the deterioration in the environment.

Vary Real World Events

It's important that the mechanisms you use to degrade the environment are representative of the real world events your system might have to cope with.  We are not looking to simulate an event such as server failing we are actually going to destroy it.

How you choose to approach the make up of the failures being introduced is likely to depend on the impact such an event could potentially have and\or the frequency at which you think such an event might occur.

The important consideration is that there should be some random element to the events. The reason for employing chaos engineering is to acknowledge the fact that for any reasonably complicated system it is virtually impossible to accurately predict how it will react. Things that you may have thought cannot happen may turn out to be possible.

Automate Continual Experiments

As you learn to implement the principles of chaos engineering you may rely on manual experimentation as part of a test and learn approach. However this can be an intensive process, the ultimate goal should be to develop the ability to run continual experiments by introducing a level of automation to the experiments.

Many automated tools, including Chaos Monkey, now exist to aid this type of automation. Once you have an appreciation on the types of experiments you want to run, and are confident your system produces the metrics necessary to judge the outcome, then these tools should be used to regularly and frequently run experiments.

The principles of chaos engineering are finding new application in many different aspects of software development, including topics such as system security for example by deliberately introducing infrastructure that doesn't conform to security best practices to measure the systems response and it's ability to enforce policy.

Not every system will lend it's self to a chaos engineering approach, for example an on-premise system where servers are not as easily destroyed as is the case in the cloud may limit options for running experiments. There also needs to be consideration as to the size of the potential blast radius for any experiment and a plan for returning to previous environmental conditions should the system fail to recover.

Your system's reaction to a large number of the experiments you run will likely surprise you in both good and bad ways. As previously stated for a system of any reasonable complexity it is unrealistic to expect to have an accurate view of how the system works under all possible conditions, the experiments you run are a learning exercise to try and fill in these gaps in your knowledge and ensure you are doing all you can to make sure your system performs the role your users want it to.

Everything in the Repo

 

Interaction with source control is a daily task for most developers, the idea of not managing source code in this way would seem unthinkable. The advantages that effective source control can give have lead many to look to include more of the material and information required to write, deploy and run software to be part of the same standard development practices.

This idea has gone by many names, at WeaveWorks they have coined the term GitOps. Although in their description of the process they assume a container based deployment using Kubernetes, the principles they define for an effective GitOps strategy could be applied too many different deployment scenarios.

The Entire System Described In The Repository

No matter the nature of the software you are writing it will need to be built and deployed. To achieve this most teams will have defined CI/CD pipelines to build and deploy the code to various deployment environments.

A GitOps strategy ensures that these pipelines, and the infrastructure they serve, are declared alongside the source code. By cloning the repo you should have access to all the information required to understand the system.

The Canonical Desired System State Versioned in Git

Once your entire system is under source control then you have a single source of truth for its current state and also for any previous state in the past. Changes to CI\CD and infrastructure are tracked alongside the code of the application allowing you to move back and forth in time and maintain a working system.

The most obvious advantage this gives is in dealing with an unintended breaking change to the application related to CI\CD or infrastructure changes. Without these things being under source control you have to follow a painful process of trying to understand the changes that have been made and defining a plan for undoing these changes or trying to fix forward. A GitOps strategy reduced this task to something as simple as a Git Revert command or redeploying from a previous release branch.

Approved Changes That Can Be Automatically Applied To The System

When applying changes to an applications source code developers are used to going through a review process before changes are applied. This may involve a peer review by another developer and\or by following a shift left strategy it may involve a series of automated tests to ensure correctness.

By following a GitOps strategy these process can be applied to changes to CI\CD and infrastructure as well as code. As with any shift left strategy this reduces the chances of the team being impacted by changes that may inadvertently break pipelines, result in a non-working application after deployment, or unintentionally increase costs due to a misconfigured infrastructure change.

Software Agents to Ensure Correctness and Alert on Divergence

Your ability to follow this principle will vary based in your deployment model, but in essence by having source control be the source of truth for your system it enables software to automatically detect when this doesn't match the reality of your deployment and make the appropriate changes.

Not only do this mean you get to see your changes reflected in your environments at a faster pace it also decreases the time to recover from human error once the bad change set has been reversed.

When looking to apply these principles you will have to analyse how they can best be implemented for your application and the environments you deploy into. As with most philosophies there is no one size fits all approach, the degree to which you are applying these principles maybe an intangible measure rather than an absolute. But as always an appreciation for the benefits is the key, and using this to guide your approach and maximise your effectiveness.

Cryptographic Basics

 

Cryptography while essential in modern software engineering is a complicated subject. While there is no need to gain an understanding of the complex mathematics that underly modern cryptographic techniques, a well rounded engineer should understand the available tools and the situations in which they should be used. 

What is presented below is by no means an in depth examination of cryptography but is a primer into the topics that are likely to come up as you try to ensure your code base is well protected.

Encryption vs. Hashing

Encryption and hashing are probably the two primary applications of cryptography but the use case for each is different.

Encryption is a two-way i.e. reversible process. In order to protect data either at rest or in transit encryption can be applied such that only those that have the corresponding key can view the underlying data. Encryption is therefore used to protect data in situations where access to the data needs to be maintained but also protected from unauthorised disclosure.  

Hashing is a one-way i.e. irreversible process. Taking data as an input a hashing algorithm produces a unique digest that cannot be used to get back to the original data source. Hashing is therefore used in situations where either the integrity of data needs to be verified or where the data being stored is very sensitive and therefore only a representation of the data should be stored rather than the data itself. A common example of the latter would be the storage of passwords.

Stream vs Block Ciphers

Encryption is implemented by the application of ciphers, algorithms that given an input (referred to as plain text) will output the same data in an encrypted form (referred to as cipher text).

These ciphers are often categorised based on how they view the input data.

Stream ciphers view the data as a constant stream of bits and bytes, they produce a corresponding stream of pseudo random data that is combined with the input data to produce the encrypted output. A block cipher divides the data up into fixed size blocks, using padding to ensure the overall size of the encrypted data is a whole number of these fixed sized blocks. 

Stream ciphers have proven to be complicated to implement correctly mainly because of their reliance on the true randomness of the generated key stream. Because of this the most popular ciphers are mostly block ciphers such as the Advanced Encryption Standard (AES).

While block ciphers are now the most widely used attention also needs to be paid to the mode they are used in. The mode largely controls how the blocks are combined during the encryption process. When using Electronic Code Book (ECB) mode then each block is encrypted separately and are simply concatenated to form the encrypted output. While this may seem logical it leads to weaknesses, when separate blocks contain the same data they will lead to the same output which can present an advantage to a possible attacker. For this reason other modes such as Cipher Block Chaining (CBC) combine each block as the algorithm progresses to ensure even if blocks contain the same data they will produce different encrypted output.

Cryptographic Hashing

As we discussed earlier a hashing function is a one-way function that produces a unique digest of a message. Not all hashing algorithms are explicitly designed for cryptographic purposes. 

A cryptographic hashing function should have the following properties:

• It should be deterministic, meaning the same input message will always lead to the same digest.
• It should be a fast operation to compute the digest of a message.
• It should be computationally infeasible to generate a message that gives a specific digest.
• It should be computationally infeasible to find two messages that produce the same digest.
• A small change in the input message should produce a large change in the corresponding digest.

When an algorithm has these qualities it can be applied to provide digital signatures of Message Authentication Codes (MACs) to protect the integrity and authenticity of data either at rest or in transit.

We said earlier that there is no need to to understand the complex mathematics behind these cryptographic techniques, to take this a step further it's important that you don't attempt to understand or implement these techniques yourselves. The complexity involved means the likelihood of making a mistake in the implementation is high, this can lead to bugs that can be exploited by attackers to undermine the security you are trying to implement.

Instead you should view cryptography as a tool box providing implements you can use to protect you and your users, the important thing to learn is which tool should be used for which job and become and expert in its application.

Transactions and Concurrency

 

The majority of applications will make use of some kind of data storage solution. In order to maintain the integrity of this data storage a large number of solutions will involve the concept of transactions in order to manage updates and modifications.

Intertwined with the concept of transactions is that of concurrency, executing transactions sequentially may ensure the protection of the data store but the application would be severely hampered by the lack of throughput.

In order to strike a balance between performance and consistency several strategies have evolved to deal with concurrently executing transactions.

ACID Transactions

Before we deal with concurrency we should first define the properties of a transaction. A transaction represents a set of instructions to run against the data store. Generally this will involve modifications to the data being stored and may result in data being returned to the caller. 

In order for transactions to be executed whilst maintaining the integrity of the data store their implementation should follow certain rules. These rules are often characterised by the acronym ACID:

Atomicity: The instructions being executed within the transaction may have side effects on the data being stored. The execution of a transaction should be atomic in nature meaning either all side effects persist or none of them. This leads to the concept of transactions being "rolled back" should an error occur during execution.

Consistency: All transactions should leave the data store in a consistent state. The definition of consistency will vary between data stores but transactions should not leave the data in an unknown or inconsistent state according to whatever rules may be in place for the data set in question.

Isolation: Transactions should not be aware of each other or otherwise interact. Any errors that may occur in a transaction should not be visible to or affect other transactions. 

Durability: Once a transaction has been successfully executed  and "committed" to the data store then its effects should be persistent from that moment on regardless of any subsequent errors that may happen within the data store. 

Concurrency Problems

So if we ensure that transactions follow the ACID rules why do we need further strategies for dealing with concurrency? Despite the ACID rules it is still possible for transactions to inadvertently cause errors when they are running concurrently against the same data set.

These problems can be intricate depending on the data being stored but some examples include:

Lost Update: Two transactions both operate on the same data item setting it to different values, this causes the update from the first transaction to be lost following the execution of the second transaction.

Dirty Read: Transactions read a value that is later rolled back following the transaction that originally set that value being aborted.

Incorrect Summary: A transaction presents an incorrect summary of a data set because a second transaction alters values while the first transaction is creating the summary.

Concurrency Control

The impact of the problems described in the previous section may vary between applications. This leads to a variety of strategies for trying to lessen or eradicate them from impacting the performance or correctness of the application. We won't here go into the detail of their implementation but they broadly fall into three categories:

Optimistic: An optimistic strategy assumes the chances of transactions clashing is low. Therefore checks on consistency and isolation are delayed until just before a transaction is committed allowing a high level of concurrency. If it turns out a problem has occurred then the transaction must be rolled back and executed again. 

Pessimistic:  A pessimistic strategy assumes the chance of errors is high and transactions should be blocked from executing concurrently if there is a possibility it could be the cause of an error.

Semi-Optimistic: A semi-optimistic strategy attempts to find a middle ground where transactions that appear safe are allowed to execute concurrently but transactions that appear to carry a risk are blocked from doing so. 

Which strategy you choose is a balance between performance and consistency of data. An optimistic approach will provide higher performance by allowing a higher level of concurrency but with the possible overhead of transactions needing to be re-executed if a problem does occur. A pessimistic strategy will offer higher protection against errors but the blocking of concurrency, for example by table locking in a database, will reduce throughput.

Choosing the correct strategy will vary depending on the nature of your data set and the transactions you need to perform on it. Understanding the impacts and benefits of each type of strategy may help you develop your schema or approach to operating on the data to make the most of each approach where appropriate.

Sometimes it will be obvious you are using the wrong strategy, you may see a high level of transactions being aborted and re-run, or you may be suffering from a lack of throughput because of an overly pessimistic approach. As with most things in software engineering it can be a grey area to decide which strategy is best for your needs but being armed with the costs and benefits of possible strategies will be invaluable in enabling you to make a choice.      

Anaemic Models and Domain Driven Design

 

Domain Driven Design (DDD) is an approach to software development where the classes ,including their methods and properties, are designed to reflect the business domain in which they operate.

Some would argue this was the whole motivation behind Object Orientated Programming (OOP) in the first place, however many code bases don't actually take this approach. Either classes are written to reflect the internal structure of the code rather than the domain, or they tend to be operated upon rather than performing the operations themselves.

This has lead to advocates of DDD defining anaemic models as a prevalent anti-pattern.

Anaemic Models

An anaemic model can be thought of as a class that purely acts as a container for data, typically they will consist solely of getter and setters with no methods that perform operations on the underlying data.

The reason this is often seen as an anti-pattern is that this lack of inherent domain logic allows the model to be in an invalid state, whether or not this invalid state is allowed to affect the system as a whole is dependent on other classes recognising this invalid state and either fixing it or raising appropriate errors.

The fact this logic exists in other classes also raises the possibility that the all important business logic of the domain is scattered across the code base rather than being in a central place. This acts as a barrier to fully understanding the logic of the domain and can easily lead to bugs when refactoring of the code base is undertaken based on these misunderstandings.

The driver behind these anaemic models is often strict adherence to principles such as the Single Responsibility Principle (SRP) driving a want to separate representation of data from the logic that acts upon it.

DDD's answer to these issues is to centralise both the storage of data and the logic that acts upon it in the same object.

Aggregate Root

An aggregate root acts as a collection of objects all bound by a common context within a domain. As an example an aggregate root representing a customer might contain objects representing that customers, address, contact details, order history, marketing preferences etc.

The job of the aggregate root is to ensure the consistency and validity of the domain by not allowing external objects to hold a reference to or operate on the data in its collection. External code that wishes to operate on the domain must call methods on the aggregate root where these methods will enforce the logic of the domain and not allow operations that would put the domain in an invalid state.

As well as ensuring validity the aggregate root also acts as central documentation for the domain and the associated business logic, aiding understanding of the domain and allowing safer refactoring.

No Anaemic Models?

So should a code base never contain anaemic models? There is a practicality argument indicating that they cannot always be avoided.

Strict adherence to domain rules does come at a price, hydration of these models from API responses or data storage is often complicated by not being friendly to standard deserialisation or instantiation from the execution of a query. This will often lead to the need for anaemic models to act as Data Transfer Objects (DTOs) in these situations to bring the data into an application before then applying it to a domain.

The important point here is that not all models are designed to represent a domain or have business rules associated with them, some models role is simply to act as a bucket of information that will be processed or transformed into a domain at some later point. In these situations taking a DDD approach would add extra complexity with no benefit.

Recognising the difference between these types of models is key to choosing the right approach. This will come from a strong understanding of the domain in which your software operates in, recognising the domain contexts this leads to and ensuring these are implemented in the proper way. Other models in your code that exist purely to ease the flow of data through the system can be implemented in a more relaxed manner.

It is very easy for developers to become distant from the domain their code operates in, this doesn't make them bad engineers but it does hinder their ability to properly model their business in the code base. Make an attempt to understand your business domain and you'll surprised how it improves your code.

Effective Logging

 

Every developer reading this will have on many occasions been knee deep in an applications logs while trying to investigate an issue in the code base. 

Being able to diagnose and resolve issues by reviewing logs is an important skill for any developer to acquire. However the application of this skill comes not just in being able to review logs but also in knowing what to log.

Presented below are a few ideas on best practices that can increase the effectiveness of your logs. Knowing exactly what to log will vary from application to application, but regardless of the nature of your code base there are certain steps that can be taken to increase the value of your logs.

Context

At the point of creating a log entry clearly the most important piece of data is the application state or request etc that you are logging. However the context that surrounds that entry can be equally important in diagnosing issues where causation is less apparent.

Sometimes you will look at logs and the problem will be obvious, the request is wrong or the response clearly indicates an error. However more often what is causing the problem is less clear and will depend on the context around the application at that time.

To be able to have a chance of getting to the bottom of these sorts of issues as much addtional context as possible should be added to each log entry. Certain contextual information is obvious like the username associated with a request but any addtional information you can collect alongside the main log entry could prove invaluable in spotting patterns.

Many logging frameworks have support for collecting this kind of contextual data, providing this can be done in a structured manner to not create noise in the logs then a good piece of general advice would be to log any additional context that it is possible to gather.

Correlation

Most applications of any complexity will have several links in the chain when processing a request. Information will flow upstream and downstream between different applications and systems, each one a potential cause of failure.

If you are only able to look at each applications logs in isolation without being able to tie a thread through each to determine the path the request took, then the amount of time taken to resolve issues is going to be extended.

To address this it's important to add an element of correlation between all log entries. This can be as simple as GUID that is passed between each application and added as contextual information to all log statements. This GUID can then act as a key when searching the logs to provide all the log entries that are associated with an individual request.

This will enable you to replay the journey the request took through your infrastructure and determine at what point things started to go wrong.

Structure

On a few occasions in this post we have mentioned having to search through an applications logs. When all you are presented with when reviewing logs is a wall of text it is very easy to induce a snow blindness that stops you from being able to garner any useful information.

If you log in a structured manner then it enables logs to be searched in more sophisticated ways than simply looking for certain snippets of text.

Many technologies exist for providing this log structure depending upon your technology platform. In general these framework rather than simply logging text will log data using a structured format such as JSON. Tools can then load these JSON entries and present a mechanism for searching the logs based on the kind of contextual information we have previously discussed.

Like so many aspects of coding that don't directly relate to the functionality users are consuming, logging can very often be an afterthought. This is ultimately self defeating since no application is perfect and you are certain to rely on logging on many occasions to resolve the issues caused by this imperfection.

The techniques described in this post are an attempt to increase the value provided by your logs to ensure when you inevitably need to review them the process is less stressful and as effective as possible.

Is TDD Dead?

 

Test Driven Development (TDD) has long been viewed as one of the universal tenets of software engineering by the majority of engineers. The perceived wisdom being that applying it to a software development lifecycle ensures quality via inherent testability and via an increased focus on the interface to the code under test and the required functionality.

In recent years some have started to challenge the ideas behind TDD and question whether or not it actually leads to higher quality code. In May 2014 Kent Beck, David Heinemeier Hansson and Matin Fowler debated TDD and challenged its dominance as part of software engineering best practices.

The full transcript of their conversation can be found here: Is TDD Dead?.

Presented below are some of my thoughts on the topics they discussed. To be clear, I am not  arguing that TDD should be abandoned. My aim is to provoke debate and try to understand if totally adherence to TDD should be relaxed or approaches modified.

Flavours of TDD

Before debating the merits or otherwise of TDD it's important to acknowledge that different approached to it exist. Strict adherence to TDD implies writing test first and using the so-called red-green refactor technique to move from failing tests to working code.

I think large numbers of teams who would purport to follow TDD will regularly not write tests first. Engineers will often find themselves needing to investigate how to implement certain functionality, with this investigation inevitability leading to writing some or all of an implementation prior to considering tests.

TDD as being discussed here applies to both scenarios, a looser definition of TDD would simply define TDD has an emphasis on code being testable. Many of the pro's and possible con's being discussed would apply equally whether or not tests were written prior to implementation or afterwards.

Test Induced Design Damage

Perhaps the most significant of the con's presented about TDD is that of test induced design damage. 

Because discussions around TDD tend to focus on unit testing then adopting a TDD approach and focusing on testability tends to focus on enabling a class under test to be isolated from its dependencies. The tool used to achieve this is indirection, placing dependencies behind interfaces that can be mocked within tests.

One of the principle causes of test induced design damage is confusion and complication that comes from excessive indirection. I would say this potential design damage is not inherent in the use of indirection but is very easy to accidentally achieve if the interface employed to de-couple a class from a dependency is badly formed.

A badly formed interface where the abstraction being presented isn't clear or is inconsistent can have a large detrimental effect on the readability of code. This damage is very often enhanced when looking at the setup and verification of these interactions on mock dependencies.

Aside from testability another perceived advantage to indirection is the ability at some later point to change the implementation of a dependency without the need for wide spared changes in dependent code. Whilst these situations certainly exist perhaps they don't occur as often as we might think.

Test Confidence

The main reason for having tests is as a source of confidence that the code being tested is in working condition. As soon as the confidence is eroded then the value of the tests is significantly reduced.

One source of this erosion of confidence can be a lack of understanding of what the tests are validating. When tests employ a large number of mocks, each with their own setup and verification steps, it is easy for tests to become unwieldy and difficult to follow.

As the class under test is refactored and the interaction with mocks is modified the complexity can easily be compounded as engineers who don't fully understand how the tests work need to modify them to get them back to a passing state.

This can easily lead to a "just get them to pass" attitude, if this means there is no longer confidence that the tests are valid and verifying the correct functionality then any confidence that the tests passing means we are in a working state is lost.

None of this should be viewed as saying that unit tests or the use indirection are inherently bad. Instead I think it is hinting at the fact that maybe the testability of code needs to be viewed based on the application of multiple types of tests.

Certain classes will lend themselves well to unit testing, the tests will be clear and confidence will be derived from them passing. Other more complex areas of code maybe better suited to integration testing where multiple classes are tested as a complete functional block. Providing these integration tests are able to test and prove functionality this should still provide the needed confidence of a working state following refactoring.

So many aspects of software engineering are imperfect with no answer being correct 100% of the time. Maybe this is also true of TDD, in general it provides many benefits but if it can on occasion have a negative impact maybe we need to employ more of a test mix so that our overall test suite gives us the confidence we need to release working software.

API Toolbox

 

The increasing application of the Software as a Service (SaaS) delivery model means that APIs are the regular means by which we interact with the services that help us write software. Even if APIs aren't the primary mechanism by which we consume the service it is increasingly common place, and potentially even expected, that an API surface will be available to analyse metrics and data related to consumption.

API is a broad term with many different approaches to implementation available to us. Sometimes changes in technology are related to trends or views on implementation correctness, such as migration from SOAP to REST being driven by the desire for a more lightweight representation of data.

However sometimes technology choice is driven by the nature of the API being implemented, the data it is expected to convey and the make-up of the likely consumers. Presented below are some of the technologies available along with the circumstances that might drive you to choose them to implement your API.

REST

REST is still by far the most common approach to implementing an API. Characterised by its use of HTTP verbs and status codes to provide a stateless interface along with the use of JSON to represent data in a human readable format, it still represents a good technology choice for the majority of APIs.

Difficulties in consuming REST are often first and foremost caused by an unintuitive surface making APIs difficult to discover or by an incoherent data model making the data returned from the API hard to work with and derive value from.

In recent years other technologies have gained transaction as alternatives to REST. I believe these do not represent alternative implementations in all circumstances, instead they look to improve upon REST in certain situations.

GraphQL

The majority of APIs relate to the exposure of data from underlying data sources. The richer the data sources the more there becomes a risk that a users will become flooded by data.

REST APIs are built around the concept of a resource, you make a request to return a particular resource and the entire resource is returned to you. With a large data model this can increase noise in the data when an entire resource is returned when only a small sub-set of data was required. This not only means bandwidth is wasted by returning unnecessary data items but performance may be further impacted by potentially making unnecessary additional calls to downstream systems.

A complex data model can also make it harder to intuitively discover what data items are available.

To address some of these issues Facebook developed GraphQL providing a language for querying data from an API. Having a query language means only the data items that are required can be requested, reducing the amount of data returned along with potentially reducing the work the API must do to acquire the entire resource. A query based approach also provides an element of discoverability and schema identification.

To a certain extent it is possible to achieve a queryable interface using REST, by use of path parameters and the query string, but GraphQL increases the flexibility that can be provided.

gRPC

Because REST will in most circumstances represent data using JSON under certain circumstances it can consume larger amounts of bandwidth than is strictly necessary. In the majority of cases the impact of this will be minimal but for certain consumers, most notably IoT devices, this can have a significant impact.

To address this technologies such as gRPC represent data in a binary format, this means only the minimum amount of bandwidth is consumed. gRPC also moves away from API endpoints in favour of remote method calls on both server and client.

Both these aspects make it ideal for scenarios where data must be transferred using minimal resources, both in terms of bandwidth but also in terms of power consumption of transceivers etc.

Personally I believe REST is never going to be a bad choice for implementing an API but under circumstances it may not be optimal. Shown here are a couple scenarios where this may be the case.

Recognising when these situations arise will enable you to consider possible alternatives. Using them when they aren't necessary is likely to make your API harder to understand and consume, but using them in the scenarios they are designed to address will optimise your API implementation and open it up to a larger and more diverse set of consumers.

Twelve Factor App - Admin Processes

The concept of the Twelve Factor app was developed by engineers at Heroku to describe the core principles and qualities that they believe are key to the adoption of the Software as a Service (SaaS) methodology.

First published in 2011 the Heroku platform is unashamedly opinionated in the enforcement of these principles, the relevance of them to effective software development has only intensified as the adoption of cloud computing has increased the number of us deploying and managing server side software.

The twelfth principle relates to the management of administrative tasks:

"Any needed admin tasks should be kept in source control and packaged with the application."

Administrative Tasks

It is quite common for engineers managing an application to need to perform one off admin processes within the environment the application is deployed into. The most common of these will be data migrations to accommodate schema changes or other updates to how data is stored.

Other examples might be needing to extract data from the environment for debugging or investigating issues or needing to inspect aspects of the application as it runs.

These administrative tasks are a natural part of managing an evolving application as requirements and needs change over time.

Process Formation

Within a twelve factor app the process formation is the mechanism that allows an application to effectively scale as demand grows. Each aspect of the application runs in its own process that can therefore be independently scaled to meet the changing scale and shape of the demand being placed on the application.

Administrative tasks should be treated no differently, they should be executed within the same process formation and the code or scripts associated with them should be part of the applications repository.

The changes being made by these administrative tasks need to be recorded alongside the applications source code. Not only so that these changes are recorded but also to allow every other environment, including a developers local development environment, to be kept in sync with the production environment.

Changes such as database migrations are often iterative in nature and so the history of the changes that have been applied are vital to understanding the structure of the data stores the application is running against.

 REPL Shells

A twelve factor app strongly favours technologies that provide a REPL shell environment. This allows administrative tasks to be consistently applied across all environments. Locally developers can simply invoke scripts via the shell within their development environment. Within a deployed environment a shell can be opened on the machine to achieve the same outcome, this can either be manual or automated via the applications deployment process.

Issues will often arise in production due to information about previous changes not being available to all team members. A previous deployment make have been tweaked or fixed by applying changes to a database or some other update of the environment. When the application is re-deployed to a fresh environment these un-recorded changes are not re-applied and the same issue raises it's head again.

An applications repository should contain the history of all changes made to an application along with all the resources necessary to get the application up and running. Secret knowledge of tweaks and changes that are needed to get the application need to be avoided at all costs.

Modern deployment techniques allow for the automation of all sorts of processes, there is ever reducing reasons for manual changes to an environment to be required during an applications deployment. That is not to say that it will never be necessary when an issues arises for manual changes to be made, but as soon as this happens the next question needs to be how do we automate this for the next deployment?    

Twelve Factor App - Logs

The concept of the Twelve Factor app was developed by engineers at Heroku to describe the core principles and qualities that they believe are key to the adoption of the Software as a Service (SaaS) methodology.

First published in 2011 the Heroku platform is unashamedly opinionated in the enforcement of these principles, the relevance of them to effective software development has only intensified as the adoption of cloud computing has increased the number of us deploying and managing server side software.

The eleventh principle relates to the handling of logs:

"Applications should produce logs as event streams and leave the execution environment to aggregate"

The Importance of Logging

One of the primary mechanisms for monitoring the behaviour and operation of a deployed application is via logging. Exactly what is being logged will vary by application but log entries will relate to events and operations taking place within the application, this time ordered list presents a history of what has taken place both good and bad.

Often logging is an after thought with not as much attention being payed to it as probably should be given its importance in maintaining application health and performance. Whenever a developer needs to investigate any issue in the application, or to validate correct operation, they are likely to use the application logs as their primary source of information.

Aggregated Streams

For logs to be useful they need to be aggregated and stored. An app following twelve factor principles does not play a part in this aggregation and storage process, it simply ensures the flow of logging data through the standard output mechanism of the tech stack being employed (such as stdout).

This simplifies the approach to logging in the code base and allows for different logging strategies to be employed in different environments.

In a development environment a developer may simply review the logs in the terminal, while in production or staging environments the logs will be collected and managed by the infrastructure.

Log Management

This approach also opens up the possibility of going beyond simple file logging to use log management services that can increase the value that can be derived from the underlying data.

Tools such as Splunk, or any other big data solution, make it possible to collect large amounts of data that would be impractical if every aspect of logging had to be implemented within the application code base. Frameworks such as Serilog give the log data structure enabling it to be queried and mined for information that might be hard to glean by simply reading the entries as text.

Effective logging along with the ability to review the data it provides are important skills for any developer to learn. When an issue has struck and stakeholders are looking for answers than this skill we help you steer a course back towards a working system. 

The simpler the approach to logging is the more the data it produces can be relied upon. Many tools exist to help in this regard and ensuring your applications only role is to feed data in to the system will help to make this as simple as possible.      

Twelve Factor App - Dev/Prod Parity

The concept of the Twelve Factor app was developed by engineers at Heroku to describe the core principles and qualities that they believe are key to the adoption of the Software as a Service (SaaS) methodology.

First published in 2011 the Heroku platform is unashamedly opinionated in the enforcement of these principles, the relevance of them to effective software development has only intensified as the adoption of cloud computing has increased the number of us deploying and managing server side software.

The tenth principle relates to the parity that should exist between all the environments where the application will run:

"All environments should be as similar as possible"

Environments

The exact number of environments that an application will run in will vary from team to team. Incidentally, one of the benefits of following the twelve factor principles is that standing up new environments should be a straight forward and repeatable process.

Despite variation in the exact number broadly speaking environments will fall into three categories. Development, Staging and Production.

In Development code is hot off the press and will more than likely be unstable. It may be running on the developers machine or in some shared environment where everyones latest and greatest is being deployed immediately after it's merged. 

In Staging we have refined our code to a point where we think it is ready for showtime. Before we make this final step we need to verify that our new code works in conjunction with everyone else's both new and existing.

Finally when everything is tested we make the final step of putting our new bits into production for real users to make use of.

Gaps

Often there are undesirable gaps between these environments that manifest themselves in three areas.

Time gaps exist when code is delayed from reaching production, it maybe days, weeks or months from a developers keystrokes to the code being deployed into production. This build up of change can lead to big bang deployments where large numbers of changes are dropped into production alongside each other. Clearly this increases the possibility that unforeseen problems will manifest themselves, the volume of change is also likely to hinder any investigation into what has gone wrong in order to find resolution.

Personal gaps exist when different individuals are responsible for the administration or management of the environments. Clearly not every developer is going to be granted access to production but when administration is undertaken by those not close to the code important context is lost. Conversely, if developers have no appreciation for the nature of the environment their code will be deployed into mistakes may be made that could have been avoided.

Tools gaps exist whenever different versions or types of tooling are used in each environment. The likelihood of this will vary greatly depending upon your technology stack, but it can often be tempting to use a more lightweight version of a particular backing service in Development compared to Production. Anytime there is a difference between Production and other environments than a small doubt will be created that testing is proving the code will work once deployed. 

Closing the Gaps

A twelve factor app looks to eliminate or at least reduce the scope of these gaps. 

Reducing the time gap by ensuring the lag between a developer writing code and it ending up in Production where it can provide benefit is as small as possible. Often implementing this speed of deployment is not a technical challenge, instead it is a battle against fear. The only way to overcome this fear is to evolve an automated testing strategy such that your build and deployment process is self evaluating and self proving.

Personal gaps are overcome by ensuring developers are closely aligned to the deployment of code and its management once in the environment. This gives developers important context and insight into what becomes of their code once it's written and merged. This principle should also be extended to include responsibility for issues and problems that may arise once the code is in use, this accountability isn't about assigning blame but to aid the development of a sense of when the seeds of an issue may be laid in the code base.

Tooling gaps can be harder too close since there may be cost and practicability arguments to be made as to why the same technology isn't being used everywhere. However a twelve factor mindset instills a wish for these gaps between production and other environments to be made as small as possible. It's important to realise that this isn't about scale, clearly having a Development environment scaled to deal with production loads is inefficient and costly. Instead this is about behaviour and functionality, these aspects of the backing services your application uses should be predictable and uniform in all environments.

Modern techniques such as so called Infrastructure as Code and containerisation mean it is now much simpler to try and ensure alignment between environments can be maintained.  Bugs grow in the gaps between environments. Discounting issues caused by insufficient testing, all bugs that have ever been shipped to production, which if we're being honest is a large number, have passed through test environments without being noticed. Ensuring close alignment between environments gives less space for these bugs to develop and grow without being spotted.        

Twelve Factor App - Disposability

The concept of the Twelve Factor app was developed by engineers at Heroku to describe the core principles and qualities that they believe are key to the adoption of the Software as a Service (SaaS) methodology.

First published in 2011 the Heroku platform is unashamedly opinionated in the enforcement of these principles, the relevance of them to effective software development has only intensified as the adoption of cloud computing has increased the number of us deploying and managing server side software.

The ninth principle relates to the disposability of processes and the relation to start-up and shutdown:

"Fast startup and shutdown are advocated for a more robust and resilient system."

Disposable Processes

An applications source code and the binary it produces represent the processor instructions that produce the intended functionality. A process is an instance, provided by the devices operating system, of these instructions running against the devices processor.

At a high level the process is made up of, the applications instructions, allocated memory and threads of execution, handles to file system resources and security attributes such as the user who owns the process etc.

Within a twelve factor app processes are first class citizens and a fundamental part of the architectural approach. They should also strive to be completely disposable. This means they should be able to be started, stopped and killed whenever necessary and complete all of these stages in a timely and efficient manner.

Startup Time

A process should not be slow to start, that is to say the time between a launch command being issued and the process being available to service requests should be as small as possible.

The reason this is desirable is because a twelve factor app scales via the application of a process formation, creating more processes to deal with demand as it comes. If these processes are slow to start-up then it limits the ability of an application to scale effectively.

Fast start up time also improves the robustness of the application. Sometimes processes may need to be migrated to new hardware or execution environment, either as part of a release or because of an underlying failure. Again a fast start-up processes allows this to happen quickly with minimum impact on the ability of the application to deal with demand.

Graceful Shutdown

A process should also be able to be gracefully shutdown or terminated. This means the ability for it finish dealing with a request quickly, to offload the necessary work to a queuing system in an effective manner.

This is mainly achieved by the process in general trying to avoid long running tasks or tasks that are non-transactional in nature and therefore can't be easily stopped or transferred to another process.

Alongside this graceful shutdown behaviour the proper implementation of a queuing system is essentially to ensure the application isn't vulnerable to sudden failure causing irreparable data loss or damage. The work the application does must be able to be easily represented in distinct items that can be queued and re-queued in an idempotent manner ensuring the application can be relied upon to operate in a clean consistent way.

When looking at the application in isolation the code executed as part of start-up and shutdown can sometimes be overlooked. Normally we are more concerned with what the code is doing while it is running.

However in a cloud based deployment environment applications don't just start and then run for an indefinitely long period of time. They are regularly required to start and stop as the application is scaled or re-deployed. Putting some thought into these areas of the code will lead to benefits in your ability to provide a robust and scaleable application that can flex to meet demand and cope with failure.