Startup Strategies

Although the concept of a startup is not unique to the technology sector it does hold a special place in its hierarchy of organisations. Many IT professionals will have worked for or with a startup at some point during their career.

The journey for a startup to become a successful organisation, or unfortunately the potentially more likely outcome of failure, can take different paths depending on the startup itself and the market it is trying to operate in.

If I was able to offer foolproof strategies to deliver success for startups then I probably wouldn't be writing this blog, what is presented below are simply thoughts on the patterns that tend to manifest themselves as startups try to grow.

Get Big Quick

Commonly startups will attempt to execute a land grab, trying to gain substantial market share in a short time period. The need for this approach is generally indicative of a startup operating in a new market with a low barrier to entry.

In this situation the startup needs to gain market share quickly before competitors enter the market. Generally this will require large amounts of capital as the startup is attempting to grow faster than profitability levels would normally support.

Returns are based on the future potential of the market as the products or services being offered become commonly consumed. At this stage the startup can convert its dominant market position into profitability. Selling this future vision of where the market will eventually end up is often the key to securing funding.

The risk of this strategy comes from the large initial capital outlay being ultimately for nothing if the market doesn't develop as predicated or competitors become more effective at gaining initial market share.

Slow Burn

The opposite approach of slow and steady growth is usually a result of a startup operating within a well established market that already has many competitors. In this situation although clearly the startup will still require funding, simply trying to gain growth via large capital investment is unlikely to succeed since existing competitors already have established market share.

The fact that the market already has many established competitors means the success of the startup is dependent on exploiting a competitive advantage. This might be based in some intellectual property that the startup has developed or by attempting to implement a disruptive strategy to change the direction of the market.

If this competitive advantage can be exploited then success will come from the steady accumulation of market share or by selling access to the startups intellectual property to competitors. The risk obviously comes from the inability to find the competitive advantage or for it to be based on technology that is easily copied or replicated by competitors.

Whereas startups operating a Get Big Fast strategy that fails may tend to go out in a blaze of glory, startups operating a Slow Burn strategy will often be subject to a slow and painful death.

Make Noise

Sometimes a startup will be based around the development of a significant new technology that has potential with a given market place. However to be able to exploit this potential advantage the startup would require large amount of capital or existing influence within the market place.

In these situations it is often unrealistic to formulate a strategy for the startup to grow to a point where this is a realistic possibility. Therefore the exit strategy of these startups is not for the startup itself to exploit the new technology, it is instead to make a lot of noise in the market place in the hope that a larger more well established organisation decides to acquire the technology.

Organisations can decide to take this action either because the technology is truly transformational or because it would require a large effort to reproduce it that would risk others gaining the advantage first.

As previously stated this is not a guide to startup success it is simply a description of the common exit strategies that can be seen when observing startups. The choice of strategy is usually influenced by the size and maturity of the market the startup is operating in along with the nature of the startups perceived technological advantage. Choosing with strategy to employ and successfully executing it is essentially the key to success and is a non-trivial problem.

Messaging Pitfalls

A natural consequence of the adoption of a micro-services architecture was the rise in the use of messaging infrastructure, sometimes also referred to as an event bus. The principle reason for this is to avoid the temporal coupling that can occur when micro-services are chained together. This coupling can reduce the availability of functionality due to direct and potentially brittle coupling between each service.

A messaging approach allows for services to be de-coupled and more independent, this can lead to an increase in the functionality that can be attached to certain events happening within your system without this causing more and more complication in your codebase.

Unfortunately as with any architectural pattern it isn't all good news, no matter what approach you take to solving a problem its possible for anti-patterns to develop. This article is by no means an argument against messaging patterns, it is simply a guide to the potential anti-patterns that can weaken, or even eradicate, the positive impact this pattern can have on your software.

Dependencies and Leaky Events

Events should not have dependencies between each other, it should be possible to process any individual event in its entirety without having to wait for any other event to be dispatched. This will often require careful consideration to strike a balance between a large event containing a lot of data and one that is too small to be of any practical use.

Events should also encapsulate a clearly defined business process and be easily understandable simply from its name. This also means they should not leak the implementation detail of the system that produced the event.

An example of this kind of leaky event can be seen when they represent CRUD operations being performed on a database or some other storage mechanism. For example an Order Updated event lacks clarity and is simply exposing an operation performed on an underlying data structure. Instead events like Order Placed, Order Dispatched or Order Cancelled have a much clearer business context and are not tied to a data storage mechanism.

Implementing Sequences

Most messaging system are asynchronous in nature and therefore guaranteeing the order of delivery and processing can be an expensive and complex process.

Many processes often do represent a series of events so it isn't always easy to avoid these sequencing problems but in these situations the consequence of event processing happening out of order needs to be considered and every effort taken to avoid these situations.

Returning to our previous example tying payment and dispatch to an order updated event opens up the possibility of a problem if an order is dispatched before payment has been successfully taken. However triggering the shipping of an order from a Payment Received event avoids this situation.

As with our previous point accurately modelling our events on the business domain, and not its implementation, leads to an easier to understand and more robust system.

Commands in Disguise

In a micro-services architecture services will often be required to interact with each other. Traditionally this will be via a REST interface but once a messaging infrastructure is introduced it can be tempting to implement similar mechanisms using events.

When a service publishes an event it should be to impart the information of a business process having taken place to the rest of the system. The service should have no knowledge or requirements around who will consume the event and what action they will take.

An indicator that this approach is being broken is when you can identify services that publish an event and subsequently wait to receive an event in return. When this happens the services involved have lost independency and have become coupled. This isn't to say that the dependency is wrong or should be avoided but it should be made more explicit and clear via the use of REST.

This also prevents another source of leaky events where a service is forced to publish many events relating to error scenarios because other services are waiting on the outcome.

Messaging patterns can enrich micro-services in a way that makes them undeniably part of an effective micro-services implementation. But they aren't a suitable way to interconnect services in all situations, when used in these situations they can actively degrade the architecture of the system. When working in a micro-services architecture develop a play book for when to use REST and when to use events, take time to carefully construct events around business processes making every effort to avoid complex scenarios around there processing.   

Data Driven Success

Over the last couple of decades the majority of businesses have realised that the data they accumulate is one of their most valuable assets. This isn't necessarily because it has inherent value as a commodity, but because of the insights it can provide to drive efficiency, innovation and effectiveness.

However none of this is possible if you don't understand how data is accumulated, how it can be analysed and how it can be used. On the surface sometimes these aspects can appear trivial, and for the more simple applications of data maybe they are. But unlocking the higher orders of the potential of data requires a more thought through and scientific approach.

Very few, if anyone, can describe themselves as having mastered the use of data. As tools and techniques evolve new potential is unlocked. What is more important is an appreciation of the traps that lie in wait for those that don't understand the nature of data.

Big Data

The concept of big data is now so prevalent that it is almost taken as a given when discussing data related topics. I think the fact that the term makes reference to the amount of data has always been problematic. Big data is about more than simply the amount of data that has been acquired, it is about volume, variety and velocity.

Of course volume plays a part, never has our capability to store and process large amounts of data been higher. As the amount of data you collect rises the potential for insight grows.

Equally important is variety. This includes variety of sources, domains and context, if you only ever collect one particular dimension of data from one particular source the insights you can gain will be limited.

Finally velocity, insights that only become visible after long periods of data collection are likely to be harder to action effectively. Being able to accumulate data at a fast rate opens up the possibility to put its learnings to use before others can gain the same understanding.

Lake vs Warehouse

At a high level data is stored in either a structured or unstructured way. Generally after it is collected it is in an unstructured form, at this stage it is a raw material requiring refinement. In order to be used to drive insight it must transition into a structured form to support querying, exporting and socialisation.

We have come to refer to these two forms of storage as data lakes (unstructured) and data warehouses (structured). The issue with moving between structured and unstructured forms is it involves bias. This is driven from the current understanding of the nature of the data, the nature of the problem that is trying to be solved and second guessing what the answers might be.

To a large extent this bias is unavoidable. What must be minimised, and ideally eliminated, is the loss of raw data during the refinement process. As your understanding of the data, the collection process and the insights grows you will often realise your initial bias. If your raw material is lost then so is your opportunity to resolve it.

When data transitions too warehouses it should remain in the lake ready for the day when you discovery better and more insightful ways to process it.

Machine Learning

A practical example of what is possible with data is machine learning. By using this technique artificial intelligence can be created that can utilise otherwise hidden patterns in data.

Machine learning involves a model being trained to make decisions based on the analysis of a large and potentially varied data set. Traditional rules based analysis, with its inherent bias towards what it expects the data to show, doesn't offer the flexibility required to simply go where the data points. Once this training has taken place the system can be given new data as input, and based on the learnings from previous data, make judgements on the meaning behind this new data.

Machine learning can look miraculous as it appears to be based on values we don't traditionally associate with software such as reason, judgement and intuition. But actually it is simply a demonstration of the power of data and what can happen when its analysis is approached with an open mind.

Technology can sometimes be prone to fads or fashions but the appreciation of data and its uses isn't one of them. The amount and variety of data we can collect and process is only going to increase, as will our ability to derive value and insight from it. Technologies like machine learning are a demonstration of this trend, now that we've discovered the genie it isn't going back in the bottle.      

Micro Anti Patterns

The premise behind micro-services is a tried and tested one, divide and conquer via the strong application of single responsibility. Joining together elements that do one thing and do it will leads to a robust and flexible architecture.

Something however that can complicate the adoption of a micro-services architecture is the slightly abstract definition of when a service is a micro-service. One of the reasons for this is that it can depend on many factors, the context of your problem domain, the nature of your existing codebase and your required performance.

Rather than trying to address this thorny problem this article instead takes the opposite approach by describing some of the pitfalls that you can encounter when implementing micro-services.

Temporal Coupling

The glue most commonly used to bind micro-services is REST. In many circumstances this is the correct choice but not always. Equally common when this pattern is applied is the formation of chains of API calls. Service A calls Service B that then needs to call Service C and D in order to fulfil the original request.

This creates temporal coupling between all the services. Service A must wait for all services to return, a failure from any service results in a failed user operation from Service A. This leaves the system intolerant to even minor interruptions in the availability of downstream services.

The answer to this can be too decouple services via the introduction of async messaging or eventing. This potentially not only allows Service A to move on with the user journey whilst backend processes continue to run but also leaves it unaffected by temporary changes in the availability of any of the downstream services.

Implementing eventing can come with its own pitfalls and this shouldn't be used to replace all REST calls, but equally it should be a weapon in the armoury that is deployed when temporal coupling begins to present itself.

Unbounded Context

Data is the lifeblood of any system and therefore many micro-services will be concerned with the querying and manipulation of data. Given this fact it is important to consider how your domain will map to your micro-services and how you will ensure clear ownership over different aspects of it.

The goal of this is to ensure that services don't become coupled by the domain such that the complexity of the model is kept to a minimum and the implementation of new features doesn't involve widespread change.

Imagine in a retail based system many micro-services will work with the notion of a product but the view of a product model is likely to be very different between the micro-service that supports the searching of a product catalogue and one that is responsible for shipping it once an order has been placed.

By using bounded contexts, a core tenant of domain driven design, it is possible to support these different views whilst allowing the micro-services involved to remain decoupled, it simply requires the domain to be thought through and divided up along these lines.

Distributed Monolith

A common starting point for wishing to implement a micro-services pattern is a monolithic application. There is actually a school of thought that this is a good starting point because you have a good working knowledge of your problems domain that can aid the segregation of your existing codebase.

However when you do this care must be take such that you don't end up with the same monolithic application simply distributed across a network of micro-services. This can happen for many reasons, but can be detected by changes and enhancements continually requiring many micro-services to all be changed and re-deployed.

This may because of attempts at over sharing code, not having clear functional boundaries between micro-services or an ineffective deployment strategy. A strategy to deal with this situation is the Common Closure Principle (CCP), simply put things that tend to change together should be packaged together.

Looking at how a codebase reacts to and deals with change is usually a very good indicator as to its health. A well defined but flexible architecture will smooth the wheels of change by narrowing its impact. This increases the likelihood that automation can be relied upon to both test and deploy change and also makes it much easier to roll back the change should the unexpected arise.

As previously stated defining exactly what constitutes a micro-service is not as easy as one might think. However an easier prospect is to define what it isn't, this can aid iterations of trying to successfully segregate a system into micro-services by providing reference points to indicate when the architecture is heading in the wrong direction. Recognising these signs will allow you to analyse where the false step came and re-evaluate the decision to try and find a better way.

It is likely there are many right answers alongside the equally lengthy list of wrong ones.