Scaling for Complexity

When talking about scaling a computer system, the discussion usually revolves around scaling the system to handle more volume. Its common to plan for an increasing number of customers and requests made of a system.

The time to scale for complexity is now

However, almost no one talks about setting the system up to scale for complexity. But scaling to complexity should be a core part of the design of a computer system.

Handling complexity is a real problem. Scaling up to handle more volume can usually be handled by adding hardware or making slight modifications to the code. An inability to scale to complexity can involve deeper changes to the design, architecture, and implementation of a system, and cost a company thousands or even millions of dollars over time. Worse, it can make a company non-competitive and lead to the company losing relevance in the marketplace.

Think of it this way: the features that a computer system supports are never going to become less complex. Even if the team removes some features for being outdated, the only business reason to do so it to make room for new code that adds features. Otherwise, they are just introducing change for no profit — and that means risking introducing bugs or breaking features that the business does want.

So systems will always increase in complexity over time. That means more features, more capabilities, and more head-scratching logic.

Hard questions

Increasing complexity brings its share of hard questions. For example, how does the system maintain an increasing number of configuration options? How flexible is the persistent data model, to allow for additional fields appearing? Where is the business logic encoded as code? How visible are the internal workings of the system running in production? When the system fails, how easy is it to determine the state of all those features when it failed? And finally, how flexible is the code — does it allow for easily injecting new features without breaking existing features, and without wholesale cut-and-paste of existing code?

Many tools exist for handling these issues, and while it’s easy for a developer to get into hacker mode and just make the system work, it’s actually faster, easier and more effective to design the system to scale for complexity up front. If it’s easy to manage complexity when the system is small, it will be much easier when the system is large.

The usual tools for building systems address this, sometimes explicitly, sometimes implcitly. Here are some of them in the context of managing complexity.

Separation of concerns frameworks

Open-source frameworks like Spring help manage complexity by separating the concerns in your system. Web page templates are managed in one way, code for business logic in managed in a different way. This allows people working on the system to focus on one concern at a time. Managing complex web pages becomes easier because its isolated from managing business logic. Managing business logic is easier because its isolated from presentation. Database access is isolated in the same way.

In order to help the team understand the system and maintain consistency and logic in the different areas of concern, really leverage a framework to break up the concerns so they don’t become an inter-tangled jumble.

Clean data model

In that vein, the data that a system runs on is one of the most important concerns. Managing that data, and keeping it’s meaning and inter-relationships clear, is one of the most important concerns in a system. So from the very start, it’s critical to use good data modelling practices both in the persistent data store, and in the objects and data structures that are in the code.

Designing for flexibility is a key point of good object-oriented, relational, or structured data design. Use those techniques to make sure your data model isn’t boxing your system in, requiring a lot of rework to allow for new extensions as the data your managing grows.

Automated unit testing

Validating logic a year, or even months, after a team writes it is a big problem. As the system grows in complexity, it becomes harder and harder to validate that everything is still running correctly and meeting business requirements. People forget, people leave, and manual validation is time-consuming and expensive to a business.

Plus, the team wants the assurance that they can add new tools and consolidate code, while still maintaining the integrity of the system. Automated unit testing is the biggest tool to assist with that.

While there is power in an acceptance testing framework that validates broad inputs and outputs to a system, it lacks the pinpoint precision of validating individual components of a system in isolation. Unit testing requires “inductive” reasoning — that is, make sure that all the pieces work individually so they can work in concert.

There’s an additional benefit in that unit testing actually speeds up development. As each component is written and unit tested, a team can rapidly assemble a system from components that are validated continuously with the assurance that their building blocks are reliable.

Plan for complexity, build for assurance

The upshot is, if your system is successful, lots of people will have lots of motive to extend and grow it. Think about that growth up front as a scaling problem, just as you’d think about the scaling problem of handling more volume. In the early stages of development, the system grows in complexity very quickly, so think about it and make sure the system will continue to both be reliable and support that kind of rapid change and improvement as the demands people make on it increase over time.


Leave a comment

Filed under computer industry, computer scaling, opinionizing

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s