Cashing with caching and the lost art of efficient software

Caching often has a negative condemnation

I'm sure you have encountered it, a situation where an application is no longer performing the way we needed it to. We come into the office and client A or B has raised an P1 because the service is no longer processing requests or is taking too long. Or a performance marketeer wants to know when today the site will be fast again in order to gain that edge over the competition, while still having no clue what exactly is slowing down. Oftentimes we discover a slow program or system and the stakeholders demand a solution now, re-engineering the slow system takes too long so caching must be set up to remedy the problem. Now you are stuck with a caching that is often not thought through.

These are the sort of feelings and stories I hear from developers working under pressure and deadlines, it has oftentimes led to people seeing caching as band-aid. But I am convinced that caching is a valuable skill to master as a software engineer. The essence of caching is pretty simple: Do not do things twice. If you say it like that, in my opinion it should be default. When you design your software in such a way that you are not repeating the same operation twice you save on time, computational power and thus electricity. These two savings are actually very big and here is why.

Why the savings matter

When developing software professionally your ultimate goal is always to deliver as much value to the client / business / employer as possible. When your software is faster it can process more requests on the same hardware and thus serve more customers. Or serve the existing customers with less hardware and thus less cost. Long story short the cost per customer goes down.

But something else happens, when your software is more efficient it uses less energy, less hardware and less cooling which all take a toll on the environment. Not to say that your code can solve climate change but it is at least not wasteful and adds as much value against the least cost.

And last but definitely not least; writing efficient software makes you a better developer. When you always have this idea of efficient code in the back of your mind you force yourself to think critically about how you develop your software. And this critical thinking over time will make you better at writing code, making you more valuable in the job market and I believe, increases the fun of writing code.

A bit of inspiration

As said earlier there is more to caching than HTTP caching, so here a some examples for inspiration:

Rearranging code to allow values to be reused

Consider these code examples

foreach ($bigSetOfTransactions as $transaction) {
	$recipientId = $transaction->recipient->id;
	
	if (!isUserAllowedToReceive($recipientId)) {
		continue;
	}
	
	// perform transaction
}

Now let's consider that isUserAllowedToRecieve makes a request to and external service, which takes time especially when you add up all these queries in a big batch. When we cache the result like this during the time of this process we can speed thing up a lot.

$usersAllowedToReceive = [];

foreach ($bigSetOfTransactions as $transaction) {
	$recipientId = $transaction->recipient->id;
	
	if ($usersAllowedToReceive[$recipientId] === null) {
		$usersAllowedToReceive[$recipientId] = isUserAllowedToReceive($recipientId);
	}
	
	if (!$usersAllowedToReceive[$recipientId]) {
		continue;
	}
	
	// perform transaction
}

This code stores the answer to the question to be reused maybe just milliseconds later. Keep in mind that when this is a long running process you might still need to do some cache invalidation when a value has been stored for a longer period of time.

Computed database columns

Lets say you run into a slow sub-query of which the value does not change that often. Consider adding a computed column that refreshes after a few minutes or hours depending on your needs. Even when you refresh the column every minute but the query in question runs a 1000 times a minute that can be an immense load and response time reduction.

Storing generated content on a shared disk

Instead of always regenerating an invoice every time it is sent by mail or downloaded through a dashboard, save it to disk. This way heavy but less frequent tasks are not done twice. If you have data that can change, put something of a version number or hash in the filename to make sure you never serve stale content.

Shared memory (Redis, Memcache or just a db table table with key/value pairs)

When you have optimized your program to the extent where it does not do operations twice without a good reason, another way to further improve response times and lower the load on servers is to share data across servers. When your application lives in multiple processes one process can use a cached value which was just used by another process only moments ago. Using shared memory extends the principle of not doing things twice even further. 

A final though

Before ending this post I want to mention the aspect of volume. Even fast functions or systems might benefit from optimization. If you have a counter on your site showing how many customers you have served, that is fast, just a SELECT count(*) on a table and that is it. But when that table grows and the page which runs this query gets requested more often, even a 2 ms query can create substantial strain on your database when run 1000 times a second. 

I hope this article helps you think more critically about your software design and makes you deliver more value with less resources.