Skip to main content

Cache strategies

Darryl Revok

An ideal cache entry is created just before its value is first needed and destroyed just after it is no longer needed. typical cache strategies do not (try to) approximate this ideal, but half a loaf is better than none?

caching doesn't necessarily help. for example, using another host to cache database query results requires more network usage. determining if the additional complexity and cost are worthwhile depends on the database server's behavior (it might be caching the results already), network connection speeds, etc.

in a simpler architecture, a server and its cache are hosted by the same machine. if there are redundant servers there will be duplicate cache entries, so analyzing cost/performance still requires effort. there is no free lunch.

Server requirements

caching should improve performance without impacting functionality. a server that caches values should not:

  • return a misleading response when a value is stale
  • fail to respond on time when a cache value is missing

when a value is missing, a server may return 200 as long as the response is clearly incomplete. we may use null to indicate that a value is missing.

though unusual in this context, this interface convention is quite common in general. when a value is missing because of a third-party service outage, immediately returning a partial response is optimal.

Server designs

servers typically use request methods to (re)initialize values. this design ensures that the server doesn't do unnecessary work, but it can:

  • induce request methods to race to (re)initialize the same cache value
  • introduce complexity/delay when responses depend on multiple data sources

cache entries often have similar lifecycles, so request processing can induce a request method to (re)initialize multiple expired cache entries. the server's response is delayed if (re)initializing any of these entries requires significant time, even when the request method is async.

(my server needs more than 10 seconds to reinitialize one of its cache entries.)

an alternative design decouples request processing from cache updating. in this design, cache entries are updated independently and request methods construct responses from cached values that may be null.

Peeker pattern

the observer pattern uses callback functions to react to state changes. this observation technique is intrusive, so i think the pattern was named badly. monitor pattern is also in use already, so let's call a decoupled observer a peeker.

the peeker pattern integrates neatly with the facade pattern, i.e. request methods can use a facade object to read cache values. the facade object also (re)initializes its cache entries when appropriate.

(i found the iconic Darryl Revok image when scanner was a candidate, and had to use it somehow. split brain was my first choice ... naming things is hard :-)

Server threads

a peeker is very fast, so server request methods are synchronous. on the other hand, asynchronous processing greatly facilitates cache maintenance.

a cache entry becomes invalid when we discover its new value via notification (observer pattern), or it expires (TTL). thus, using an async Python method to maintain a cache would require significant effort.

on the other hand, many caches can be maintained with per-entry worker threads. this design is easy to implement and it performs well, because Python's global interpreter lock limits performance only when tasks are CPU-bound.

(for TTL-based cache entry reinitialization, i override the Thread.run() method in a threading.Timer subclass.)

TL;DR: Guido was right.

Labels:

Comments

Post a Comment

Popular posts from this blog

Improper english

Before retirement ended my last spell of unemployment, i wondered if the timing of that dismissal was ideal. one month earlier or later might have been better? improving a server log was my last assignment. like many other companies, their senior management believed in their culture, technology, and tools. like other well-funded companies, they used Splunk and wanted to use JSON format. nobody reviewed the pull request that would have established a baseline for my work. their Splunk dashboard code was not versioned. Overcommunicating JSON can be ideal, and creating a data structure to discover if a log entry describes an error is easy and reasonably fast, but computers find strings very quickly. a faster algorithm uses less electricity; computer activity is human activity. ...
Post a Comment
Read more

Deliberate logging

While diagnosing an error, it's not unusual to discover that the application log does not contain relevant information while also being overly verbose. DEBUG-level entries can be cryptic, and maintainers often insist that their applications emit them in production environments. the universal acceptance of low quality logs is a management failure. Holistic development a software development protocol should synergize best practices, but we should run away whenever someone claims that success depends on a protocol's complete implementation. incremental development of software is common, as is reusing large frameworks. the development process can mature similarly. Holistic Development is a protocol i should describe someday, but piecemeal presentation makes sense because partial implementation is help...
Post a Comment
Read more

Sphinx supplement

Once upon a time, i was in a team led by a developer who was devoted to introducing modern standards. he didn't have much experience, so i think this devotion was essentially a cloaking device. though we didn't generate documentation, most pull requests elicited a request for a docstring change. my first post about Holistic Development focused on logging . this one is about generated documentation. though software developers habitually read code, generated documentation can become their primary source of information because it's easier to read. it also assures product owners that features are addressed, and that the design is orderly. Documentation review a docstring is just a comment until documentation is generated. at that point, evaluation of docstring quality should be based on the generate...
Post a Comment
Read more