Fritzy.io

Redis reliability for realtime apps

andyet postrealtimeopsarchitecturehowto

The Problem

When I was at FOSDEM last weekend, I talked to several people who couldn't believe that I would use Redis as a primary database in single page webapps. When mentioning that on Twitter, someone said, "Redis really only works if it's acceptable to lose data after a crash."

For starters, read http://redis.io/topics/persistence. What makes Redis different from other databases in terms of reliability is that a command can return "OK" before the data is written to disk (I'll get to this). Beyond that, it is easy to take snapshots, compress append-only log files, configure fsync behavior in Redis. There are tests for dealing with disk access suddenly cut off while writing, and steps are taken to prevent this from causing corruption. In addition, you have redis-check-aof for dealing with log file corruption.

Note that because you have fine tuned control over how fsync works, you don't have to rely on the operating system to make sure that operations are written to disk.

No Really, What Was the Problem Again?

Since commands fail in any database, client libraries wait for OKs, Errors, and Timeouts to deal with data reliability. Every database based application has to deal with the potential error. The difference is that we expect the pattern to be command-result based, when in fact, we can take a more asynchronous approach with Redis.

Asynchronous reliability

The real difference is that Redis will return an OK as long as it was written to RAM (see Antirez's clarification in the comments) while other databases tend to send OK only after the data is written to disk. We can still get on par (and beyond) with other database reliability easily enough by having a very simple check that you may be doing anyway without realizing it. When sending any command or atomic group of commands to Redis in the context of a single page app, I always send some sort of PUBLISH at the end. This publish bubbles back up to update the user clients as well as inform any other interested party (separate cluster processes for example) about what is going on in the database application. If the client application lets the user know that it didn't get an update corresponding with a user action within a certain amount of time, then we know the command didn't complete. Beyond this, we can write to a Redis master and LISTEN for publishes on a Redis slave! Now the client application can know that the data has been saved on more than one server; that sounds pretty reliable to me.

Using this information, the client application can intelligently deal with user action reliability all the way to the slave, and inform users with a simple error, resubmit their action without prompting, or request that the server do some sort of reliability check (in or out of context of the user action), etc.

tl;dr

  1. Single page app sends a command
  2. Application server runs an atomic action on Redis master.
  3. Redis master syncs to Redis slave
  4. PUBLISH at the end of said atomic action routes to application server from Redis slave.
  5. PUBLISH routes to single page app that sent the command, and thus the client application knows that said atomic action succeeded on two servers.
  6. If the client application hasn't heard a published confirmation, the client can deal with this as an error however it deems appropriate.

Further Thoughts

Data retention, reliability, scaling, and high availability are all related concepts, but not the same thing. This post specifically deals with data retention. There are existing strategies and efforts for the other related problems that aren't covered in this post.

If data retention is your primary need from a database, I recommend giving Riak a look. I believe in picking your database based on your primary needs. With Riak, commands can wait for X number of servers in the cluster to agree on a result, and while we can do something similar on the application level with Redis, Riak comes with this baked in.

David Search commented while reviewing this post, "Most people don't realize that a fsync doesn't actually guarantee data is written these days either (depending on the disk type/hardware raid setup/etc)." This further strengthens the concept of confirming that data exists on multiple servers, either asynchronously as this blog post outlines, or synchronously like with Riak.

About Nathan Fritz

Nathan Fritz aka @fritzy works at &yet as the Chief Architect. He is currently working on a book called "Redis Theory and Patterns."

If you’re building a single page app, keep in mind that &yet offers consulting, training and development services. Send Fritzy an email (nathan@andyet.net) and tell us what we can do to help.

Update: Comment From Antirez

Antirez chimed in the comments to correct this post.

"actually, it is much better than that ;)

Redis with AOF enabled returns OK only after the data was written on disk. Specifically (sometimes just transmitted to the OS via write() syscall, sometimes after also fsync() was called, depending on the configuration).

  1. It returns OK when aof fsync mode is set to 'no', after the wirte(2) syscall is performed. But in this mode no fsync() is called.

  2. It returns OK when aof fsync mode is set to 'everysec' (the default) after write(2) syscall is performed. With the exception of a really busy disk that has still a fsync operation pending after one seconds. In that case, it logs the incident on disk and forces the buffer to be flushed on disk blocking if at least another second passes and still the fsync is pending.

  3. It returns OK both after write(2) and fsync(2) if the fsync mode is 'always', but in that setup it is extremely slow: only worth it for really special applications.

Redis persistence is not less reliable compared to other databases, it is actually more reliable in most of the cases because Redis writes in an append-only mode, so there are no crashed tables, no strange corruptions possible."