High Availability with Redis Cluster

  • High-Availability with Redis Cluster.
  • Automatic Failover with Redis Cluster.
  • Split-Brain Problem with Redis Cluster and its solution.
  • Say, we have one replica for every primary shard. If all our data is divided between three Redis Servers, we would need a six-membered cluster, with three primary shards and three replicas.
  • All six shards are connected to each other over TCP and constantly ping each other and exchange messages. These messages allow the cluster to determine which shards are alive.
  • When enough shards report that a given primary shard is not responding to them, they can agree to trigger a fail-over and promote the shard’s replica to become the new primary. The number of shards that needs to agree that a fellow shard is offline before fail-over is triggered, is configurable at the time of cluster-creation.
  • Now, both cluster-groups will think that they are offline and both shall trigger a fail-over of any primary shards, resulting in left side with all primary shards, as well as right side also would have all primary shards.
  • To prevent something called a split brain situation in a Redis cluster, always keep an odd number of shards in your cluster.
  • Now, when we get a Network-Split, left and right group shall do a count and see if they are in a bigger (majority) or smaller group (minority) ?
  • If a particular group is in Minority, it shall NOT try to trigger a fail-over and shall NOT accept any client write requests.
  • Here, Left side group (set of nodes), is in Minority and therefore it shall NOT try to trigger a fail-over and shall STOP accepting any client write requests.
  • Right side group (set of nodes), is in Majority and therefore it has authority and capability to trigger a fail-over of any primary shards.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
aditya goel

aditya goel

Software Engineer for Big Data distributed systems