High Availability with REDIS Replication and Sentinel

  • Overview of High-Availability.
  • Setting up Redis-replication.
  • Setting up security with Redis-Server, replica & Client.
  • Overview of Redis-Sentinel for Automatic-failover.
  • Setting up Redis-Sentinel for Automatic-failover.
  • Fault-Tolerant.
  • Highly Dependable.
  • Operating continuously without intervention
  • Doesn’t have any single point of failure.
  • Replication
  • Automatic failover.

Part #1 : Replication

  • Replication is a continuous copying of data from a primary database to a backup or a replica database.
  • The two databases are usually located on different physical servers, so that we can have a functional copy of our data in case, we lose a server where our primary database sits.
  • Replication in Redis follows a simple primary replica model where the replication happens in one direction, from the primary to one or multiple replicas.
  • Data is only written to the primary instance, and replicas are kept in sync so that they are exact copies of the primaries.
  • To transfer all of its data as efficiently as possible, the primary instance will produce a compacted version of the data as a snapshot in an RDB file and send it to the replica.
  • The replica will then read the snapshot file and load all of its data into memory, which will bring it to the same state the primary instance had at the moment of creating the RDB file.
  • When the loading stage is done, the primary instance will send the backlog of any write commands run since the snapshot was made.
  • Finally, the primary instance will send the replica, a live stream of all subsequent commands.
  • To transfer all of its data as efficiently as possible, the primary instance will produce a compacted version of the data in a .rdb file and send it to the replica.
  • The replica will then read the snapshot file and load all of its data into memory, which will bring it to the same state the primary instance had at the moment of creating the .rdb file.
  • When the loading stage is done, the primary instance will send the backlog of any write commands run since the snapshot was made.
  • The client will block (i.e. primary-server shall not return a response to the client) until all the previous write commands issued on that connection have been written to at least two replicas.
  • The second argument, 0, will instruct the server to block indefinitely. But we can set it to a number in milliseconds, so that it times out after a while and returns the number of replicas that successfully acknowledged the commands.
  • Note that, our original redis-server runs @ port 6379.
  • And, our replica-redis-server would now be running @ port 6380.
  • As you can see here, it says MASTER <-> SLAVE sync started, so we are, this one is the slave, so the process running at port number 6380 is the slave.
  • Eventually, we shall see that, we have both the master & slave processes running successfully and synchronised.

Part #2 : Automatic RollOver

  • The group of Sentinels monitors a primary Redis instance and its replicas.
  • If the sentinels detect that the primary instance has failed, the sentinel processes will look for the replica that has the latest data and will promote that replica to be the new primary.
  • This way, the clients talking to the REDIS-SYSTEM will be able to reconnect to the new primary and continue functioning as usual, with minimal disruption to the users.
  • We need to have enough Sentinels agree that the server is unreachable from their point of view.
  • Having a number of Sentinels agreeing that they need to take an action is called reaching a quorum. If the Sentinels can’t reach quorum, they cannot decide that the primary has failed. The exact number of Sentinels needed for quorum is configurable.
  • A leader can only be chosen if the majority of the Sentinels agree on it.
  • In the final step, the leader will reconfigure the chosen replica to become a primary by sending the command REPLICAOF NO ONE and it will reconfigure the other replicas to follow the newly promoted primary.
  • One Primary Redis Instance, being powered @ port : 6379.
  • Another Primary Redis Instance, being powered @ port : 6380.
$ touch sentinel1.conf
port 5000
sentinel monitor myprimary 127.0.0.1 6379 2
sentinel down-after-milliseconds myprimary 5000
sentinel failover-timeout myprimary 60000
sentinel auth-pass myprimary a_strong_password
  • port → The port on which the Sentinel would run.
  • sentinel monitor → It monitor the primary redis instance on a specific IP address and port. By having the address of the Primary Redis Instance, the Sentinels will be able to discover all the replicas on their own. The last argument on this line is the number of Sentinels needed for quorum. In our example — the number is 2.
  • sentinel down-after-milliseconds → How many milliseconds should an instance be unreachable so that it’s considered down.
  • sentinel failover-timeout → If a Sentinel voted another Sentinel for the failover of a given master, it will wait this many milliseconds to try to failover the same master again.
  • sentinel auth-pass → In order for Sentinels to connect to Redis server instances when they are configured with requirepass, the Sentinel configuration must include the sentinel auth-pass directive.
# Tab 1
$ redis-server ./sentinel1.conf --sentinel
# Tab 2
$ redis-server ./sentinel2.conf --sentinel
# Tab3
$ redis-server ./sentinel3.conf --sentinel
# Provides information about the Primary
SENTINEL master myprimary
# Gives you information about the replicas connected to the Primary
SENTINEL replicas myprimary
# Provides information on the other Sentinels
SENTINEL sentinels myprimary
# Provides the IP address of the current Primary
SENTINEL get-master-addr-by-name myprimary
redis> SENTINEL get-master-addr-by-name myprimary
1) "127.0.0.1"
2) "6380"

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
aditya goel

aditya goel

Software Engineer for Big Data distributed systems