Infinite Horizontal Scalability with Redis Cluster

  • General Understanding of Scalability.
  • Types of Scalability.
  • Need for Scalability.
  • Why Horizontal-Scaling is preferred, in context of Redis ?
  • Launching a Redis cluster with 3 Master and 3 Replica Nodes.
  • Algorithmic Sharding with Redis and disadvantage.
  • Problem of Resharding and trivial solution of data re-distribution.
  • Addressing Resharding problem with HashSlots.
  • Adding additional master shard and replica to redis-cluster.
  • Reallocating hashslots to newly added shard.
  • Vertical scaling.
  • Horizontal scaling.
  • You can either add more RAM to your server, so it can fit the 300 gigabyte data set.
  • You can add two more servers and split the 300 gigabytes of data between the three of them.
  • Hitting your server’s RAM limit.
  • Reaching the performance limits in terms of throughput or operations per second is another.
  • Redis is mostly single-threaded, a single Redis Server instance cannot make use of the multiple cores of your server’s CPU for command processing.
  • Now, If we split the data between two Redis instances, our system can process requests in parallel, effectively doubling the throughput.
  • As a matter of fact, performance will scale close to linearly by adding more Redis instances to the system.
# redis.conf fileport 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
# redis.conf fileport 7001
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7001/nodes.conf
cluster-node-timeout 5000
appendonly yes
# redis.conf fileport 7002
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7002/nodes.conf
cluster-node-timeout 5000
appendonly yes
# redis.conf fileport 7003
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7003/nodes.conf
cluster-node-timeout 5000
appendonly yes
## redis.conf fileport 7004
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7004/nodes.conf
cluster-node-timeout 5000
appendonly yes
#redis.conf fileport 7005
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7005/nodes.conf
cluster-node-timeout 5000
appendonly yes
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1
# redis.conf file
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
  • port 7000 → The first line we specify the port on which the server should run.
  • cluster-enabled yes → At this line, we state that we want the server to run in cluster mode, with the cluster-enabled yes directive
  • cluster-config-file nodes.conf → This line defines the name of the file where the configuration for this node is stored, in case of a server restart.
  • cluster-node-timeout 5000 → It’s the number of milliseconds a node must be unreachable for it to be considered in failure state.
  • appendonly yes → This is for setting up AOF Redis Persistence Option. Recall from our previous blogs in this series that, Redis offers the AOF persistence option which works by logging every incoming write command to disk as it happens.
  • Because we’re using a deterministic hash function, this function will always assign a given key to the same shard always.
  • For example, Say that, we have got 2 Shards. Then, we first hash the key and then mod the hash-result by 2.
  • Slots 0 through 5460 might be assigned to Primary-Shard-0.
  • Slots 5461 to 10,922 might be assigned to Primary-Shard-1.
  • Slots 10,923 to 16,383 might be assigned to Primary-Shard-2.
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 cluster slots
  • This process allows mitigating data grown by adding more and more instances and dividing the data to smaller parts (shards or partitions).
  • Not only that, it also means that more and more computation power is available to handle your data, effectively supporting horizontal scaling. Let’s now see the procedure of doing the same :-
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000
  • The first parameter is the address of the new shard (to be added).
  • The second parameter is the address of any of the current shards in the cluster.
  • Master Shard running on Redis Instance powered @ port 7000.
  • Master Shard running on Redis Instance powered @ port 7001.
  • Master Shard running on Redis Instance powered @ port 7002.
  • Master Shard running on Redis Instance powered @ port 7006 → Latest Addition.
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 cluster nodes12d3c2c84682271220e7297589c2a3d674931ff5 127.0.0.1:7005@17005 slave 170746a71e7e69c88afd27fd6e95a2c1e98bae54 0 1655623286392 3 connectede04aee2f44b08f84e22ef3a0ed286e14fc44a50a 127.0.0.1:7001@17001 master - 0 1655623285379 2 connected 5461-1092201dffb9f96867fe75c4d3d247aa84973c4d77013 127.0.0.1:7004@17004 slave e04aee2f44b08f84e22ef3a0ed286e14fc44a50a 0 1655623286000 2 connected170746a71e7e69c88afd27fd6e95a2c1e98bae54 127.0.0.1:7002@17002 master - 0 1655623286696 3 connected 10923-16383bca6906f860abf8096d26391f10984b93bc35b6c 127.0.0.1:7000@17000 myself,master - 0 1655623285000 1 connected 0-5460fbc5f5e157d154cd41b606a9d88eb320c06ef9d4 127.0.0.1:7003@17003 slave bca6906f860abf8096d26391f10984b93bc35b6c 0 1655623286392 1 connected4b99501f01c953ab9273185c6f878255befb8839 127.0.0.1:7006@17006 master - 0 1655623286000 0 connected
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 --cluster add-node 127.0.0.1:7007 127.0.0.1:7000 --cluster-slave --cluster-master-id 4b99501f01c953ab9273185c6f878255befb883
  • We shall be using the same add-node command, and a few extra arguments indicating the shard is joining as a replica and what will be its primary shard. If we don’t specify a primary shard, Redis will assign one itself.
  • We had already found the IDs of our shards by running the cluster nodes command on any of the shards, as shown in above screenshot. The port of the primary shard we added in the last step was 7006, and we can see it’s ID on the last line : 4b99501f01c953ab9273185c6f878255befb8839.
  • The flag — cluster-slave indicates that the shard should join as a replica .
  • The flag — cluster-master-id 4b99501f01c953ab9273185c6f878255befb8839 specifies which primary shard our newly-added-shard should replicate to ?
  • Master Shard running on Redis Instance powered @ port 7000.
  • Master Shard running on Redis Instance powered @ port 7001.
  • Master Shard running on Redis Instance powered @ port 7002.
  • Master Shard running on Redis Instance powered @ port 7006 → Latest Addition.
  • Replica Shard running on Redis Instance powered @ port 7003.
  • Replica Shard running on Redis Instance powered @ port 7004.
  • Replica Shard running on Redis Instance powered @ port 7005.
  • Replica Shard running on Redis Instance powered @ port 7007 → Latest Addition.
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli cluster nodes
(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 --cluster reshard 127.0.0.1:7000
  • The first question you’ll get is about the number of slots you want to move. If we have 16384 slots in total, and four primary shards, let’s get a quarter of all shards, so the data is distributed equally. 16384 ÷ 4 is 4096, so let’s use that number.
  • The next question is about the receiving shard id; the ID of the primary shard we want to move the data to, which we learned how to get in the previous step, with the cluster nodes command.
  • Finally, we need to enter the IDs of the shards we want to copy data from. Alternatively, we can type “all” and the shard will move a number of hash slots from all available primary shards.
  • Slots 1365 through 5460 (i.e. total of 4095 slots) have now been assigned to Primary-Shard-0, which is running @ port 7000.
  • Slots 6827 through 10922 (i.e. total of 4095 slots) have now been assigned to Primary-Shard-1, which is running @ port 7001.
  • Slots 12288 through 16383 (i.e. total of 4095 slots) have now been assigned to Primary-Shard-2, which is running @ port 7002.
  • Slots 0 through 1364 i.e. total of 1365 slots.
  • Slots 5461 through 6826 i.e. total of 1365 slots.
  • Slots 10923 through 12287 i.e. total of 1364 slots.
  • Before adding a new additional shard in our above demonstration, the number of hashslots were 16384. Now, say a client tries to read the key foo, they will run the hash function and mod with the number of hashslots. The result comes out to be 12,182.
  • Note that, hash-slot 12,182 is sitting on Shard #3. Now, we added additional shard, so that our total number of shards is FOUR now.
  • Now, when a client tries to read the key foo, they will run the hash function and mod the number of hashslots as before. This time, the number of hashslots is still 16384. Understandably, the result would be same, pointing us to the same hashslot.
  • Note that, hash-slot 12,182 is now sitting on Shard #4, but we are just concerned about identifying the correct hashslot and we are done.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store