Infinite Horizontal Scalability with Redis Cluster

21 min readJun 18, 2022

Welcome to this blog. If you are coming here directly, it’s highly recommended to read through this story first. We shall be looking at following topics in this blog :-

General Understanding of Scalability.
Types of Scalability.
Need for Scalability.
Why Horizontal-Scaling is preferred, in context of Redis ?
Launching a Redis cluster with 3 Master and 3 Replica Nodes.
Algorithmic Sharding with Redis and disadvantage.
Problem of Resharding and trivial solution of data re-distribution.
Addressing Resharding problem with HashSlots.
Adding additional master shard and replica to redis-cluster.
Reallocating hashslots to newly added shard.

Question:- What’s the meaning of Scalability ?

Answer :- Scalability is the property of a system to handle a growing amount of work by adding resources to the system.

Question:- What are the scaling strategies available ?

Answer :- The two most common scaling strategies are :-

Vertical scaling.
Horizontal scaling.

Question:- What does Vertical-Scaling means ?

Answer:- Vertical scaling, or also called scaling up, means adding more resources like CPUs or memory to your server.

Question:- What does Horizontal-Scaling means ?

Answer:- Horizontal scaling, or scaling out, implies adding more servers to your pool of resources.

Note:- The difference between Vertical & Horizontal Scaling is just getting a bigger server & deploying a whole fleet of servers respectively.

Question:- Can you showcase a practical example for Vertical Vs Horizontal Scaling ?

Answer:- Suppose you have a server with 128 gigabytes of RAM, but you know that your database will need to store 300 gigabytes of data. In this case, you’ll have two choices :-

You can either add more RAM to your server, so it can fit the 300 gigabyte data set.
You can add two more servers and split the 300 gigabytes of data between the three of them.

Question:- What are the usual practical reasons, for which we need Scaling?

Answer:- There are two usual reasons for doing the Scaling :-

Hitting your server’s RAM limit.
Reaching the performance limits in terms of throughput or operations per second is another.

Question:- Why should we really go for Horizontal-Scaling (Sharding) and not for Vertical-Scaling ?

Answer:- Let’s see why we should prefer Horizontal-Scaling after a certain limit :-

Redis is mostly single-threaded, a single Redis Server instance cannot make use of the multiple cores of your server’s CPU for command processing.
Now, If we split the data between two Redis instances, our system can process requests in parallel, effectively doubling the throughput.
As a matter of fact, performance will scale close to linearly by adding more Redis instances to the system.

This pattern of splitting data between multiple servers for the purpose of scaling is called Sharding.

Question:- Can you explain, what is a Shard ?

Answer:- Whenever we split the overall data between multiple Redis-Servers for the purpose of scaling, these resulting servers or processes that hold these chunks of the data are called shards.

Question:- Demonstrate the process of launching up of a Simple Redis Cluster.

Answer:- We shall be setting up following Redis Cluster :-

Step #1.) Let’s first launch a fresh Redis Instance with following configuration :-

# redis.conf fileport 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

Step #2.) Next, we shall launch 2nd Redis Instance now @ port no. 7001 :-

# redis.conf fileport 7001
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7001/nodes.conf
cluster-node-timeout 5000
appendonly yes

Step #3.) Next, we shall launch 3rd Redis Instance now @ port no. 7002 :-

# redis.conf fileport 7002
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7002/nodes.conf
cluster-node-timeout 5000
appendonly yes

Step #4.) Next, we shall launch 4th Redis Instance now @ port no. 7003 :-

# redis.conf fileport 7003
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7003/nodes.conf
cluster-node-timeout 5000
appendonly yes

Step #5.) Next, we shall launch 5th Redis Instance now @ port no. 7004 :-

## redis.conf fileport 7004
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7004/nodes.conf
cluster-node-timeout 5000
appendonly yes

Step #6.) Next, we shall launch 6th Redis Instance now @ port no. 7005 :-

#redis.conf fileport 7005
cluster-enabled yes
cluster-config-file /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/7005/nodes.conf
cluster-node-timeout 5000
appendonly yes

Step #7.) Next, we shall launch a Redis-Cluster with the help of afore-launched 6 Redis-nodes :- Here we list the ports and IP addresses of all six servers and use the CREATE command to instruct Redis to join them in a cluster, creating one replica for each primary. Redis-cli will propose a configuration; accept it by typing yes. The cluster will be configured and joined, which means, instances will be bootstrapped into talking with each other.

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1

Note: We shall see the concept of Hash-Slots very soon.

Question:- Can you kindly explain the meaning of configuration, that we used to launch Redis-Cluster ?

Answer:- We shall be setting up following Redis Cluster :-

# redis.conf file
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

port 7000 → The first line we specify the port on which the server should run.
cluster-enabled yes → At this line, we state that we want the server to run in cluster mode, with the cluster-enabled yes directive
cluster-config-file nodes.conf → This line defines the name of the file where the configuration for this node is stored, in case of a server restart.
cluster-node-timeout 5000 → It’s the number of milliseconds a node must be unreachable for it to be considered in failure state.
appendonly yes → This is for setting up AOF Redis Persistence Option. Recall from our previous blogs in this series that, Redis offers the AOF persistence option which works by logging every incoming write command to disk as it happens.

Question:- Does Redis Cluster uses consistent hashing ?

Answer → Redis-Cluster doesn’t uses consistent-hashing.

But rather Redis-Cluster uses a different form of sharding, known as “Algorithmic Sharding” where every key is conceptually part of what we call a hash slot.
There are 16384 hash slots in Redis Cluster, and to compute the hash slot for a given key, we simply take the CRC16 of the key modulo 16384.

Every node in a Redis Cluster is responsible for a subset of the hash slots, so, for example, you may have a cluster with 3 nodes, where:

Node A contains hash slots from 0 to 5500.
Node B contains hash slots from 5501 to 11000.
Node C contains hash slots from 11001 to 16383.

Question:- Now that, we have actually performed the setup of Redis-Cluster with multiple Redis-Shards, How will the Redis know, which particular Shard to look for a given key ?

Answer:- We need to have a way to consistently map a key to a specific shard. There are multiple ways to do this. The one Redis uses is called “Algorithmic Sharding”.

Question:- How does “Algorithmic Sharding” works ?

Answer:- We basically define the shard for a given key i.e. map the key to the shard. We hash the key and then mod the result by the total number of shards.

Because we’re using a deterministic hash function, this function will always assign a given key to the same shard always.
For example, Say that, we have got 2 Shards. Then, we first hash the key and then mod the hash-result by 2.

Question:- Now, say our business is growing and we have got more data to handle and therefore we want to increase our shards count (i.e. ReSharding). So, how does our data-querying is affected now ?

Answer:- Let’s say we add one new shard so that our total number of shards is three. Now, when a client tries to read the key foo, they will run the hash function and mod the number of shards as before. This time, the number of shards is THREE and therefore we’re modding with three instead of two. Understandably, the result may be different, pointing us to the different shard. For example, data for key “foo” was originally housed in Shard-0, but now post this fresh re-sharding, we are being re-directed to Shard-1. Thus, we may not get the data at all now.

Question:- How the Resharding issue can be solved manually with the Algorithmic — Sharding Strategy ?

Answer:- This can be solved by rehashing all the keys in the keys base and moving them to the shard appropriate to the new shard count. This is not a trivial task, though. And it can require a lot of time and resources, during which the database will not be able to reach its full performance or might even become unavailable.

Question:- Does there not exists, some smarter approach for solving the problem of ReSharding with Algorithmic Sharding ?

Answer:- Redis uses a clever approach to solve this problem, using HashSlot. Using hashslots allows the cluster to scale through the addition of new shards without having to compute new hashes for each key.

Part #1.) A logical unit that sits between a key and a shard called a hashslot.

Part #2.) The total number of hashslots in a database is always 16,384, or 16K.

Refer to our latest snapshot of Redis-Cluster Creation, where it also shown the similar message :-

Part #3.) The hashslots are divided roughly even across the shards. For example :- Say we have got overall 3 primary shards, then :-

Slots 0 through 5460 might be assigned to Primary-Shard-0.
Slots 5461 to 10,922 might be assigned to Primary-Shard-1.
Slots 10,923 to 16,383 might be assigned to Primary-Shard-2.

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 cluster slots

Part #4.) In a Redis cluster, we actually mod by the number of hashslots, not by the number of shards. Each key is assigned to a particular hashslot. For example, In below diagram, though the 3rd Shard (i.e. Shard #2) was added recently, but since we are always hashing on the hashslot-count, therefore the answer of our mod, shall always remain same-hashslot. That is the smart solution provided by Redis-Cluster.

Question:- Demonstrate with an example, where say our we have got more data to handle and therefore we want to add another shard (i.e. we now wanted to perform ReSharding). So, how would HashSlots be moved to newly added Shard ?

Answer:- In Redis, data sharding (partitioning) is the technique to split all data across multiple Redis instances so that every instance will only contain a subset of the keys.

This process allows mitigating data grown by adding more and more instances and dividing the data to smaller parts (shards or partitions).
Not only that, it also means that more and more computation power is available to handle your data, effectively supporting horizontal scaling. Let’s now see the procedure of doing the same :-

Step #1.) Now, say we wish to add another primary-shard to our Redis-Cluster. With this additional Shard, we would scale-up both Read & Write throughputs. Recall from our previous original demonstration that, we had got overall 3 primary shards and 1 replica for each.

Now, Let’s add a new master node to our Redis-Cluster :- We launch another Redis Instance and this shall run on port 7006 :-

Step #2.) Similarly, for this newly added master primary shard, we shall also have to have the slave-node, so let’s go ahead and also launch another Redis Instance (which would eventually become the replica) and this shall run on port 7007 :-

Step #3.) Let’s add this new Redis Instance running at port 7006 to our Redis Cluster, so as to make this part of cluster. Note that, after this step, we shall have 4 primary shards/nodes in our cluster :-

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

Note here in the command that :-

The first parameter is the address of the new shard (to be added).
The second parameter is the address of any of the current shards in the cluster.

Step #4.) Now, let’s investigate our Redis Cluster. We have got 4 primary shards/nodes in our Redis-Cluster as follows :-

Master Shard running on Redis Instance powered @ port 7000.
Master Shard running on Redis Instance powered @ port 7001.
Master Shard running on Redis Instance powered @ port 7002.
Master Shard running on Redis Instance powered @ port 7006 → Latest Addition.

All Master Nodes have been highlighted with blue-color in below screenshot :-

Observe the below output that, every node in our Redis-Cluster have got some unique ID. For example, our newly added Redis Instance (powered @ port 7006) have got the ID as : 4b99501f01c953ab9273185c6f878255befb8839

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 cluster nodes12d3c2c84682271220e7297589c2a3d674931ff5 127.0.0.1:7005@17005 slave 170746a71e7e69c88afd27fd6e95a2c1e98bae54 0 1655623286392 3 connectede04aee2f44b08f84e22ef3a0ed286e14fc44a50a 127.0.0.1:7001@17001 master - 0 1655623285379 2 connected 5461-1092201dffb9f96867fe75c4d3d247aa84973c4d77013 127.0.0.1:7004@17004 slave e04aee2f44b08f84e22ef3a0ed286e14fc44a50a 0 1655623286000 2 connected170746a71e7e69c88afd27fd6e95a2c1e98bae54 127.0.0.1:7002@17002 master - 0 1655623286696 3 connected 10923-16383bca6906f860abf8096d26391f10984b93bc35b6c 127.0.0.1:7000@17000 myself,master - 0 1655623285000 1 connected 0-5460fbc5f5e157d154cd41b606a9d88eb320c06ef9d4 127.0.0.1:7003@17003 slave bca6906f860abf8096d26391f10984b93bc35b6c 0 1655623286392 1 connected4b99501f01c953ab9273185c6f878255befb8839 127.0.0.1:7006@17006 master - 0 1655623286000 0 connected

Step #5.) Next, now let’s add our another Redis instance (running @ port 7007) to our Redis Cluster. Note that, this redis-instance shall be joining the cluster as a Slave Node :-

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 --cluster add-node 127.0.0.1:7007 127.0.0.1:7000 --cluster-slave --cluster-master-id 4b99501f01c953ab9273185c6f878255befb883

We shall be using the same add-node command, and a few extra arguments indicating the shard is joining as a replica and what will be its primary shard. If we don’t specify a primary shard, Redis will assign one itself.
We had already found the IDs of our shards by running the cluster nodes command on any of the shards, as shown in above screenshot. The port of the primary shard we added in the last step was 7006, and we can see it’s ID on the last line : 4b99501f01c953ab9273185c6f878255befb8839.
The flag — cluster-slave indicates that the shard should join as a replica .
The flag — cluster-master-id 4b99501f01c953ab9273185c6f878255befb8839 specifies which primary shard our newly-added-shard should replicate to ?

Step #6.) Now, let’s investigate our Redis Cluster again. We have got 4 primary shards/nodes in our Redis-Cluster as follows. All Master Nodes have been highlighted with blue-color in below screenshot :-

Master Shard running on Redis Instance powered @ port 7000.
Master Shard running on Redis Instance powered @ port 7001.
Master Shard running on Redis Instance powered @ port 7002.
Master Shard running on Redis Instance powered @ port 7006 → Latest Addition.

We have also got 4 replica shards/nodes in our same Redis-Cluster :-

Replica Shard running on Redis Instance powered @ port 7003.
Replica Shard running on Redis Instance powered @ port 7004.
Replica Shard running on Redis Instance powered @ port 7005.
Replica Shard running on Redis Instance powered @ port 7007 → Latest Addition.

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli cluster nodes

Step #7.) Now our cluster has eight shards (four primary and four replica), but if we run the cluster slots command we’ll see that the newly added shard (i.e. Newly added primary shard running at port 7006) don’t host any hash slots yet and thus — No data at All. Finally, We would observe that, hash-slots are not at all assigned to the new primary-shard.

Step #8.) Let’s assign some hash slots to the newly added Shard (Recall that, newly added shard is running at port 7006 and it’s corresponding replica is running at port 7007) :-

(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % cd /Users/adityagoel/Downloads/redis-cluster/redis-7.0.0/(base) adityagoel@Adityas-MacBook-Pro redis-7.0.0 % ./src/redis-cli -p 7000 --cluster reshard 127.0.0.1:7000

The first question you’ll get is about the number of slots you want to move. If we have 16384 slots in total, and four primary shards, let’s get a quarter of all shards, so the data is distributed equally. 16384 ÷ 4 is 4096, so let’s use that number.
The next question is about the receiving shard id; the ID of the primary shard we want to move the data to, which we learned how to get in the previous step, with the cluster nodes command.
Finally, we need to enter the IDs of the shards we want to copy data from. Alternatively, we can type “all” and the shard will move a number of hash slots from all available primary shards.

Step #9.) Once the command finishes we can run the cluster slots command again and we’ll see that our new primary and replica shards have been assigned some hash slots:

Slots 1365 through 5460 (i.e. total of 4095 slots) have now been assigned to Primary-Shard-0, which is running @ port 7000.
Slots 6827 through 10922 (i.e. total of 4095 slots) have now been assigned to Primary-Shard-1, which is running @ port 7001.
Slots 12288 through 16383 (i.e. total of 4095 slots) have now been assigned to Primary-Shard-2, which is running @ port 7002.

Net total of 4094 hash-slots have been now successfully assigned to newly-added Primary-Shard-3 (this primary shard is running @ port 7006) :-

Slots 0 through 1364 i.e. total of 1365 slots.
Slots 5461 through 6826 i.e. total of 1365 slots.
Slots 10923 through 12287 i.e. total of 1364 slots.

Following is how the hash-slots allocation now looks like within our Redis Cluster :-

That’s how, hash-slot reallocation is usually done within Redis Cluster. Next, whenever we get any request to GET a particular key, since we are always hashing on the hashslot-count, therefore the answer of our mod, shall always remain same. Example :-

Before adding a new additional shard in our above demonstration, the number of hashslots were 16384. Now, say a client tries to read the key foo, they will run the hash function and mod with the number of hashslots. The result comes out to be 12,182.

Note that, hash-slot 12,182 is sitting on Shard #3. Now, we added additional shard, so that our total number of shards is FOUR now.
Now, when a client tries to read the key foo, they will run the hash function and mod the number of hashslots as before. This time, the number of hashslots is still 16384. Understandably, the result would be same, pointing us to the same hashslot.

Note that, hash-slot 12,182 is now sitting on Shard #4, but we are just concerned about identifying the correct hashslot and we are done.

Question:- Showcase some production grade Redis-Cluster and how the hashslots are being divided across these nodes ?

Answer → There are 9 Masters and 9 Slaves for one of our ReTargeting-Redis-Cluster and this contains hashslots being divided across these 9 masters. Here is how I am running the Redis Command through my local machine for one of the Clusters :-

The full output of this Redis-Cluster is being shown below :-

MASTER #1 

1) 1) (integer) 3641
    2) (integer) 5460
    3) 1) "10.41.7.170"
       2) (integer) 6379
       3) "c14e33e1d714ae956e946afac63dd034d3e2b985"
    4) 1) "10.41.7.165"
       2) (integer) 6379
       3) "e8aafccd05f36c2e5fe9d229f192967344dae325"


MASTER #2 

 2) 1) (integer) 0
    2) (integer) 227
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 3) 1) (integer) 296
    2) (integer) 393
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 4) 1) (integer) 550
    2) (integer) 679
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 5) 1) (integer) 956
    2) (integer) 1183
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 6) 1) (integer) 1777
    2) (integer) 2004
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 7) 1) (integer) 3413
    2) (integer) 3640
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 8) 1) (integer) 6144
    2) (integer) 6371
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
 9) 1) (integer) 8875
    2) (integer) 9102
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"
10) 1) (integer) 14336
    2) (integer) 14563
    3) 1) "10.41.117.60"
       2) (integer) 6379
       3) "9da5975e4c8d10dbc15ce834c44de14c06097a23"
    4) 1) "10.41.117.56"
       2) (integer) 6379
       3) "2105d2edb9cd38f8ea46bfdf6d85b4fe5f78226a"


MASTER #3 

11) 1) (integer) 2005
    2) (integer) 2184
    3) 1) "10.41.117.38"
       2) (integer) 6379
       3) "9b5718fd7d8b15c2b2c6bfc4feff4b68eb6e3f49"
    4) 1) "10.41.117.111"
       2) (integer) 6379
       3) "cf8057051806501e77f0a73d7f3028f7d674b492"
12) 1) (integer) 6827
    2) (integer) 7646
    3) 1) "10.41.117.38"
       2) (integer) 6379
       3) "9b5718fd7d8b15c2b2c6bfc4feff4b68eb6e3f49"
    4) 1) "10.41.117.111"
       2) (integer) 6379
       3) "cf8057051806501e77f0a73d7f3028f7d674b492"
13) 1) (integer) 12288
    2) (integer) 13107
    3) 1) "10.41.117.38"
       2) (integer) 6379
       3) "9b5718fd7d8b15c2b2c6bfc4feff4b68eb6e3f49"
    4) 1) "10.41.117.111"
       2) (integer) 6379
       3) "cf8057051806501e77f0a73d7f3028f7d674b492"



MASTER #4 


14) 1) (integer) 1184
    2) (integer) 1364
    3) 1) "10.41.116.249"
       2) (integer) 6379
       3) "33eb83304da64cc880d2284dfce841e611ab78b1"
    4) 1) "10.41.117.3"
       2) (integer) 6379
       3) "f98d4d16c228651e4ac80c23e4868c8739b71bef"
15) 1) (integer) 2185
    2) (integer) 2730
    3) 1) "10.41.116.249"
       2) (integer) 6379
       3) "33eb83304da64cc880d2284dfce841e611ab78b1"
    4) 1) "10.41.117.3"
       2) (integer) 6379
       3) "f98d4d16c228651e4ac80c23e4868c8739b71bef"
16) 1) (integer) 5461
    2) (integer) 5461
    3) 1) "10.41.116.249"
       2) (integer) 6379
       3) "33eb83304da64cc880d2284dfce841e611ab78b1"
    4) 1) "10.41.117.3"
       2) (integer) 6379
       3) "f98d4d16c228651e4ac80c23e4868c8739b71bef"
17) 1) (integer) 7647
    2) (integer) 8192
    3) 1) "10.41.116.249"
       2) (integer) 6379
       3) "33eb83304da64cc880d2284dfce841e611ab78b1"
    4) 1) "10.41.117.3"
       2) (integer) 6379
       3) "f98d4d16c228651e4ac80c23e4868c8739b71bef"
18) 1) (integer) 13108
    2) (integer) 13653
    3) 1) "10.41.116.249"
       2) (integer) 6379
       3) "33eb83304da64cc880d2284dfce841e611ab78b1"
    4) 1) "10.41.117.3"
       2) (integer) 6379
       3) "f98d4d16c228651e4ac80c23e4868c8739b71bef"




MASTER #5

19) 1) (integer) 9103
    2) (integer) 10922
    3) 1) "10.41.7.168"
       2) (integer) 6379
       3) "89774408ab52c0b8e921e2a26bcaf9733564a77a"
    4) 1) "10.41.7.169"
       2) (integer) 6379
       3) "009a2d76ebe199d57a156fd72ddb9a74ab549308"


MASTER #6


20) 1) (integer) 6372
    2) (integer) 6826
    3) 1) "10.41.117.109"
       2) (integer) 6379
       3) "152122c2769c8680fcd535cecefe009b8a26d1df"
    4) 1) "10.41.117.88"
       2) (integer) 6379
       3) "d79c8d017d432ee69a53bda178b392268f2dcbf9"
21) 1) (integer) 10923
    2) (integer) 12287
    3) 1) "10.41.117.109"
       2) (integer) 6379
       3) "152122c2769c8680fcd535cecefe009b8a26d1df"
    4) 1) "10.41.117.88"
       2) (integer) 6379
       3) "d79c8d017d432ee69a53bda178b392268f2dcbf9"



MASTER #7


22) 1) (integer) 680
    2) (integer) 819
    3) 1) "10.41.119.99"
       2) (integer) 6379
       3) "6fcd6cf698c67192f607fbe612ed6cb41671d2bc"
    4) 1) "10.41.118.63"
       2) (integer) 6379
       3) "5d0794f5422f8fed4c86bdc8a5e71a7a5edb2cd3"
23) 1) (integer) 1365
    2) (integer) 1484
    3) 1) "10.41.119.99"
       2) (integer) 6379
       3) "6fcd6cf698c67192f607fbe612ed6cb41671d2bc"
    4) 1) "10.41.118.63"
       2) (integer) 6379
       3) "5d0794f5422f8fed4c86bdc8a5e71a7a5edb2cd3"
24) 1) (integer) 2731
    2) (integer) 3120
    3) 1) "10.41.119.99"
       2) (integer) 6379
       3) "6fcd6cf698c67192f607fbe612ed6cb41671d2bc"
    4) 1) "10.41.118.63"
       2) (integer) 6379
       3) "5d0794f5422f8fed4c86bdc8a5e71a7a5edb2cd3"
25) 1) (integer) 5462
    2) (integer) 5851
    3) 1) "10.41.119.99"
       2) (integer) 6379
       3) "6fcd6cf698c67192f607fbe612ed6cb41671d2bc"
    4) 1) "10.41.118.63"
       2) (integer) 6379
       3) "5d0794f5422f8fed4c86bdc8a5e71a7a5edb2cd3"
26) 1) (integer) 8193
    2) (integer) 8582
    3) 1) "10.41.119.99"
       2) (integer) 6379
       3) "6fcd6cf698c67192f607fbe612ed6cb41671d2bc"
    4) 1) "10.41.118.63"
       2) (integer) 6379
       3) "5d0794f5422f8fed4c86bdc8a5e71a7a5edb2cd3"
27) 1) (integer) 13654
    2) (integer) 14043
    3) 1) "10.41.119.99"
       2) (integer) 6379
       3) "6fcd6cf698c67192f607fbe612ed6cb41671d2bc"
    4) 1) "10.41.118.63"
       2) (integer) 6379
       3) "5d0794f5422f8fed4c86bdc8a5e71a7a5edb2cd3"



MASTER #8


28) 1) (integer) 228
    2) (integer) 295
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
29) 1) (integer) 394
    2) (integer) 549
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
30) 1) (integer) 820
    2) (integer) 955
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
31) 1) (integer) 1485
    2) (integer) 1776
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
32) 1) (integer) 3121
    2) (integer) 3412
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
33) 1) (integer) 5852
    2) (integer) 6143
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
34) 1) (integer) 8583
    2) (integer) 8874
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"
35) 1) (integer) 14044
    2) (integer) 14335
    3) 1) "10.41.117.74"
       2) (integer) 6379
       3) "763e3393a27a17e5500b5679f63cde15968dae0b"
    4) 1) "10.41.116.254"
       2) (integer) 6379
       3) "be01528d2b514c7927ba942f4d5d526d0804b3c3"


MASTER #9


36) 1) (integer) 14564
    2) (integer) 16383
    3) 1) "10.41.7.166"
       2) (integer) 6379
       3) "1205c7e6e1e523666e14df8090645b26f3d26f80"
    4) 1) "10.41.7.167"
       2) (integer) 6379
       3) "6297c274e7512fdc0c0627216cc52e73c0b81ad9"

Question:- Showcase some production grade way of creating the VMs ?

Answer → We usually do it with the help of Terraform :-

include {
  path = find_in_parent_folders()
}

terraform {
  # Docs: https://github.com/tokopedia/tf-gcp-modules/tree/master/compute-instance-cff
  source = "git::git@github.com:tokopedia/tf-gcp-modules.git//compute-instance-cff"
}

inputs = {
  # compute-instance
  ci_num_instances = 18
  ci_hostname      = "topads-retargeting-redis-cluster"

  # Allow stopping of instance to update instance type
  ci_allow_stopping_for_update = false

  ci_attached_disks = [
    {
      name                      = "var/log" # Auto mount at /var/log
      size                      = 40
      snapshot_retention_policy = null
      snapshot_schedule         = null
      snapshot                  = null
      options = {
        auto_delete = true
        mode        = "READ_WRITE"
        type        = "pd-standard"
      }
    },
  ]

  # instance-template
  ci_machine_type              = "n2-custom-2-4096"
  ci_disk_source_image_family  = "packer-base-redis-v5-0-8"
  ci_disk_source_image_project = "tkpd-infra-production"
  ci_disk_type                 = "pd-ssd"
  ci_disk_size_gb              = 100

  # networking
  ci_subnetwork         = "data-90a8834"
  ci_subnetwork_project = "tkpd-infra-production"

  # labels
  ci_labels_namespace = "ads"
  ci_labels_hostgroup = "topads-retargeting-redis-cluster"
  ci_labels_type      = "data"
  ci_optional_labels = {
    directorate     = "advt"
    under_migration = "false"
  }

  # startup script
  ci_metadata_startup_script = <<-EOF
    #!/bin/bash

    # Only used by base-* images!
    export SERVICE_NAME="topads-retargeting-redis-cluster"

    wget https://tokopedia-dpkg.s3-ap-southeast-1.amazonaws.com/packer/general/instance-userdata/userdata.sh
    chmod +x userdata.sh
    bash userdata.sh "$SERVICE_NAME" >/var/log/cloud-init-output.log 2>&1
  EOF

  # shutdown script
  ci_metadata_shutdown_script = <<-EOF
    #!/bin/bash

    wget https://tokopedia-dpkg.s3-ap-southeast-1.amazonaws.com/packer/general/instance-userdata/shutdown-userdata.sh
    chmod +x shutdown-userdata.sh
    bash shutdown-userdata.sh >/var/log/cloud-init-output.log 2>&1
  EOF

Question:- Showcase some production grade way of setting up the Redis Cluster ?

Answer → We usually do it with the help of Ansible :-

---
- name: Redis v5.0.8
  hosts: "{{ custom_hosts | default('tag_Hostgroup_topads_retargeting_redis_cluster') }}"
  vars:
    common_hostname: topads-retargeting-redis-cluster
    common_namespace: ads
    common_directorate: adss
    common_owner: topads-1
    common_owner_team: boss
    common_type: 'redis'
    common_disable_transparent_huge_pages: true
    metricsapp_configname: redis

    redis_version: 5.0.8
    redis_service_name: redis

    # --tags redis_config
    redis_cluster_enabled: 'yes'
    # Auto detect appropriate maxmemory based on machine Memory
    redis_maxmemory_auto: true
    redis_maxmemory_policy: volatile-lru

    # --tags redis_cluster_bootstrap
    # Specify number of replicas per master
    redis_cluster_bootstrap_replicas: 1
    # Used to detect all addresses to form the cluster on bootstrap
    redis_cluster_bootstrap_domain: '{{ consul_services[0].name | default(common_hostname) }}.service.consul'

    nrinfragent_integrations:
      - name: nri-redis
        state: latest
    nrinfragent_logs:
      - name: systemd-redis
        systemd: redis

    consul_services:
      - name: topads-retargeting-redis-cluster
        port: 6379
        address: '{{ ansible_default_ipv4.address }}'
        checks:
          - name: Check Redis Port
            interval: 5s
            tcp: 127.0.0.1:6379
      - name: redis_exporter
        port: 9121
        address: '{{ ansible_default_ipv4.address }}'
        tags:
          - topads-retargeting-redis-cluster
        checks:
          - name: Check redis_exporter HTTP API
            interval: 5s
            tcp: 127.0.0.1:9121
      - name: node_exporter
        port: 9100
        address: '{{ ansible_default_ipv4.address }}'
        tags:
          - topads-retargeting-redis-cluster
        checks:
          - name: Check node_exporter HTTP API
            interval: 5s
            tcp: 127.0.0.1:9100
    common_ansible_integration:
      inventories:
        - production:
            - gcp:
                - asia-southeast1: gcp/prod-ads/asia-southeast1/production
        - staging:
            - gcp:
                - asia-southeast1: gcp/stag-ads/asia-southeast1/staging

  roles:
    - common
    - consul
    - forensic_monitor
    - node_exporter
    - nrinfragent
    - redis
    - redis_exporter
    - { role: teleport_agent, tags: teleport-agent }
    - metricsapp

That’s all in this section. If you liked reading this blog, kindly do press on clap button multiple times, to indicate your appreciation. We would see you in next part of this series with Hands-On with Redis-Cluster.

References :-

Infinite Horizontal Scalability with Redis Cluster

Written by aditya goel