-
Notifications
You must be signed in to change notification settings - Fork 45
How to Cluster
Redis-trib.py provides a set of CLI approaches to manipulate or retrieve information from Redis clusters. Here are some typical problems that Redis-trib.py could help to solve.
Precondition is that several Redis nodes, with cluster-enabled set to yes, are running, and those Redis nodes do not belong to any cluster yet.
We describe those kind of Redis as "free node", which means its reply to a CLUSTER INFO comamnd is like the following
cluster_state:fail # cluster is failed
cluster_slots_assigned:0 # no slots assigned
...
cluster_known_nodes:1 # no other nodes
...
Suppose there are 2 free nodes listening at 127.0.0.1:7000 and 127.0.0.1:7001. To create a cluster with them, the command is
redis-trib.py create 127.0.0.1:7000 127.0.0.1:7001
After the command, you could check the cluster status by a CLUSTER NODES command to any of the nodes
shell> redis-cli -p 7000 cluster nodes
02cf4d87f361ec8424d49eeb7939bd4f92842677 127.0.0.1:7001 master - 0 1472609932247 1 connected 0-8191
5b785cff85591b52c99f1bfc9dd5738637780fa7 127.0.0.1:7000 myself,master - 0 0 0 connected 8192-16383
Suppose you have a free node at 127.0.0.1:7002, and another node at 127.0.0.1:7000 who is already in a cluster as a master. To make 127.0.0.1:7002 replicate 127.0.0.1:7000 as a slave, the command is
redis-trib.py replicate --master-addr 127.0.0.1:7000 --slave-addr 127.0.0.1:7002
After the command, the reply to a CLUSTER NODES command may look like
shell> redis-cli -p 7000 cluster nodes
17e632ea496cdb872dad7cae74c0d4c89ddc7194 127.0.0.1:7002 slave 5b785cff85591b52c99f1bfc9dd5738637780fa7 0 1472610279860 2 connected
02cf4d87f361ec8424d49eeb7939bd4f92842677 127.0.0.1:7001 master - 0 1472610279359 1 connected 0-8191
5b785cff85591b52c99f1bfc9dd5738637780fa7 127.0.0.1:7000 myself,master - 0 0 0 connected 8192-16383
Suppose you have a free node at 127.0.0.1:7003 and want it to an existing cluster, and one of the cluster nodes is at 127.0.0.1:7000. The command is
redis-trib.py add_node --new-addr 127.0.0.1:7003 --existing-addr 127.0.0.1:7000
And the cluster nodes will become
shell> redis-cli -p 7000 cluster nodes
17e632ea496cdb872dad7cae74c0d4c89ddc7194 127.0.0.1:7002 slave 5b785cff85591b52c99f1bfc9dd5738637780fa7 0 1472610419624 2 connected
3c7d2a6f0e8ad16fd1767985f41e48e517955ab0 127.0.0.1:7003 master - 0 1472610419123 3 connected
02cf4d87f361ec8424d49eeb7939bd4f92842677 127.0.0.1:7001 master - 0 1472610420126 1 connected 0-8191
5b785cff85591b52c99f1bfc9dd5738637780fa7 127.0.0.1:7000 myself,master - 0 0 0 connected 8192-16383
You may notice that in the above operation the new comer does not undertake any data slots yet. So you may want to migrate slots between masters (you may also call this process as "reshard the data", or "balance the load").
Let's migrate 3000 slots from 127.0.0.1:7000 to 127.0.0.1:7003
redis-trib.py migrate --src-addr 127.0.0.1:7000 --dst-addr 127.0.0.1:7003 8192-11191
The last part of the command "8192-11191" is a slot range that contains slot #8192, #8193, ... #11191. It could have multiple pieces, like
redis-trib.py migrate --src-addr 127.0.0.1:7001 --dst-addr 127.0.0.1:7003 0 2 4-7
The slot ranges are "0 2 4-7", which meas a single slot #0 and another single slot #2, and a range #4 #5 #6 #7 are migrated from 127.0.0.1:7001 to 127.0.0.1:7003.
After all those migration, the cluster nodes are
shell> redis-cli -p 7000 cluster nodes
17e632ea496cdb872dad7cae74c0d4c89ddc7194 127.0.0.1:7002 slave 5b785cff85591b52c99f1bfc9dd5738637780fa7 0 1472611295594 2 connected
3c7d2a6f0e8ad16fd1767985f41e48e517955ab0 127.0.0.1:7003 master - 0 1472611295093 3 connected 0 2 4-7 8192-11191
02cf4d87f361ec8424d49eeb7939bd4f92842677 127.0.0.1:7001 master - 0 1472611296596 1 connected 1 3 8-8191
5b785cff85591b52c99f1bfc9dd5738637780fa7 127.0.0.1:7000 myself,master - 0 0 0 connected 11192-16383
There are some accidents that could happen while migrating slots and leaving some slots in an unstable status
shell> redis-cli -p 7001 cluster nodes
3c7d2a6f0e8ad16fd1767985f41e48e517955ab0 127.0.0.1:7003 master - 0 1472611630683 3 connected 0 2 4-6 8192-11191
5b785cff85591b52c99f1bfc9dd5738637780fa7 127.0.0.1:7000 master - 0 1472611630182 0 connected 11192-16383
17e632ea496cdb872dad7cae74c0d4c89ddc7194 127.0.0.1:7002 slave 5b785cff85591b52c99f1bfc9dd5738637780fa7 0 1472611631184 2 connected
02cf4d87f361ec8424d49eeb7939bd4f92842677 127.0.0.1:7001 myself,master - 0 0 4 connected 1 3 7-8191 [7->-3c7d2a6f0e8ad16fd1767985f41e48e517955ab0]
You could see the unstable slots in the reply of a CLUSTER NODES. [7->-3c7d2a6f0e8ad16fd1767985f41e48e517955ab0] indicates that slot #7 is being migrated.
To fix the migrating status at 127.0.0.1:7001, the command is
redis-trib.py fix --addr 127.0.0.1:7001
Suppose that 127.0.0.1:7003, who has no slave, crashes. And for some reason you're unable to restart it. However there is a free node 127.0.0.1:7004 that you would like it to take over the failed slots. The command is
redis-trib.py rescue --new-addr 127.0.0.1:7004 --existing-addr 127.0.0.1:7000
To delete the master node at 127.0.0.1:7004 from the cluster, the command is
redis-trib.py del_node --addr 127.0.0.1:7004
The command will randomly migrate the slots assigned by 127.0.0.1:7004 to other masters, and 127.0.0.1:7004 will become a free node.
To delete the master node at 127.0.0.1:7002 from the cluster, the command is
redis-trib.py del_node --addr 127.0.0.1:7002
After the command 127.0.0.1:7002 will become a free node.
Suppose there is a cluster consisting of the following nodes
shell> redis-cli -p 7004 cluster nodes
0d48baf3b863e6d421b2625455819875e5bdfbfb 127.0.0.1:7005 master - 0 1472613526218 9 connected 5462-10922
b3c2b6eeba9d1d28fadf3ea7602513c989963090 127.0.0.1:7007 slave b485fe5cd3b8deb12d31c45a04bdb5c885ae0a8a 0 1472613527220 10 connected
47da5bb50e6fc00c30dcc96b46d20d3c7f26e0e6 127.0.0.1:7006 master - 0 1472613527220 8 connected 10923-16383
b485fe5cd3b8deb12d31c45a04bdb5c885ae0a8a 127.0.0.1:7004 myself,master - 0 0 0 connected 0-5461
You may want to know the maxmemory configuration of each node. The command is
shell> redis-trib.py execute --addr 127.0.0.1:7004 config get maxmemory
127.0.0.1:7005 +['maxmemory', '2000000000']
127.0.0.1:7007 +['maxmemory', '0']
127.0.0.1:7006 +['maxmemory', '2000000000']
127.0.0.1:7004 +['maxmemory', '1000000000']
The option --addr 127.0.0.1:7004 indicates one of the cluster nodes is at 127.0.0.1:7004. Redis-trib.py will discover all other nodes and send the command to each of them.
The output is one line per node. Since CONFIG GET command returns a list, it will be printed in a Python list form after the node address.
To only execute the command at masters or slaves, add --master-only or --slave-only option
shell> redis-trib.py execute --addr 127.0.0.1:7004 --master-only config get maxmemory
127.0.0.1:7005 +['maxmemory', '2000000000']
127.0.0.1:7006 +['maxmemory', '2000000000']
127.0.0.1:7004 +['maxmemory', '1000000000']
shell> redis-trib.py execute --addr 127.0.0.1:7004 --slave-only config get maxmemory
127.0.0.1:7007 +['maxmemory', '0']
Another example is to set the same maxmemory across each node. The command is
redis-trib.py execute --addr 127.0.0.1:7004 config set maxmemory 2000m
You could flush all the data in the cluster (this is DANGEROUS!) by
redis-trib.py execute --addr 127.0.0.1:7004 --master-only FLUSHALL