May 30, 2020

The Nuts and Bolts of Redis Cluster

Lately, I’ve been playing around with Redis (with cluster-mode enabled) quite a lot. And it’s a lot of fun.

Clustered Redis provides better availability and scalability. To have initial understanding of what it is and how it works, I would recommend reading the official Redis cluster 101 tutorial. It gives a very clear overview of Redis cluster and some basic usages.

However, this article will touch on some more practical topics, such as redis clusters in Docker, its Golang library support and some discussion on its limitations.

Setup using Docker

The tutorial mentioned above covered a way to manually start a Redis cluster locally. It would be great if we could recreate it using container technologies and streamline the process. Luckily we have Docker Compose to the rescue.

If you google keywords docker, redis and cluster, a top result would likely be a docker image from the company Bitnami, exactly what we’re gonna be using today.

Pull the image

The Docker Hub page for this image is at here.

1
$ docker pull bitnami/redis-cluster:latest

Write a Docker Compose file for it

Docker Compose is a tool for managing or composing multiple containers. You describe multiple containers in one file and can easily manage them through a single CLI entrance, which is docker-compose or docker compose.

The image that Bitnami provides is for a single Redis node, which we’ll need a multiple of to create a cluster. To do that, we are utilizing Redis Compose to help us create them, manage the network and setup the persistent storage.

As mentioned in the official tutorial, a Redis cluster should at least have 6 nodes (3 master nodes + 3 replicas).

Here is how the containers are laid out in the docker-compose.yml file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
version: "2"
services:
redis-node-0:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-0:/redis/data
environment:
- "ALLOW_EMPTY_PASSWORD=yes"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

redis-node-1:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-1:/redis/data
environment:
- "ALLOW_EMPTY_PASSWORD=yes"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

redis-node-2:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-2:/redis/data
environment:
- "ALLOW_EMPTY_PASSWORD=yes"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

redis-node-3:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-3:/redis/data
environment:
- "ALLOW_EMPTY_PASSWORD=yes"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

redis-node-4:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-4:/redis/data
environment:
- "ALLOW_EMPTY_PASSWORD=yes"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

redis-node-5:
image: docker.io/bitnami/redis-cluster:6.2
volumes:
- redis-cluster_data-5:/redis/data
environment:
- "ALLOW_EMPTY_PASSWORD=yes"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

redis-cluster-init:
image: docker.io/bitnami/redis-cluster:6.2
depends_on:
- redis-node-0
- redis-node-1
- redis-node-2
- redis-node-3
- redis-node-4
- redis-node-5
environment:
- "REDIS_CLUSTER_REPLICAS=1"
- "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
- "REDIS_CLUSTER_CREATOR=yes"

volumes:
redis-cluster_data-0:
driver: local
redis-cluster_data-1:
driver: local
redis-cluster_data-2:
driver: local
redis-cluster_data-3:
driver: local
redis-cluster_data-4:
driver: local
redis-cluster_data-5:
driver: local

Then, we can easily start the services by docker-compose up -d. Stop them by docker-compose stop. Stop and remove them by docker-compose down.

Network issues

Notice how we don’t have any services exposing ports to the host. This means none of the Redis node is accessible from the outside, not even from the host.

“Wouldn’t it be super easy if we just exposed the port 6379 of every Redis node service”, you may ask. Well, that’s where we’re running into the limitations of this approach, this is mostly related to how nodes in a Redis cluster communicate with each other.

The current mode we make the Docker Compose containers run in is the host mode, which means containers can access each other through their hostnames like redis-node-0. But the host or any other machine from outside the network cannot access them at all. If we were to expose the port 6379 of every Redis node, and map them to a port available on the host machine, e.g. 7000, 7001, 7002, 7003, 7004 and 7005 (6 nodes that is), it might not work the way we expected. According to the official tutorial, Redis under cluster mode listens on 2 ports other than just 1, one being what we’re already familiar with, which is used for the client to connect to. The other one is used to sync data and track status between the nodes. So this is one of the reasons why simply exposing one port isn’t enough.

As mentioned above, in host mode, containers connect with each other through hostnames. When our client randomly connects to one of the nodes and request to fetch a value for a specific key, if the key is unfortunately sharded elsewhere, and need a redirection to the actual node that stores the key, the node we randomly landed should return the address of the desired node. This address, however, is where the issues arise. Because nodes are aware of each other in the docker network using each other’s hostname, which is the internal hostname of the docker network the client from outside cannot successfully reach. So the redirect address the node returns is not useful to us, and we lose a way to know where the right node we should be landing.

Make our client be in the network

One way to solve this is to make the client exist in the network so that our client can access the nodes using their hostnames. In this way, the redirect address nodes respond us with is also a valid address to our client.

What we do is to add a new service to the docker-compose.yml. For example, if our client is a Golang service:

1
2
3
4
5
6
7
golang-cli:
image: golang:1.16.3
container_name: redis-cluster-golang-cli
entrypoint: /bin/sh
volumes:
- ../golib:/mnt/project/
tty: true

Then after you start all the services, you can attach to the golang-cli console by the command:

1
docker exec -it redis-cluster-golang-cli /bin/bash

-it or --interactive flag means we are getting into the interactive mode that will keep stdin open so that we can type things. redis-cluster-golang-cli is the container name, you can also use the container id. /bin/bash is the shell we’re using, /bin/sh is also fine if bash isn’t available in the image.

Golang library

One of the popular Golang library that has cluster support is go-redis. It handles the redirection for us, so we don’t have to worry about the redirect address the nodes are returning.

Update: The Golang library that I’ve been working on build upon this and have cluster support with asynchronous request and automatically batching features. I’ll probably talk about this in the future.

The library has made it very easy. In most cases, you just treat it as a normal client that does secretly clustering stuffs in the background.

1
2
3
4
5
6
7
8
9
10
11
12
13
// You can either provide 1 or more node addresses since
// the library will find the rest of the cluster itself.
rdb := redis.NewClusterClient(&redis.ClusterOptions{
Addrs: []string{"redis-node-0:6379"},
})

// Treat it as a normal Redis client
err := rdb.Set(ctx, "key", "value", 0).Err()
val, err := rdb.Get(ctx, "key").Result()
if err != nil {
panic(err)
}
fmt.Println("key", val)

But there are some limitations especially when it comes to batch commands. For example, use command del to delete multiple keys may not work. Redis Cluster DOES NOT support multiple-key command if these keys don’t belong to the same slot. In fact, many other commands that would touch multiple-key have this restriction too.

I probably would write more on this later and provide more details.

About this Post

This post is written by Dizy, licensed under CC BY-NC 4.0.