We are not able to use the redis-trib fix command to fix a cluster when the master and slave for a particular set of slots both go down at the same time.

Jun 01, 2018  How to Install and Configure a Redis Cluster on Ubuntu 16.04. Updated Friday, June 1. All nodes agree about slots configuration. Check for open slots. Check slots coverage. OK All 16384 slots covered. See all the current nodes connected to the cluster by using the redis-cli tool. Aug 19, 2015  From command LIne. First connect to the local instance, running by default on port 6379. In case you've changed the port, modify the command accordingly. OK All nodes agree about slots configuration. Check for open slots. Check slots coverage. OK All 16384 slots covered. Aug 11, 2016  ERR Not all 16384 slots are covered by nodes. Fixing slots coverage. List of not covered slots: 5460 Slot 5460 has keys in 0 nodes: The folowing uncovered slots have no keys across the cluster: 5460 Fix these slots by covering with a random node?

redis 3.2.11
redis-cli 4.0.1
redis-trib (redis 3.3.3 gem)

Our use case is we are writing a redis cluster orchestrator, where nodes are added and removed often. When a node goes down, we want slots to be covered as quickly as possible by other masters in the cluster. Also, we are only using redis as a cache right now, so we don't necessarily care that assigning slots to the another master results in data loss.

Steps to reproduce

1. 3 masters, 1 slave per master
2. Choose a master/slave pair (which covers a single set of slots) and delete both VM hosts at the same time
3. Run /bin/redis-cli -p [port] -h [host] cluster forget [nodeIdOfDeletedMaster] on all other active nodes in the cluster
4. /bin/redis-trib check [host:port]

Result:

redis-trib check shows missing slots message (as expected):

What is the recommended way to recover in this situation? Both rebalance and fix do not work for us. Should we be using addslots to manually assign slots to other masters?

Result of rebalance:
Rebalance is not allowed to run until we fix the cluster.

Output of rebalance:

Fix distributes slots to both master and slave IPs (which surprised us, as we thought it would only use masters). After fix completes, the nodes don't agree about configuration, and gossip does not correct the disagreement, which is an even worse situation for the cluster. The issue is easy for us to reproduce.

Redis Not All 16384 Slots Are Covered By Nodes

Is there an issue with fix, or is it not correct to attempt to use it in our use case where it is possible that slots will be lost because both the master and its backup slaves will all be lost at once. (We noticed in this issue #3007 (comment) that it was mentioned that fix will not fix this situation anyway). In our use case, we have nodes coming and going often, so we do not always know when a set of slots is going away completely.

Not All 16384 Slots Are Covered By Nodes Located

Output of fix (truncated):