MySQL Orchestrator is the go-to tool for automated failover and replication topology management. This guide walks through a production-ready HA cluster setup from scratch.
What is MySQL Orchestrator?
Orchestrator is an open-source MySQL replication topology manager that provides:
- Automated master failover
- Replication topology visualization
- Safe replica promotion with anti-split-brain logic
- REST API and web UI for topology control
Architecture Overview
A production Orchestrator deployment typically looks like:
text
App Tier
└─ ProxySQL / HAProxy
├─ MySQL Primary (writes)
├─ MySQL Replica1 (reads)
└─ MySQL Replica2 (reads)
Orchestrator (x3 for Raft consensus)
monitors topology, triggers failoverInstallation
bash
# Download Orchestrator binary
wget https://github.com/openark/orchestrator/releases/download/v3.2.6/orchestrator-3.2.6-linux-amd64.tar.gz
tar xzf orchestrator-3.2.6-linux-amd64.tar.gz
sudo mv orchestrator /usr/local/bin/
# Create config directory
sudo mkdir -p /etc/orchestratorCore Configuration
json
{
"MySQLTopologyUser": "orchestrator",
"MySQLTopologyPassword": "s3cr3t",
"MySQLOrchestratorHost": "127.0.0.1",
"MySQLOrchestratorPort": 3306,
"MySQLOrchestratorDatabase": "orchestrator",
"RecoverMasterClusterFilters": ["*"],
"RecoverIntermediateMasterClusterFilters": ["*"],
"FailMasterPromotionOnLagMinutes": 0,
"MasterFailoverDetachReplicaMasterHost": true,
"PostMasterFailoverProcesses": [
"notify-failover.sh {failedHost} {successorHost}"
]
}MySQL User for Orchestrator
sql
CREATE USER 'orchestrator'@'%' IDENTIFIED BY 's3cr3t';
GRANT SUPER, PROCESS, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'orchestrator'@'%';
GRANT SELECT ON mysql.slave_master_info TO 'orchestrator'@'%';
GRANT SELECT ON performance_schema.* TO 'orchestrator'@'%';Raft Mode for High Availability
Run three Orchestrator nodes in Raft consensus mode so no single Orchestrator node is a SPOF:
json
{
"RaftEnabled": true,
"RaftDataDir": "/var/lib/orchestrator/raft",
"RaftBind": "10.0.1.10",
"RaftNodes": ["10.0.1.10", "10.0.1.11", "10.0.1.12"]
}Testing Failover
bash
# Simulate primary failure
orchestrator-client -c recover -i primary-host:3306
# Check topology after recovery
orchestrator-client -c topology -i cluster-name
# Graceful master move
orchestrator-client -c graceful-master-takeover -i primary-host:3306 -d replica1:3306Tip: Always test failover in staging before relying on Orchestrator in production. Verify your
PostMasterFailoverProcesses scripts update DNS or ProxySQL routing correctly.Monitoring Orchestrator
Orchestrator exposes a rich REST API. Scrape it with Prometheus using the orchestrator-exporter sidecar:
bash
curl http://orchestrator:3000/api/clusters-info | jq .Key Takeaways
- Use Raft mode (3 nodes) so Orchestrator itself is not a SPOF
- Grant minimal but sufficient MySQL privileges to the orchestrator user
- Hook
PostMasterFailoverProcessesto update ProxySQL or DNS - Test failover regularly — at least once per quarter
JusDB Can Help
Setting up MySQL HA with Orchestrator is complex. JusDB's engineers have deployed Orchestrator for dozens of production clusters. Talk to us about designing your failover strategy.