I got swarmed by a replication issue

Дата публикации: 02-06-2026 20:18:02

I recently worked with a client who runs a mature CI/CD pipeline on GitLab. Docker Swarm as the orchestrator, config files versioned and pulled at deploy time, the whole nine yards. They had been deploying standalone MariaDB instances this way for a while, without a hitch.Then came the ask : stand up an async replication cluster. One primary, two replicas, one MaxScale instance sitting in front of it all. Quite classical, nothing out of the ordinary really.First deployment ? Smooth. Replication is running, MaxScale routing reads to the replicas, writes to the primary. We are all happy.Then we had to redeploy the replicas.Every time we redeployed the replica containers, replication broke. Every time, the fix was the same manual ceremony : connect to each replica, run CHANGE MASTER TO, START SLAVE, check SHOW SLAVE STATUS. Everything works fine after that.Until the next redeployment that is.Same config files, same image. No changes, just as promised by the CI/CD pipeline. Or so i thought.The error log is always your first stop in this situation. And it told us exactly what was wrong :[ERROR] Failed to open the relay log \'./568165be8cc3-relay-bin.000002\' (relay_log_pos 4463864)[ERROR] Could not find target log during relay log initialization[ERROR] Failed to initialize the master info structureThe replica was looking for a relay log file that did not exist. But it took me a while (3 hours actually) to connect the dots as i focused on the last line for a while.Here is the thing about MariaDB relay log file naming : by default MariaDB derives the relay log basename from the server\'s hostname. On a bare metal or VM setup, that hostname is immutable. You set it once, it never changes.In Docker, not so much. By default, Docker does set a container’s hostname to its short container ID — a random hash that changes at every docker run or container recreation. Swarm makes it even worse as it also rotates task IDs on every redeployment, so even if we would try and rely on some predictable naming pattern, Swarm would break it further. A task that was replica_1.1.xk3f8a9b2c becomes replica_1.1.yz9q2m7nkp after redeployment.So on the previous deployment, the relay logs were named something like xk3f8a9b2c-relay-bin.000001. On the new deployment, the relay log index file references those old names. The new container starts up, checks the index, finds filenames that don\'t match its own generated basename (yz9q2m7nkp-relay-bin.000001) , can\'t locate the relay logs, and replication fails to resume.The root cause : i had not explicitly set the relay log basename in configuration files. I had set everything else. Just not that.Fixing it only took one line in my.cnf. Yes, one line and voilà !relay_log = relay-binHardcoding the relay log basename decouples replication state from container identity. Whatever hostname Docker Swarm decides to assign to the replica container, the relay log filenames stay constant. Replication resumes cleanly.While you’re at it, a few other directives are worth reviewing in any containerized replication setup :log_slave_updates = 1 — useful if you ever plan to chain replicas or use the replicas as a source for another downstream replica. Good habit regardless.relay_log_purge = 1 — keeps relay logs cleaned up automatically. In a container environment with limited storage you really want this on.relay_log_recovery = 1 — instructs the replica to recover relay log state from the master position on startup rather than relying on the relay log index. A solid safety net in ephemeral environments.expire_logs_days = 5 — instructs the replica to delete any binary log file in which the last event is older than 5 days. Helps with disk capacity planning.MaxScale was configured to monitor replication state on the replicas. And it did exactly what it was supposed to : the moment replication stopped, it pulled both replicas out of the read pool and flagged them as unavailable.MaxScale hid this bug from the app as expected. It also helped notice the issue. Proper replication monitoring in your proxy layer is not optional.One missing line in a configuration file ended up with three hours of head-scratching because of the following (wrong) assumptions :Docker Swarm and stateful services do not share the same concept of identity. Containers are stateless disposable resources, cattle. Rename, kill, redeploy as you see fit. Databases on the other hand are very much stateful. Stability is an implicit contract, including the hostname.Config-as-code is not the same as config completeness. The GitLab-driven deployment pipeline was running smooth. The config was versioned, reviewed, deployed automatically. The pipeline does exactly what it is instructed to, it does not tell what one forgot.Any MariaDB system variable that derives its default from a system value is a redeployment time-bomb in orchestrated environments. relay_log is the one that got our focus here. But the same logic applies to many other variables. If you run MariaDB in a stateless environment set these explicitly. Always.Stateful services in stateless environments will always bite back if you let the environment make assumptions on your behalf. The fix is one line but the overall lesson is to never let the orchestrator decide what you should control.Lessons learned !!If you’ve ever spent an afternoon staring at SHOW SLAVE STATUS wondering why a cluster that worked yesterday doesn\'t work today after a \"routine\" redeployment — I hope this saves you some trouble.Do not hesitate to reach out if you want to discuss your replication architecture or containerized database setup.
I got swarmed by a replication issue appeared first on MariaDB.org

Основное содержимое страницы с новостью.

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	MaxScale configuration synchronisation	0	3.91	04-04-2024
2	The Failover Brownout: Rethinking High Availability in MySQL Group Replication	0	4.06	15-06-2026
3	Understanding MySQL Replication “fatal error 1236”: “Replica has more GTIDs than the source has, using the source’s SERVER_UUID”	0	8.13	03-08-2026
4	MDEV-34412	0	21.11	17-06-2024
5	MariaDB High Availability: Maximum Availability Solutions in MariaDB 12.3 LTS	0	9.37	17-08-2026
6	Group Replication VS Percona XtraDB Cluster: The True Cost of Consistency	0	5.62	15-06-2026
7	Making the PostgreSQL Documentation Even Better	0	6.5	11-08-2026
8	Debugging with Ephemeral Containers	0	7.28	30-06-2026
9	Why I haven’t run my databases on Kubernetes	0	3.68	30-06-2026
10	MySQL and MariaDB High Availability vs. Disaster Recovery: What’s the Difference (and Why It Matters)	0	6.45	08-08-2026

Классификация: . Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 6.83. Источник: mariadb.org.