Recently, I upgraded my servers as usual. I'm guilty of relying on a script that does basically everything you shouldn't do. It basically does apt-get update && apt-get upgrade -y and then clean stale packages. Not only that but I use watchtower to update my docker images being mostly on latest tags. This is a bad idea, but I do it anyway. So I do basically everything wrong. But it works for me, so I keep doing it. I didn't face many problems in the past, usually a couple of migration in some of the apps I'm using but nothing more than that.

I don't run anything critical on my servers, so I don't care much about the risk of breaking something. I just want to keep things up to date and running. But this time, I faced a problem that I didn't expect. The upgrade to Docker v29 changed the minimum API version required for compatibility, and it broke many things.

The obvious one was that my traefik container stopped working and then everything else stopped working on that server. It is a mini PC server in my closet at home. I'm currently even in the Netherlands not in my home. I use pangolin hosted on a VPS as replacement for Cloudflare tunnels (which had huge downtime recently) and I use it to access my home server. But after running the upgrade script, I got bombarded with ntfy notifications from my uptime kuma instance telling me that many services/apps are down. I was like "what the hell is going on?". I checked the logs and saw that traefik was not starting because of an API version mismatch.

Short story short, I had two options, either to downgrade Docker to v28 or to upgrade traefik to a version that supports the new API version. I didn't know if the latter was possible, but then I kagi'd it and found out that it was possible so the traefik got release v3.6.1 which updates Docker client API version to 1.44, and then it was a matter of changing my old tag to the new one in my docker-compose.yml file and then running docker compose up -d to update the container.

Traefik is one of the few things that I don't put latest tag on, so I had to change the tag from v3.5.3 to v3.6.1. The reason is that I got bitten by the latest tag in the past with traefik and I don't want to repeat that mistake. Ironically, in that case latest would have worked, but I don't want to take that risk again.

So, I learned my lesson and would want a better upgrade workflow in the future. I will try to avoid using latest tags and instead use specific versions for my containers. I will probably need some way of checking updates for the containers I use and then update them manually. But to be honest, I wouldn't be doing this anytime soon.