Appserver-buster-upgrade-2021

Observations of differences between MediaWiki servers on stretch vs buster, before and after task T245757, January 2021

TCP errors

At first glance it appears as if TCP errors have been reduced when looking at these 2 example hosts. The data gap in the middle is when the reimaging to buster happened.

mw1263 - TCP errors before and after the upgrade from stretch to buster on 2021-01-27

mw1268 - TCP errors before and after the upgrade from stretch to buster on 2021-01-27

But once you zoom out and look at an entire week before, it turns out it isn't actually a pattern.

mw1268 - TCP errors - over an entire week before the upgrade

disk utilization

Similary it first looks as if disk utilization went through the roof after the upgrade:

mw1268 - disk utilization before and after the upgrade from stretch to buster on 2021-01-27

But once you zoom out.. you see we have these spikes separate from the upgrade event:

mw1268 - disk utilization over a week leading up to the buster upgrade day

performance (avg response time)

Looking at average response time it can appear as if a buster server is actually slower if we look at mw1268 (stretch) vs mw1267 (buster) over a 6 hour span:

mw1268 (stretch) vs mw1267 (buster) - avg response time - over 6 hours on 2021-01-27

Similarly if we compare these hosts over a week:

mw1268 (stretch) vs mw1267 (buster) - avg response time - over a week in Jan 2021

sources

example grafana links, dashboards used: host-overview, application-servers-red-dashboard-wkandek

• https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&from=now-7d&to=now&var-server=mw1268&var-datasource=thanos&var-cluster=appserver

• https://grafana.wikimedia.org/d/MLl49gLGk/application-servers-red-dashboard-wkandek?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=appserver&var-method=GET&var-code=200&var-serverA=mw1268&var-serverB=mw1267