You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)

9 December 2022

  • curprev 01:1101:11, 9 December 2022imported>Stashbot 384,656 bytes +64,721 eevans@cumin1001: START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS buster

8 December 2022

  • curprev 01:0501:05, 8 December 2022imported>Stashbot 319,935 bytes +71,471 bblack: lvsNNNN: restart pybal to apply etcd key changes on all "high-traffic1" lvs at all sites - T324336

7 December 2022

  • curprev 00:2100:21, 7 December 2022imported>Stashbot 248,464 bytes +42,999 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1024.eqiad.wmnet with OS bullseye

6 December 2022

  • curprev 01:2501:25, 6 December 2022imported>Stashbot 205,465 bytes +81,524 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P42379 and previous config saved to /var/cache/conftool/dbconfig/20221206-012539-ladsgroup.json

4 December 2022

  • curprev 04:1904:19, 4 December 2022imported>Stashbot 123,941 bytes +177 TheresNoTime: T302486 : `[samtar@mwmaint1002 ~]$ mwscript maintenance/fixMergeHistoryCorruption.php --wiki enwiki --dry-run --ns 828`

3 December 2022

  • curprev 00:1700:17, 3 December 2022imported>Stashbot 123,764 bytes +12,040 cwhite: draining shards from logstash1010, logstash1033, logstash1034, logstash1035 - T321410

2 December 2022

  • curprev 00:0900:09, 2 December 2022imported>Stashbot 111,724 bytes −1,825,217 rzl@cumin1001: conftool action : set/pooled=no; selector: name=mw14(45|46).eqiad.wmnet,cluster=jobrunner

30 November 2022

  • curprev 01:2201:22, 30 November 2022imported>Stashbot 1,936,941 bytes +129,140 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T321126)', diff saved to https://phabricator.wikimedia.org/P41834 and previous config saved to /var/cache/conftool/dbconfig/20221130-012218-marostegui.json

29 November 2022

  • curprev 01:1701:17, 29 November 2022imported>Stashbot 1,807,801 bytes +121,792 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P41530 and previous config saved to /var/cache/conftool/dbconfig/20221129-011707-ladsgroup.json

27 November 2022

  • curprev 03:0103:01, 27 November 2022imported>Stashbot 1,686,009 bytes +943 ladsgroup@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Maint', diff saved to https://phabricator.wikimedia.org/P41257 and previous config saved to /var/cache/conftool/dbconfig/20221127-030126-ladsgroup.json

26 November 2022

  • curprev 21:3421:34, 26 November 2022imported>Stashbot 1,685,066 bytes +3,790 urandom: initiating Cassandra bootstrap, aqs1021-b -- T307802
  • curprev 01:1601:16, 26 November 2022imported>Stashbot 1,681,276 bytes +58,470 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P41244 and previous config saved to /var/cache/conftool/dbconfig/20221126-011647-ladsgroup.json

25 November 2022

  • curprev 01:1801:18, 25 November 2022imported>Stashbot 1,622,806 bytes +76,429 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P41071 and previous config saved to /var/cache/conftool/dbconfig/20221125-011818-marostegui.json

24 November 2022

  • curprev 01:2601:26, 24 November 2022imported>Stashbot 1,546,377 bytes +91,450 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321126)', diff saved to https://phabricator.wikimedia.org/P40894 and previous config saved to /var/cache/conftool/dbconfig/20221124-012652-marostegui.json

23 November 2022

  • curprev 01:1601:16, 23 November 2022imported>Stashbot 1,454,927 bytes +128,094 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp2041.codfw.wmnet with OS bullseye

22 November 2022

  • curprev 01:1401:14, 22 November 2022imported>Stashbot 1,326,833 bytes +77,259 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T323214)', diff saved to https://phabricator.wikimedia.org/P40411 and previous config saved to /var/cache/conftool/dbconfig/20221122-011404-ladsgroup.json

21 November 2022

  • curprev 01:0801:08, 21 November 2022imported>Stashbot 1,249,574 bytes +1,268 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5029.eqsin.wmnet with OS buster

19 November 2022

  • curprev 22:5122:51, 19 November 2022imported>Stashbot 1,248,306 bytes +1,417 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS buster
  • curprev 01:2401:24, 19 November 2022imported>Stashbot 1,246,889 bytes +59,627 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage

18 November 2022

  • curprev 01:2601:26, 18 November 2022imported>Stashbot 1,187,262 bytes +56,176 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['db2173']

17 November 2022

  • curprev 00:5900:59, 17 November 2022imported>Stashbot 1,131,086 bytes +67,747 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2125 (T323214)', diff saved to https://phabricator.wikimedia.org/P40031 and previous config saved to /var/cache/conftool/dbconfig/20221117-005929-ladsgroup.json

16 November 2022

  • curprev 01:1501:15, 16 November 2022imported>Stashbot 1,063,339 bytes +102,599 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye

15 November 2022

  • curprev 01:0701:07, 15 November 2022imported>Stashbot 960,740 bytes +99,923 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P39630 and previous config saved to /var/cache/conftool/dbconfig/20221115-010745-marostegui.json

12 November 2022

  • curprev 23:3423:34, 12 November 2022imported>Stashbot 860,817 bytes +12,581 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T318605)', diff saved to https://phabricator.wikimedia.org/P39371 and previous config saved to /var/cache/conftool/dbconfig/20221112-233420-ladsgroup.json
  • curprev 01:2101:21, 12 November 2022imported>Stashbot 848,236 bytes +62,212 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P39331 and previous config saved to /var/cache/conftool/dbconfig/20221112-012122-marostegui.json

11 November 2022

  • curprev 01:2201:22, 11 November 2022imported>Stashbot 786,024 bytes +125,544 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P39163 and previous config saved to /var/cache/conftool/dbconfig/20221111-012250-ladsgroup.json

10 November 2022

8 November 2022

  • curprev 22:0022:00, 8 November 2022imported>Stashbot 604,525 bytes +104,241 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • curprev 01:2501:25, 8 November 2022imported>Stashbot 500,284 bytes +154,243 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

6 November 2022

5 November 2022

  • curprev 12:5612:56, 5 November 2022imported>Stashbot 345,478 bytes +537 mfossati@deploy1002: Finished deploy [airflow-dags/platform_eng@c849762]: (no justification provided) (duration: 00m 49s)

4 November 2022

  • curprev 18:3118:31, 4 November 2022imported>Stashbot 344,941 bytes +23,994 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS buster

3 November 2022

  • curprev 22:4522:45, 3 November 2022imported>Stashbot 320,947 bytes +90,964 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 00s)

2 November 2022

  • curprev 23:2523:25, 2 November 2022imported>Stashbot 229,983 bytes +129,100 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37874 and previous config saved to /var/cache/conftool/dbconfig/20221102-232540-ladsgroup.json
  • curprev 01:1901:19, 2 November 2022imported>Stashbot 100,883 bytes −1,663,965 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37541 and previous config saved to /var/cache/conftool/dbconfig/20221102-011937-ladsgroup.json

31 October 2022

  • curprev 22:2322:23, 31 October 2022imported>Stashbot 1,764,848 bytes +82,444 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance

29 October 2022

28 October 2022

  • curprev 20:4220:42, 28 October 2022imported>Stashbot 1,681,955 bytes +34,207 mutante: clouddumps* - deployed gerrit:848444 - as kind of expected it fails - most likely the project dirs are not automatically created before rsync runs the first time - T57503
  • curprev 01:1501:15, 28 October 2022imported>Stashbot 1,647,748 bytes +124,757 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T318950)', diff saved to https://phabricator.wikimedia.org/P36942 and previous config saved to /var/cache/conftool/dbconfig/20221028-011505-ladsgroup.json

27 October 2022

  • curprev 01:1601:16, 27 October 2022imported>Stashbot 1,522,991 bytes +90,843 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS buster

25 October 2022

  • curprev 22:3322:33, 25 October 2022imported>Stashbot 1,432,148 bytes +92,794 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp4052.mgmt.ulsfo.wmnet with reboot policy FORCED
  • curprev 01:0901:09, 25 October 2022imported>Stashbot 1,339,354 bytes +82,471 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T321312)', diff saved to https://phabricator.wikimedia.org/P36164 and previous config saved to /var/cache/conftool/dbconfig/20221025-010943-ladsgroup.json

22 October 2022

  • curprev 03:1603:16, 22 October 2022imported>Stashbot 1,256,883 bytes +83 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
  • curprev 00:0300:03, 22 October 2022imported>Stashbot 1,256,800 bytes +69,045 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance

21 October 2022

  • curprev 01:1401:14, 21 October 2022imported>Stashbot 1,187,755 bytes +85,034 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 (T321312)', diff saved to https://phabricator.wikimedia.org/P35798 and previous config saved to /var/cache/conftool/dbconfig/20221021-011452-ladsgroup.json

19 October 2022

  • curprev 23:3323:33, 19 October 2022imported>Stashbot 1,102,721 bytes +35,546 wfan: civicrm upgraded from 477323fe to c96dd3ae
  • curprev 01:2001:20, 19 October 2022imported>Stashbot 1,067,175 bytes +11,210 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage

17 October 2022

  • curprev 23:1623:16, 17 October 2022imported>Stashbot 1,055,965 bytes +26,027 bblack@puppetmaster2001: conftool action : set/pooled=yes; selector: service=git-ssh

15 October 2022

  • curprev 23:2723:27, 15 October 2022imported>Stashbot 1,029,938 bytes +932 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool db1131 T320879', diff saved to https://phabricator.wikimedia.org/P35497 and previous config saved to /var/cache/conftool/dbconfig/20221015-232716-ladsgroup.json

14 October 2022

  • curprev 22:5622:56, 14 October 2022imported>Stashbot 1,029,006 bytes +9,465 mutante: pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud - out of disk space again - deleted 3.5GB job "1460" to unblock puppet compiling
  • curprev 01:2601:26, 14 October 2022imported>Stashbot 1,019,541 bytes +26,484 tstarling@deploy1002: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 03m 36s)

13 October 2022

  • curprev 00:5800:58, 13 October 2022imported>Stashbot 993,057 bytes +28,159 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4045.ulsfo.wmnet with OS buster

11 October 2022

  • curprev 21:3621:36, 11 October 2022imported>Stashbot 964,898 bytes +24,881 mutante: phab1001 / phab2001 - temp. disabled puppet; stopped ssh-phab service; scheduled icinga downtimes for ssh-phab pybal backend alerts - effectively "soft shutting down" the service - T296022

10 October 2022

  • curprev 21:1921:19, 10 October 2022imported>Stashbot 940,017 bytes +15,468 robh@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS bullseye

8 October 2022

  • curprev 06:5606:56, 8 October 2022imported>Stashbot 924,549 bytes +110 hashar: Restarting Gerrit to fix up replicaton to GitHub - T320305

7 October 2022

  • curprev 21:2921:29, 7 October 2022imported>Stashbot 924,439 bytes +12,004 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: debugging

6 October 2022

  • curprev 21:1321:13, 6 October 2022imported>Stashbot 912,435 bytes +26,746 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 01:1201:12, 6 October 2022imported>Stashbot 885,689 bytes +23,589 reedy@deploy1002: Finished deploy [integration/docroot@dc380cb]: Update jQuery (duration: 00m 11s)

5 October 2022

  • curprev 00:0500:05, 5 October 2022imported>Stashbot 862,100 bytes +37,349 sukhe: disable puppet on dns4003 till we resolve the puppet failures

3 October 2022

  • curprev 21:4521:45, 3 October 2022imported>Stashbot 824,751 bytes +32,264 robh@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

2 October 2022

  • curprev 08:1308:13, 2 October 2022imported>Stashbot 792,487 bytes +109 elukey: `apt-get clean` on an-airflow1001 to free some space on the root partition

1 October 2022

  • curprev 13:2413:24, 1 October 2022imported>Stashbot 792,378 bytes +460 fab@deploy1002: Finished deploy [airflow-dags/research@44a1158]: (no justification provided) (duration: 00m 08s)

30 September 2022

  • curprev 23:2623:26, 30 September 2022imported>Stashbot 791,918 bytes +15,804 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • curprev 00:3100:31, 30 September 2022imported>Stashbot 776,114 bytes +53,878 robh@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4045.ulsfo.wmnet with OS bullseye

29 September 2022

  • curprev 01:0101:01, 29 September 2022imported>Stashbot 722,236 bytes +72,509 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2037.codfw.wmnet with OS buster

28 September 2022

  • curprev 01:2201:22, 28 September 2022imported>Stashbot 649,727 bytes +31,209 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T314041)', diff saved to https://phabricator.wikimedia.org/P34972 and previous config saved to /var/cache/conftool/dbconfig/20220928-012205-ladsgroup.json

27 September 2022

25 September 2022

  • curprev 17:2917:29, 25 September 2022imported>Stashbot 587,776 bytes +2,487 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1053.eqiad.wmnet with OS bullseye

23 September 2022

  • curprev 19:1019:10, 23 September 2022imported>Stashbot 585,289 bytes +2,764 mforns@deploy1002: Finished deploy [airflow-dags/analytics@4c973d6]: (no justification provided) (duration: 00m 12s)

22 September 2022

  • curprev 22:2022:20, 22 September 2022imported>Stashbot 582,525 bytes +8,984 joal@deploy1002: Finished deploy [airflow-dags/analytics@901f810]: (no justification provided) (duration: 00m 11s)

21 September 2022

  • curprev 20:5120:51, 21 September 2022imported>Stashbot 573,541 bytes +10,680 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

20 September 2022

19 September 2022

  • curprev 22:5922:59, 19 September 2022imported>Stashbot 549,728 bytes +6,720 ebernhardson: T317200 start cirrussearch in-place reindex process for eqiad, codfw and cloudelastic

17 September 2022

16 September 2022

  • curprev 21:2921:29, 16 September 2022imported>Stashbot 538,373 bytes +24,573 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • curprev 00:1400:14, 16 September 2022imported>Stashbot 513,800 bytes +30,264 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 17s)

14 September 2022

  • curprev 22:0822:08, 14 September 2022imported>Stashbot 483,536 bytes +39,587 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1190 (T314041)', diff saved to https://phabricator.wikimedia.org/P34739 and previous config saved to /var/cache/conftool/dbconfig/20220914-220822-ladsgroup.json
  • curprev 01:1601:16, 14 September 2022imported>Stashbot 443,949 bytes +58,148 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P34670 and previous config saved to /var/cache/conftool/dbconfig/20220914-011637-ladsgroup.json

13 September 2022

  • curprev 00:5000:50, 13 September 2022imported>Stashbot 385,801 bytes +62,019 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab2001.codfw.wmnet with reason: syntax error in sudo

12 September 2022

  • curprev 01:2101:21, 12 September 2022imported>Stashbot 323,782 bytes +10,360 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T314041)', diff saved to https://phabricator.wikimedia.org/P34424 and previous config saved to /var/cache/conftool/dbconfig/20220912-012118-ladsgroup.json

10 September 2022

  • curprev 21:3321:33, 10 September 2022imported>Stashbot 313,422 bytes +11,937 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2146 (T312863)', diff saved to https://phabricator.wikimedia.org/P34396 and previous config saved to /var/cache/conftool/dbconfig/20220910-213300-ladsgroup.json
  • curprev 00:5000:50, 10 September 2022imported>Stashbot 301,485 bytes +17,350 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2137:3314 (T314041)', diff saved to https://phabricator.wikimedia.org/P34358 and previous config saved to /var/cache/conftool/dbconfig/20220910-005046-ladsgroup.json

9 September 2022

  • curprev 00:0900:09, 9 September 2022imported>Stashbot 284,135 bytes +69,901 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 22s)

8 September 2022

6 September 2022

  • curprev 23:3823:38, 6 September 2022imported>Stashbot 155,293 bytes +69,531 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T314041)', diff saved to https://phabricator.wikimedia.org/P33981 and previous config saved to /var/cache/conftool/dbconfig/20220906-233809-ladsgroup.json
  • curprev 01:0301:03, 6 September 2022imported>Stashbot 85,762 bytes +31,159 TimStarling: multi-DC stage 3: 2% of codfw/ulsfo/eqsin traffic going to codfw appservers, rolling out via puppet 00:54-01:24

5 September 2022

  • curprev 00:3600:36, 5 September 2022imported>Stashbot 54,603 bytes −976,269 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance

3 September 2022

  • curprev 23:5023:50, 3 September 2022imported>Stashbot 1,030,872 bytes +5,252 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T312863)', diff saved to https://phabricator.wikimedia.org/P33759 and previous config saved to /var/cache/conftool/dbconfig/20220903-235001-ladsgroup.json

2 September 2022

  • curprev 19:0319:03, 2 September 2022imported>Stashbot 1,025,620 bytes +16,632 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

1 September 2022

  • curprev 20:5120:51, 1 September 2022imported>Stashbot 1,008,988 bytes +25,876 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 01:2001:20, 1 September 2022imported>Stashbot 983,112 bytes +32,085 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase201[3-8].codfw.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001

31 August 2022

  • curprev 00:1500:15, 31 August 2022imported>Stashbot 951,027 bytes +47,602 ryankemper@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw es7 cluster upgrade - ryankemper@cumin2002 - T316719

30 August 2022

  • curprev 01:0401:04, 30 August 2022imported>Stashbot 903,425 bytes +47,117 TimStarling: setting scaling_governor=performance on all mediawiki servers, via puppet gerrit 826405

28 August 2022

  • curprev 21:0321:03, 28 August 2022imported>Stashbot 856,308 bytes +47,266 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P33561 and previous config saved to /var/cache/conftool/dbconfig/20220828-210336-ladsgroup.json
  • curprev 01:0501:05, 28 August 2022imported>Stashbot 809,042 bytes +25,117 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P33423 and previous config saved to /var/cache/conftool/dbconfig/20220828-010522-ladsgroup.json

27 August 2022

  • curprev 01:0301:03, 27 August 2022imported>Stashbot 783,925 bytes +51,766 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T316186)', diff saved to https://phabricator.wikimedia.org/P33346 and previous config saved to /var/cache/conftool/dbconfig/20220827-010313-ladsgroup.json

26 August 2022

  • curprev 00:3800:38, 26 August 2022imported>Stashbot 732,159 bytes +75,646 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2131 (T312160)', diff saved to https://phabricator.wikimedia.org/P33172 and previous config saved to /var/cache/conftool/dbconfig/20220826-003819-ladsgroup.json

25 August 2022

  • curprev 01:2501:25, 25 August 2022imported>Stashbot 656,513 bytes +49,388 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P32985 and previous config saved to /var/cache/conftool/dbconfig/20220825-012538-ladsgroup.json

23 August 2022

  • curprev 22:3122:31, 23 August 2022imported>Stashbot 607,125 bytes +44,996 mutante: mwmaint1002 - find /var/lib/puppet/clientbucket -type f -size +100M -delete
  • curprev 01:1101:11, 23 August 2022imported>Stashbot 562,129 bytes +42,364 TimStarling: on mw1411, mw1413, mw1419, mw1429, mw1431, mw1433: set scaling_governor to powersave and energy_performance_preference to performance
(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)