You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)

9 June 2023

  • curprev 01:0801:08, 9 June 2023imported>Stashbot 340,662 bytes +52,887 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002"

8 June 2023

  • curprev 01:0801:08, 8 June 2023imported>Stashbot 287,775 bytes +111,676 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T336886)', diff saved to https://phabricator.wikimedia.org/P49233 and previous config saved to /var/cache/conftool/dbconfig/20230608-010853-ladsgroup.json

7 June 2023

  • curprev 01:1601:16, 7 June 2023imported>Stashbot 176,099 bytes +73,106 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P48979 and previous config saved to /var/cache/conftool/dbconfig/20230607-011602-ladsgroup.json

6 June 2023

  • curprev 01:2101:21, 6 June 2023imported>Stashbot 102,993 bytes +67,142 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T336886)', diff saved to https://phabricator.wikimedia.org/P48826 and previous config saved to /var/cache/conftool/dbconfig/20230606-012058-ladsgroup.json

3 June 2023

  • curprev 13:4113:41, 3 June 2023imported>Stashbot 35,851 bytes +556 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on an-test-worker1001.eqiad.wmnet with reason: Host under testing/upgrade

2 June 2023

  • curprev 20:1620:16, 2 June 2023imported>Stashbot 35,295 bytes +4,802 apergos: rsync in ariel screen session, bwlimit 100000, running on dumpsdata1003, pulling from dumpsdata1002, copying over 'other dumps'

1 June 2023

30 May 2023

29 May 2023

  • curprev 15:1915:19, 29 May 2023imported>Stashbot 851,144 bytes +7,988 eoghan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: This is being worked on

28 May 2023

  • curprev 13:1913:19, 28 May 2023imported>Stashbot 843,156 bytes +471 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/thumbor: sync

27 May 2023

  • curprev 21:4021:40, 27 May 2023imported>Stashbot 842,685 bytes +225 Amir1: insert into templatelinks (tl_from, tl_from_namespace, tl_target_id) values (686, 0, 199); on db1154:3113 (T337446)
  • curprev 00:0300:03, 27 May 2023imported>Stashbot 842,460 bytes +17,800 tzatziki: removing 1 file for legal compliance

25 May 2023

24 May 2023

22 May 2023

  • curprev 23:2923:29, 22 May 2023imported>Stashbot 720,989 bytes +27,509 eileen: civicrm upgraded from cc9593d0 to 7eae24d5

21 May 2023

  • curprev 07:4507:45, 21 May 2023imported>Stashbot 693,480 bytes +523 jelto@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply

20 May 2023

19 May 2023

  • curprev 21:2221:22, 19 May 2023imported>Stashbot 691,547 bytes +18,498 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

18 May 2023

  • curprev 23:2623:26, 18 May 2023imported>Stashbot 673,049 bytes +20,985 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.9 refs T330215

17 May 2023

  • curprev 22:3022:30, 17 May 2023imported>Stashbot 652,064 bytes +51,758 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:2100:21, 17 May 2023imported>Stashbot 600,306 bytes +39,703 krinkle@deploy1002: Synchronized src/: I4cfa4a2474b4e (duration: 06m 01s)

15 May 2023

  • curprev 23:3723:37, 15 May 2023imported>Stashbot 560,603 bytes +15,254 eileen: civicrm upgraded from db6e8d69 to ef7b3822

12 May 2023

  • curprev 22:5922:59, 12 May 2023imported>Stashbot 545,349 bytes +13,865 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudswift1001.eqiad.wmnet with OS bullseye
  • curprev 01:0801:08, 12 May 2023imported>Stashbot 531,484 bytes +31,985 denisse@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts prometheus6001.drmrs.wmnet

10 May 2023

  • curprev 22:0822:08, 10 May 2023imported>Stashbot 499,499 bytes +53,507 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2021.codfw.wmnet with OS buster

9 May 2023

  • curprev 23:4323:43, 9 May 2023imported>Stashbot 445,992 bytes +64,686 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad
  • curprev 00:4300:43, 9 May 2023imported>Stashbot 381,306 bytes +82,838 eileen: civicrm upgraded from d5229d22 to 301e24e4

7 May 2023

  • curprev 00:5400:54, 7 May 2023imported>Stashbot 298,468 bytes +1,045 sukhe: restart haproxy on cp1087: T334448

5 May 2023

  • curprev 23:2423:24, 5 May 2023imported>Stashbot 297,423 bytes +37,811 tzatziki: removing emails from 230 users per self-requests
  • curprev 01:1701:17, 5 May 2023imported>Stashbot 259,612 bytes +123,759 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P47724 and previous config saved to /var/cache/conftool/dbconfig/20230505-011700-ladsgroup.json

3 May 2023

  • curprev 23:5523:55, 3 May 2023imported>Stashbot 135,853 bytes +103,007 eileen: config revision changed from 2995f558 to 26147e89
  • curprev 01:2401:24, 3 May 2023imported>Stashbot 32,846 bytes +22,306 eileen: civicrm upgraded from 09d2eefd to c6149ad2

1 May 2023

29 April 2023

  • curprev 23:0323:03, 29 April 2023imported>Stashbot 738,510 bytes +490 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: Maint

28 April 2023

  • curprev 22:4622:46, 28 April 2023imported>Stashbot 738,020 bytes +9,846 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

27 April 2023

  • curprev 22:1722:17, 27 April 2023imported>Stashbot 728,174 bytes +23,740 zabe@deploy1002: Finished scap: T334295 (duration: 06m 58s)
  • curprev 00:0100:01, 27 April 2023imported>Stashbot 704,434 bytes +39,904 zabe@deploy1002: Finished scap: T334295 (duration: 06m 53s)

25 April 2023

  • curprev 21:4021:40, 25 April 2023imported>Stashbot 664,530 bytes +14,368 mutante: gerrit1003 - chown -R gerrit2:gerrit2 /var/lib/gerrit2/review_site/ - T326368

24 April 2023

  • curprev 23:1523:15, 24 April 2023imported>Stashbot 650,162 bytes +20,174 eileen: civicrm upgraded from c17c8db2 to 26150ed4

22 April 2023

  • curprev 05:4105:41, 22 April 2023imported>Stashbot 629,988 bytes +821 joe: <thumbor/codfw>$ helmfile --state-values-set roll_restart=1 -e codfw sync

21 April 2023

  • curprev 18:2818:28, 21 April 2023imported>Stashbot 629,167 bytes +2,295 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.remove-ghost-objects (exit_code=99) from container wikipedia-en-local-public.a8 in codfw
  • curprev 00:3700:37, 21 April 2023imported>Stashbot 626,872 bytes +23,363 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2010.codfw.wmnet with OS bullseye

20 April 2023

  • curprev 00:0200:02, 20 April 2023imported>Stashbot 603,509 bytes +50,285 mutante: LDAP - adding uid fnavas-foundation to group wmf - T331482

19 April 2023

  • curprev 01:1701:17, 19 April 2023imported>Stashbot 553,224 bytes +56,948 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T333332)', diff saved to https://phabricator.wikimedia.org/P47161 and previous config saved to /var/cache/conftool/dbconfig/20230419-011754-ladsgroup.json

18 April 2023

  • curprev 00:5400:54, 18 April 2023imported>Stashbot 496,276 bytes +58,776 eevans@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cassandra-dev2001.codfw.wmnet

16 April 2023

  • curprev 07:5407:54, 16 April 2023imported>Stashbot 437,500 bytes +394 vgutierrez: restart haproxy on cp2033 to clear unexpected service restart alerts - T334448

15 April 2023

  • curprev 07:1307:13, 15 April 2023imported>Stashbot 437,106 bytes +11,341 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T333332)', diff saved to https://phabricator.wikimedia.org/P46929 and previous config saved to /var/cache/conftool/dbconfig/20230415-071327-ladsgroup.json
  • curprev 01:2301:23, 15 April 2023imported>Stashbot 425,765 bytes +83,532 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P46892 and previous config saved to /var/cache/conftool/dbconfig/20230415-012305-ladsgroup.json

14 April 2023

  • curprev 01:0801:08, 14 April 2023imported>Stashbot 342,233 bytes +39,811 fab@deploy2002: Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 10s)

13 April 2023

  • curprev 01:2301:23, 13 April 2023imported>Stashbot 302,422 bytes +92,852 fab@deploy2002: Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 10s)

12 April 2023

  • curprev 01:1601:16, 12 April 2023imported>Stashbot 209,570 bytes +52,844 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T333332)', diff saved to https://phabricator.wikimedia.org/P46380 and previous config saved to /var/cache/conftool/dbconfig/20230412-011619-ladsgroup.json

11 April 2023

  • curprev 00:3700:37, 11 April 2023imported>Stashbot 156,726 bytes +28,171 eileen: civicrm upgraded from 001e156a to bc2f5ccc

8 April 2023

  • curprev 17:5717:57, 8 April 2023imported>Stashbot 128,555 bytes +152 jclark@cumin1001: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1073']

7 April 2023

  • curprev 18:1918:19, 7 April 2023imported>Stashbot 128,403 bytes +1,699 xcollazo@deploy2002: Finished deploy [airflow-dags/platform_eng@5c4ebda]: (no justification provided) (duration: 00m 35s)
  • curprev 01:1701:17, 7 April 2023imported>Stashbot 126,704 bytes +39,724 urandom: rebooting sessionstore1001 — T327954

6 April 2023

  • curprev 00:5000:50, 6 April 2023imported>Stashbot 86,980 bytes +44,195 urandom: rebooting sessionstore1001 — T327954

4 April 2023

  • curprev 23:4023:40, 4 April 2023imported>Stashbot 42,785 bytes +27,465 jclark@cumin1001: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirtlocal1002.mgmt.eqiad.wmnet with reboot policy FORCED

3 April 2023

  • curprev 21:5321:53, 3 April 2023imported>Stashbot 15,320 bytes +15,053 ryankemper: T331896 `sudo -E cumin -b 4 'wdqs*' 'sudo run-puppet-agent'`

1 April 2023

31 March 2023

  • curprev 01:0801:08, 31 March 2023imported>Stashbot 921,204 bytes +54,765 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gerrit1003.wikimedia.org with OS bullseye

30 March 2023

  • curprev 00:2700:27, 30 March 2023imported>Stashbot 866,439 bytes +44,733 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['db1207']

29 March 2023

  • curprev 00:4200:42, 29 March 2023imported>Stashbot 821,706 bytes +28,993 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2035.codfw.wmnet,service=ats-be

27 March 2023

  • curprev 23:4723:47, 27 March 2023imported>Stashbot 792,713 bytes +20,868 mutante: people1003 - taking down apache to provoke monitoring alert (inactive instances) and confirm IRC alerting change works

25 March 2023

  • curprev 07:5407:54, 25 March 2023imported>Stashbot 771,845 bytes +267 hashar@deploy2002: Finished deploy [integration/docroot@ab848e3]: build: Updating eslint-config-wikimedia to 0.24.0 (duration: 00m 08s)
  • curprev 00:5900:59, 25 March 2023imported>Stashbot 771,578 bytes +6,013 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on doc1002.eqiad.wmnet with reason: WIP-known-to-be-debugged-new-host

24 March 2023

  • curprev 00:3800:38, 24 March 2023imported>Stashbot 765,565 bytes +28,346 gmodena@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply

23 March 2023

  • curprev 01:0501:05, 23 March 2023imported>Stashbot 737,219 bytes +29,786 denisse@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "doc1003 - denisse@cumin1001 - T332812"

22 March 2023

  • curprev 00:5700:57, 22 March 2023imported>Stashbot 707,433 bytes +22,655 zabe@deploy2002: Finished scap: update interwiki cache (duration: 07m 02s)

20 March 2023

19 March 2023

  • curprev 18:2718:27, 19 March 2023imported>Stashbot 664,250 bytes +466 AndyRussG: update config (to re-enable old PayPal orphan slayer job) 27a5b481 -> 6359222d
  • curprev 00:1700:17, 19 March 2023imported>Stashbot 663,784 bytes +1,354 fab@deploy2002: Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 05s)

18 March 2023

17 March 2023

  • curprev 01:0501:05, 17 March 2023imported>Stashbot 655,551 bytes +22,974 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2010.codfw.wmnet

15 March 2023

14 March 2023

12 March 2023

  • curprev 10:4710:47, 12 March 2023imported>Stashbot 500,011 bytes +444 elukey: reset offsets on kafka jumbo for benthos webrequest live (as indicated in https://phabricator.wikimedia.org/T331801#8685569)

10 March 2023

  • curprev 22:4322:43, 10 March 2023imported>Stashbot 499,567 bytes +21,702 jhathaway@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • curprev 00:3300:33, 10 March 2023imported>Stashbot 477,865 bytes +75,245 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2002.codfw.wmnet with OS bullseye

9 March 2023

  • curprev 01:1201:12, 9 March 2023imported>Stashbot 402,620 bytes +115,579 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T329260)', diff saved to https://phabricator.wikimedia.org/P45600 and previous config saved to /var/cache/conftool/dbconfig/20230309-011251-marostegui.json

8 March 2023

  • curprev 01:1601:16, 8 March 2023imported>Stashbot 287,041 bytes +135,771 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P45358 and previous config saved to /var/cache/conftool/dbconfig/20230308-011624-marostegui.json

7 March 2023

  • curprev 01:1101:11, 7 March 2023imported>Stashbot 151,270 bytes +84,939 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P45146 and previous config saved to /var/cache/conftool/dbconfig/20230307-011117-marostegui.json

4 March 2023

  • curprev 14:5614:56, 4 March 2023imported>Stashbot 66,331 bytes +1,014 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: Updating member dashboard to reflect new role names -- T330759 (duration: 02m 17s)

3 March 2023

  • curprev 20:5820:58, 3 March 2023imported>Stashbot 65,317 bytes +20,951 inflatador: bking@cumin2002 persistently unban all elastic nodes in eqiad T322082
  • curprev 01:1201:12, 3 March 2023imported>Stashbot 44,366 bytes +15,955 mutante: releases1002: deleting /usr/local/sbin/sync-srv-org-wikimedia-reprepro-releases1002.eqiad.wmnet which confusingly contains an rsync command to rsync from releases1001 which does not exist anymore T330960

2 March 2023

  • curprev 01:0801:08, 2 March 2023imported>Stashbot 28,411 bytes −866,644 mutante: releases2002 - stopping apache2 to test alerting (active server is 1002 but should be switched) T327975 T330960

1 March 2023

  • curprev 00:2500:25, 1 March 2023imported>Stashbot 895,055 bytes +37,440 ejegg: civicrm rolled back from d199694e to ffc16d2d

27 February 2023

  • curprev 23:5423:54, 27 February 2023imported>Stashbot 857,615 bytes +59,717 bking@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

26 February 2023

  • curprev 02:0702:07, 26 February 2023imported>Stashbot 797,898 bytes +147 Amir1: foreachwikiindblist s5 maintenance/migrateExternallinks.php --batch-size=100 --sleep 1 (T326314)

25 February 2023

  • curprev 15:3015:30, 25 February 2023imported>Stashbot 797,751 bytes +618 apergos: resized lvm and filesystem for /data on dumpsdata1004,5,7; was <100G, now is 38T usable (left some room for growth later)

24 February 2023

  • curprev 23:1523:15, 24 February 2023imported>Stashbot 797,133 bytes +27,610 mutante: people2002 - for each user who has a public_html dir that is not empty (for pubdir in $(find . -name public_html -type d -not -empty); ..); rsync it from people1003 with --delete (rsync -avp rsync://people1003.eqiad.wmnet/people-home/${pubdiruser}/public_html/ /home/${pubdiruser}/public_html/); T330091
  • curprev 01:1301:13, 24 February 2023imported>Stashbot 769,523 bytes +31,636 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2017.codfw.wmnet with OS bullseye

23 February 2023

21 February 2023

18 February 2023

  • curprev 08:2908:29, 18 February 2023imported>Stashbot 652,075 bytes +417 elukey: kill leftover processes of user `mepps` (offboarded) from stat100[4,5] to unblock puppet
(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)