You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

3 March 2020

  • curprev 00:5300:53, 3 March 2020imported>Stashbot 409,645 bytes +17,902 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T240055 Update special scandium configuration to load from /srv/parsoid-testing (duration: 00m 58s)

1 March 2020

  • curprev 17:5417:54, 1 March 2020imported>Stashbot 391,743 bytes +926 marostegui: Start replication on db1111 new host on s8 - T246447

29 February 2020

  • curprev 12:3712:37, 29 February 2020imported>Stashbot 390,817 bytes +350 reedy@deploy1001: Synchronized wmf-config/config/viwiki.yaml: T246511 (duration: 00m 56s)

28 February 2020

  • curprev 21:3121:31, 28 February 2020imported>Stashbot 390,467 bytes +6,686 mutante: using planet1001 to manually hack APT sources to test new apt1001.wikimedia.org

27 February 2020

26 February 2020

  • curprev 00:4800:48, 26 February 2020imported>Stashbot 354,983 bytes +15,242 eileen: civicrm revision changed from bec2d6ad9f to 62e62e107c, config revision is c0ef31e2fd

25 February 2020

  • curprev 00:5900:59, 25 February 2020imported>Stashbot 339,741 bytes +14,768 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T238803: Drop ability to load SkinPerPage, EUCopyrightCampaign, and EUCopyrightCampaignSkin (duration: 00m 56s)

23 February 2020

22 February 2020

  • curprev 03:4103:41, 22 February 2020imported>Stashbot 324,738 bytes +596 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • curprev 00:4500:45, 22 February 2020imported>Stashbot 324,142 bytes +6,187 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

20 February 2020

19 February 2020

  • curprev 00:4900:49, 19 February 2020imported>Stashbot 291,837 bytes +16,528 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

17 February 2020

15 February 2020

  • curprev 01:0101:01, 15 February 2020imported>Stashbot 273,144 bytes +176 cdanis: ✔️ cdanis@an-coord1001.eqiad.wmnet ~ 🕗🍺 sudo systemctl restart hive-server2.service ; sudo systemctl restart hive-metastore.service

14 February 2020

  • curprev 23:4223:42, 14 February 2020imported>Stashbot 272,968 bytes +4,182 XenoRyet: updated civicrm from cf86495d44 to 8c77e9e915
  • curprev 00:5300:53, 14 February 2020imported>Stashbot 268,786 bytes +18,037 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T245202 cirrus: Move all move_like traffic to codfw (duration: 01m 02s)

13 February 2020

  • curprev 00:5800:58, 13 February 2020imported>Stashbot 250,749 bytes +16,260 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

12 February 2020

  • curprev 00:4400:44, 12 February 2020imported>Stashbot 234,489 bytes +182,486 jforrester@deploy1001: Synchronized php-1.35.0-wmf.19/includes/page/ImageHistoryPseudoPager.php: T244937 ImageHistoryPseudoPager: Update doQuery() for IndexPager changes (duration: 01m 03s)

11 February 2020

  • curprev 00:2600:26, 11 February 2020imported>Stashbot 52,003 bytes +14,854 ebernhardson@deploy1001: Synchronized php-1.35.0-wmf.18/skins/MinervaNeue: SWAT: Revert: Reduce userContributions icon code (duration: 01m 06s)

9 February 2020

8 February 2020

  • curprev 19:1219:12, 8 February 2020imported>Stashbot 37,073 bytes +318 _joe_: set cpufreq governor to performance on mw1328
  • curprev 00:0500:05, 8 February 2020imported>Stashbot 36,755 bytes +12,166 Jeff_Green: switched payments.wikimedia.org to codfw datacenter due to T244610

7 February 2020

  • curprev 00:1300:13, 7 February 2020imported>Stashbot 24,589 bytes +17,865 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

5 February 2020

  • curprev 23:3023:30, 5 February 2020imported>Stashbot 6,724 bytes −158,385 ebernhardson: delete search indices duplicated on multiple clusters for: hywwiki, chrwiktionary, gcrwiki, mnwwiki, noboard_chapterswikimedia nqowiki nrmwiki outreachwiki and srnwiki

4 February 2020

  • curprev 22:3522:35, 4 February 2020imported>Stashbot 165,109 bytes +4,681 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.18 refs T233866
  • curprev 00:1400:14, 4 February 2020imported>Stashbot 160,428 bytes +9,252 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [enwiki] Add Commons as an import source T242884 (duration: 00m 57s)

2 February 2020

1 February 2020

31 January 2020

  • curprev 01:0901:09, 31 January 2020imported>Stashbot 149,050 bytes +2,034 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@322ee4c]: Update mobileapps to 3eec28d (duration: 06m 53s)

29 January 2020

27 January 2020

  • curprev 23:4023:40, 27 January 2020imported>Stashbot 144,743 bytes +1,366 eileen: civicrm revision changed from fbd5c35fb0 to ac730a6bcb, config revision is 837b9d0703

26 January 2020

24 January 2020

  • curprev 22:3122:31, 24 January 2020imported>Stashbot 141,377 bytes +2,319 mutante: ganeti1003 - sudo gnt-instance remove etherpad1001.eqiad.wmnet (T224580)
  • curprev 00:4600:46, 24 January 2020imported>Stashbot 139,058 bytes +9,927 mutante: cp4032 - starting varnishmtail.service

23 January 2020

  • curprev 01:4001:40, 23 January 2020imported>Stashbot 129,131 bytes +8,387 jforrester@deploy1001: Synchronized php-1.35.0-wmf.16/extensions/AbuseFilter/includes/AFComputedVariable.php: T243469 When no registration date is recorded, use 2008-01-15 (duration: 01m 08s)

22 January 2020

  • curprev 01:1301:13, 22 January 2020imported>Stashbot 120,744 bytes +8,885 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: resync, the last sync only took on half the appservers (duration: 01m 05s)

20 January 2020

19 January 2020

  • curprev 12:0212:02, 19 January 2020imported>Stashbot 106,204 bytes +350 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2085:3311, db2085:3318 T243148', diff saved to https://phabricator.wikimedia.org/P10210 and previous config saved to /var/cache/conftool/dbconfig/20200119-120236-marostegui.json
  • curprev 00:4600:46, 19 January 2020imported>Stashbot 105,854 bytes +429 cdanis: T238305 cp3053.mgmt /admin1-> racadm serveraction hardreset

17 January 2020

  • curprev 21:5621:56, 17 January 2020imported>Stashbot 105,425 bytes +5,582 urandom: bootstrapping restbase2023-c — T243000
  • curprev 00:4500:45, 17 January 2020imported>Stashbot 99,843 bytes +13,373 mutante: urldownloaders - rm /etc/logrotate.d/squid3 ; systemctl start logrotate (this fixes failed logrotate because of squid3 vs squid file = duplicate entry, but puppet will recreate it)

16 January 2020

  • curprev 00:4000:40, 16 January 2020imported>Stashbot 86,470 bytes +11,747 bstorm_: set max_connections on db1133 (m5-master) back to 500 since the neutron connections seem fairly stable now T242817

15 January 2020

  • curprev 00:2200:22, 15 January 2020imported>Stashbot 74,723 bytes +6,560 bstorm_: restarted maintain-dbusers on labstore1004 after recovering the m5 DB's connection issue

14 January 2020

  • curprev 01:0301:03, 14 January 2020imported>Stashbot 68,163 bytes +7,915 catrope@deploy1001: Synchronized php-1.35.0-wmf.14/extensions/GrowthExperiments/: Various topic search-related cherry-picks (duration: 00m 57s)

13 January 2020

  • curprev 00:5800:58, 13 January 2020imported>Stashbot 60,248 bytes +1,047 jiji@cumin1001: conftool action : set/pooled=yes; selector: name=cp3061.esams.wmnet

11 January 2020

  • curprev 05:3405:34, 11 January 2020imported>Stashbot 59,201 bytes +242 volker-e@deploy1001: Finished deploy [design/style-guide@6a44c69]: Deploy design/style-guide: (duration: 00m 08s)

10 January 2020

  • curprev 22:3322:33, 10 January 2020imported>Stashbot 58,959 bytes +3,964 mutante: ms-be1026 sudo systemctl reset-failed (failed Session 372989 of user debmonitor)
  • curprev 00:4500:45, 10 January 2020imported>Stashbot 54,995 bytes +8,040 catrope@deploy1001: Synchronized php-1.35.0-wmf.14/extensions/GrowthExperiments/: Expose tasktype/topic API parameter info (T240512) (duration: 01m 01s)

9 January 2020

  • curprev 00:3300:33, 9 January 2020imported>Stashbot 46,955 bytes +13,245 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no-op) set config page for newcomer tasks (T233465) (duration: 01m 05s)

8 January 2020

  • curprev 00:3800:38, 8 January 2020imported>Stashbot 33,710 bytes +15,858 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@024488f]: airflow: set mjolnir dag start date to today (20200108) (duration: 00m 42s)

7 January 2020

  • curprev 00:5900:59, 7 January 2020imported>Stashbot 17,852 bytes +9,482 mutante: ganeti - creating urldownloader1002.wikimedia.org in eqiad_C with 1 CPU, 1 GB RAM, 10 GB disk, public IP (T241979)

6 January 2020

4 January 2020

  • curprev 16:3416:34, 4 January 2020imported>Stashbot 7,863 bytes +313 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

3 January 2020

  • curprev 22:1422:14, 3 January 2020imported>Stashbot 7,550 bytes +3,165 volker-e@deploy1001: Finished deploy [design/style-guide@8054026]: Deploy design/style-guide: (duration: 00m 08s)

2 January 2020

1 January 2020

30 December 2019

  • curprev 16:0016:00, 30 December 2019imported>Stashbot 868,829 bytes +235 ema: cp1083: restart ats-tls and varnish-fe after crashes - T241593

29 December 2019

28 December 2019

  • curprev 22:5422:54, 28 December 2019imported>Stashbot 868,069 bytes +602 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: cirrus: re-sync move more_like traffic to codfw (duration: 00m 54s)

27 December 2019

25 December 2019

  • curprev 10:1210:12, 25 December 2019imported>Stashbot 866,771 bytes +312 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

24 December 2019

  • curprev 15:3315:33, 24 December 2019imported>Stashbot 866,459 bytes +1,211 gehel@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 00:1300:13, 24 December 2019imported>Stashbot 865,248 bytes +2,405 volker-e@deploy1001: Finished deploy [design/style-guide@efc240b]: Deploy design/style-guide: (duration: 00m 07s)

22 December 2019

  • curprev 16:4816:48, 22 December 2019imported>Stashbot 862,843 bytes +468 bd808: Mastodon feed moved to https://mastodon.social/@wikimedia_sal (T52109)
  • curprev 00:1000:10, 22 December 2019imported>Stashbot 862,375 bytes +612 bd808: Wikimedia SAL messages now available on Mastodon! https://fosstodon.org/@wikimedia_sal

21 December 2019

  • curprev 01:4401:44, 21 December 2019imported>Stashbot 861,763 bytes +8,163 reedy@deploy1001: update-interwiki-cache aborted: Update interwiki cache (duration: 00m 07s)

20 December 2019

  • curprev 00:5200:52, 20 December 2019imported>Stashbot 853,600 bytes +18,576 mholloway-shell@deploy1001: Synchronized php-1.35.0-wmf.11/extensions/MachineVision: Fix: Allow a single period in $basePath in maintenance scripts (duration: 00m 54s)

19 December 2019

  • curprev 01:4401:44, 19 December 2019imported>Stashbot 835,024 bytes +20,843 twentyafterfour: deploying phabricator update (tagged release/2019-12-19/1)

18 December 2019

  • curprev 00:2200:22, 18 December 2019imported>Stashbot 814,181 bytes +17,525 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Glent M0 for dewiki, enwiki and frwiki - T237365 (duration: 01m 02s)

16 December 2019

  • curprev 23:2823:28, 16 December 2019imported>Stashbot 796,656 bytes +15,656 ejegg: updated fundraising civicrm from 505c653da1 to b2d0b5d66d

14 December 2019

13 December 2019

  • curprev 22:3522:35, 13 December 2019imported>Stashbot 780,908 bytes +3,189 cdanis: T229686 ✔️ cdanis@mwdebug1001.eqiad.wmnet /srv/mediawiki 🕠🍺 scap pull
  • curprev 00:3200:32, 13 December 2019imported>Stashbot 777,719 bytes +8,893 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Begin "initiation test" for suggested edits (T238888) (duration: 00m 55s)

12 December 2019

  • curprev 01:4201:42, 12 December 2019imported>Stashbot 768,826 bytes +11,852 volker-e@deploy1001: Finished deploy [design/style-guide@481eaf6]: Deploy design/style-guide: (duration: 00m 07s)

11 December 2019

9 December 2019

8 December 2019

  • curprev 20:4020:40, 8 December 2019imported>Stashbot 741,665 bytes +605 ejegg: disabled Ingenico Connect recurring charge job
  • curprev 01:4701:47, 8 December 2019imported>Stashbot 741,060 bytes +2,785 andrew@deploy1001: Started deploy [horizon/deploy@accbbd1]: (no justification provided)

7 December 2019

  • curprev 00:5900:59, 7 December 2019imported>Stashbot 738,275 bytes +8,615 andrew@deploy1001: Started deploy [horizon/deploy@0f70602]: (no justification provided)

6 December 2019

  • curprev 00:4700:47, 6 December 2019imported>Stashbot 729,660 bytes +15,748 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Use PHP serialization with HMAC for Kask session serialization (T222099) (duration: 01m 01s)

5 December 2019

  • curprev 00:4900:49, 5 December 2019imported>Stashbot 713,912 bytes +15,602 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=phab1001-vcs.eqiad.wmnet

4 December 2019

  • curprev 00:4900:49, 4 December 2019imported>Stashbot 698,310 bytes +15,609 ejegg: changed donations queue consumer and thank you mailer to use 3 minute cycles

3 December 2019

  • curprev 01:4201:42, 3 December 2019imported>Stashbot 682,701 bytes +128,943 dzahn@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
  • curprev 01:0001:00, 3 December 2019imported>Stashbot 681,404 bytes +127,646 James_F: mwscript sql.php --wiki=sewikimedia php-1.35.0-wmf.8/extensions/Translate/sql/translate_{…}.sql T239091

19 November 2019

18 November 2019

  • curprev 00:4400:44, 18 November 2019imported>Stashbot 545,042 bytes +199 tstarling@deploy1001: Synchronized php-1.35.0-wmf.5/includes/Rest/Handler/PageHistoryCountHandler.php: fix extremely slow query T238378 (duration: 00m 59s)

16 November 2019

  • curprev 20:2720:27, 16 November 2019imported>Stashbot 544,843 bytes +277 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

15 November 2019

  • curprev 22:1422:14, 15 November 2019imported>Stashbot 544,566 bytes +7,139 jeh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

14 November 2019

  • curprev 23:0323:03, 14 November 2019imported>Stashbot 537,427 bytes +12,247 mutante: restarting gerrit to ncrease defaultThreadPoolSize to 2
  • curprev 00:4100:41, 14 November 2019imported>Stashbot 525,180 bytes +9,904 ebernhardson: T237849 Start CirrusSearch forceSearchIndex.php commonswiki 2019-10-20T00:00:00 - 2019-11-14T01:00:00 pushing into jobqueue

12 November 2019

  • curprev 23:5723:57, 12 November 2019imported>Stashbot 515,276 bytes +11,275 mholloway-shell@deploy1001: Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Fix: Do not return after inserting a single suggestion (duration: 00m 52s)

11 November 2019

  • curprev 22:5122:51, 11 November 2019imported>Stashbot 504,001 bytes +3,806 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3065.esams.wmnet

9 November 2019

  • curprev 20:2520:25, 9 November 2019imported>Stashbot 500,195 bytes +323 reedy@deploy1001: Synchronized langlist-labs: T237823 (duration: 00m 54s)
  • curprev 01:0701:07, 9 November 2019imported>Stashbot 499,872 bytes +7,974 volker-e@deploy1001: Finished deploy [design/style-guide@ef82b69]: Deploy design/style-guide: (duration: 00m 07s)

8 November 2019

  • curprev 01:2301:23, 8 November 2019imported>Stashbot 491,898 bytes +10,234 reedy@deploy1001: Synchronized php-1.35.0-wmf.5/extensions/GlobalBlocking/: Prevent some extra db queries (duration: 00m 53s)

7 November 2019

  • curprev 00:2100:21, 7 November 2019imported>Stashbot 481,664 bytes +9,074 XioNoX: enable interface damping on primary eqsin-codfw link - T236878

6 November 2019

  • curprev 00:0700:07, 6 November 2019imported>Stashbot 472,590 bytes +5,234 mdholloway: created table wikimedia_editor_tasks_edit_streak on x1/wikishared (T234956)

5 November 2019

  • curprev 00:1500:15, 5 November 2019imported>Stashbot 467,356 bytes +6,242 mutante: gerrit - restarting service to re-enable jgit gc (T217497)

3 November 2019

  • curprev 03:5403:54, 3 November 2019imported>Stashbot 461,114 bytes +719 andrew@deploy1001: Finished deploy [horizon/deploy@0c024d4]: one more prefix fix (duration: 03m 35s)

2 November 2019

  • curprev 00:5800:58, 2 November 2019imported>Stashbot 460,395 bytes +3,957 mutante: gerrit-replica - created missing /var/lib/gerrit2/review_site/tmp and restarted service - service back up on buster (T176774)

1 November 2019

  • curprev 01:2101:21, 1 November 2019imported>Stashbot 456,438 bytes +10,859 jforrester@deploy1001: Synchronized php-1.35.0-wmf.4/resources/src/mediawiki.widgets/mw.widgets.UsersMultiselectWidget.js: T236460 mw.widgets.UsersMultiselectWidget: Fix property name (duration: 00m 54s)

31 October 2019

  • curprev 00:4100:41, 31 October 2019imported>Stashbot 445,579 bytes +20,560 jforrester@deploy1001: Synchronized php-1.35.0-wmf.4/extensions/WikiLove/resources/ext.wikiLove.icon.vector.css: T236958 Fix Vector icon after upstream change (duration: 01m 02s)

28 October 2019

  • curprev 23:2323:23, 28 October 2019imported>Stashbot 425,019 bytes +8,544 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy Echo kask migration to officewiki for testing, part 3 (T222851) (duration: 00m 52s)

26 October 2019

25 October 2019

  • curprev 22:5522:55, 25 October 2019imported>Stashbot 416,347 bytes +9,927 mutante: moscovium rm /dev/shm/envoy_shared_memory_0 to revive envoy which failed to run after changing ports and reinstalling it (T180641)
  • curprev 00:5200:52, 25 October 2019imported>Stashbot 406,420 bytes +18,865 krinkle@deploy1001: Synchronized php-1.35.0-wmf.3/extensions/LiquidThreads/classes/View.php: (no justification provided) (duration: 00m 54s)

24 October 2019

  • curprev 00:0300:03, 24 October 2019imported>Stashbot 387,555 bytes +11,289 mutante: restarting gerrit to increase heap_size from 20G to 32G (T225166 T222391)

22 October 2019

  • curprev 23:5723:57, 22 October 2019imported>Stashbot 376,266 bytes +12,691 maxsem@deploy1001: Synchronized php-1.35.0-wmf.3/includes/block/DatabaseBlock.php: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/545373/ (duration: 00m 59s)

21 October 2019

  • curprev 23:1523:15, 21 October 2019imported>Stashbot 363,575 bytes +15,479 thcipriani: ops/puppet:sudo -u gerrit2 git update-ref refs/changes/66/535966/meta d6909e0537b7fe7ca6fb8989e22ed5e1c1e46e89 && sudo -u gerrit2 git update-ref refs/changes/66/535966/meta 8494c28eee163aaa48f6274e899e0e79f3a757f3 on gerrit1001

19 October 2019

  • curprev 08:4108:41, 19 October 2019imported>Stashbot 348,096 bytes +48 XioNoX: add user papaul to fasw-c-eqiad
  • curprev 00:0500:05, 19 October 2019imported>Stashbot 348,048 bytes +12,443 mutante: LDAP - adding verenali to wmde and nda groups, to match raja_wmde (T233807, T231677)

17 October 2019

  • curprev 21:4221:42, 17 October 2019imported>Stashbot 335,605 bytes +9,045 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@d663006]: Update mobileapps to f345673 (duration: 05m 38s)
  • curprev 00:5500:55, 17 October 2019imported>Stashbot 326,560 bytes +13,865 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

16 October 2019

  • curprev 00:0900:09, 16 October 2019imported>Stashbot 312,695 bytes +11,193 mutante: wikitech - make JBond a "content administrator" to give the ability to create server fingerprint pages

14 October 2019

  • curprev 23:2723:27, 14 October 2019imported>Stashbot 301,502 bytes +9,644 Krinkle: Delete 2019-09-01––2019-09-10 arclamp trace logs from webperf1002, and decompress the rest of 2019-09 (this will trigger svg re-generation), T235425

13 October 2019

  • curprev 00:5200:52, 13 October 2019imported>Stashbot 291,858 bytes +984 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ec77b1b515940c73 (duration: 00m 55s)

11 October 2019

10 October 2019

  • curprev 22:2722:27, 10 October 2019imported>Stashbot 287,642 bytes +12,182 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting wgMFMobileFormatterHeadings, unread T232690 (duration: 00m 51s)
  • curprev 00:3900:39, 10 October 2019imported>Stashbot 275,460 bytes +16,804 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)

8 October 2019

  • curprev 23:2823:28, 8 October 2019imported>Stashbot 258,656 bytes +11,791 mutante: phab1001 - replacing tin.eqiad.wmnet with deploy1001.eqiad.wmnet in phabricator/deployment-cache/.config:git_server - wondering if we can ever get rid of tin (T190568)
  • curprev 00:3300:33, 8 October 2019imported>Stashbot 246,865 bytes +11,958 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)

6 October 2019

  • curprev 20:1120:11, 6 October 2019imported>Stashbot 234,907 bytes +365 Urbanecm: mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Racconish /home/urbanecm/T234741 (T234741)

5 October 2019

  • curprev 06:4806:48, 5 October 2019imported>Stashbot 234,542 bytes +151 elukey: force umount/remount of /mnt/hdfs on an-coord1001 - processes stuck in D state, fuser proc consuming a ton of memory

4 October 2019

3 October 2019

  • curprev 23:5023:50, 3 October 2019imported>Stashbot 231,365 bytes +5,107 mutante: gerrit - restarting for replication config tweaks
  • curprev 00:5100:51, 3 October 2019imported>Stashbot 226,258 bytes +6,323 mutante: gerrit1001 - removing wrong IPv6 address from interface, running puppet

2 October 2019

  • curprev 00:5400:54, 2 October 2019imported>Stashbot 219,935 bytes +8,918 ejegg: updated fundraising CiviCRM from 6d90d0cf06 to 12c5727a23

1 October 2019

  • curprev 00:1200:12, 1 October 2019imported>Stashbot 211,017 bytes +7,795 mutante: phabricator - upgrading PHP version to 7.2.22 - T230024

28 September 2019

27 September 2019

  • curprev 22:4422:44, 27 September 2019imported>Stashbot 203,145 bytes +3,427 mutante: phab2001 - apt-get autoremove - remove unused python and ruby packages
  • curprev 00:5300:53, 27 September 2019imported>Stashbot 199,718 bytes +16,143 twentyafterfour: hotfixing phabricator fatal exception refs T233998

25 September 2019

  • curprev 21:5921:59, 25 September 2019imported>Stashbot 183,575 bytes +17,154 bblack: remove GRE MTU hacks on archiva1001 gerrit2001 cobalt install1002 - T232602
  • curprev 01:0001:00, 25 September 2019imported>Stashbot 166,421 bytes +13,033 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 3373247e123b53 - create new file (duration: 01m 05s)

24 September 2019

  • curprev 00:4300:43, 24 September 2019imported>Stashbot 153,388 bytes +8,330 mutante: db2060 - remove PXE flag boot override - set Boot Device to none

22 September 2019

21 September 2019

19 September 2019

  • curprev 23:2323:23, 19 September 2019imported>Stashbot 138,735 bytes +9,963 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

18 September 2019

  • curprev 23:1823:18, 18 September 2019imported>Stashbot 128,772 bytes +12,473 krinkle@deploy1001: Synchronized php-1.34.0-wmf.23/extensions/MobileFrontend/resources/dist/: T233260, 1667ed957a19067 (duration: 01m 04s)

17 September 2019

  • curprev 23:2623:26, 17 September 2019imported>Stashbot 116,299 bytes +14,935 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.23 refs T220748
  • curprev 00:3400:34, 17 September 2019imported>Stashbot 101,364 bytes +15,480 AndyRussG: updated fruec from fb29cb7407 to 97128874bf

15 September 2019

14 September 2019

  • curprev 23:4223:42, 14 September 2019imported>Stashbot 85,524 bytes +133 onimisionipe: force shard allocation (dewiki_content_1566659363[4]) on eqiad cluster
  • curprev 01:1401:14, 14 September 2019imported>Stashbot 85,391 bytes +7,579 ejegg: updated fundraising python tools from 1e405864d7 to e1b81688c6

12 September 2019

11 September 2019

10 September 2019

  • curprev 20:4620:46, 10 September 2019imported>Stashbot 53,851 bytes +9,981 ejegg: updated payments-wiki from 15baf7f58b to 5432f9c3a4
  • curprev 00:5900:59, 10 September 2019imported>Stashbot 43,870 bytes +5,338 krinkle@deploy1001: Finished deploy [performance/navtiming@f2a0863]: (no justification provided) (duration: 00m 05s)

8 September 2019

  • curprev 22:0422:04, 8 September 2019imported>Stashbot 38,532 bytes +153 eileen: civicrm revision changed from c3172daf95 to d1d65f37ea, config revision is 5a6a9c6c03

6 September 2019

5 September 2019

  • curprev 23:1323:13, 5 September 2019imported>Stashbot 33,990 bytes +8,363 ayounsi@deploy1001: Finished deploy [netbox/deploy@367ca84]: test (duration: 00m 42s)
  • curprev 00:5500:55, 5 September 2019imported>Stashbot 25,627 bytes +10,215 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 53s)

4 September 2019

  • curprev 01:0201:02, 4 September 2019imported>Stashbot 15,412 bytes +11,302 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s)

2 September 2019

1 September 2019

31 August 2019

  • curprev 13:3313:33, 31 August 2019imported>Stashbot 747,212 bytes +134 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: 88ba4f8f4d49 (duration: 00m 55s)

30 August 2019

29 August 2019

  • curprev 23:2323:23, 29 August 2019imported>Stashbot 742,673 bytes +4,768 ejegg: updated payments-wiki from 1d5d7503b0 to 51d9ed79b6

28 August 2019

  • curprev 23:1723:17, 28 August 2019imported>Stashbot 737,905 bytes +7,538 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 8bfe43a: Add scielo.br to wgCopyUploadsDomains for commonswiki (T231402) (duration: 00m 55s)

27 August 2019

25 August 2019

  • curprev 13:4613:46, 25 August 2019imported>Stashbot 714,674 bytes +176 volans: uploaded spicerack_0.0.27-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • curprev 00:3500:35, 25 August 2019imported>Stashbot 714,498 bytes +283 herron: set icinga downtimes on flapping cr2-eqiad and cr2-codfw alerts until monday

23 August 2019

  • curprev 23:3423:34, 23 August 2019imported>Stashbot 714,215 bytes +1,363 robh@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)

22 August 2019

  • curprev 23:4123:41, 22 August 2019imported>Stashbot 712,852 bytes +5,069 urbanecm@deploy1001: Synchronized php-1.34.0-wmf.19/extensions/WikibaseLexeme: SWAT: e4a5457: Fix Lexemes RDF generation (T230974) (duration: 00m 49s)
  • curprev 00:3700:37, 22 August 2019imported>Stashbot 707,783 bytes −393,674 XioNoX: run /usr/local/sbin/restart-php7.2-fpm on mwdebug1001/2

20 August 2019

  • curprev 23:5323:53, 20 August 2019imported>Stashbot 1,101,457 bytes +7,677 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@a7bf6cf]: bulk_daemon: Increase bulk request_timeout (duration: 03m 40s)

19 August 2019

  • curprev 21:2121:21, 19 August 2019imported>Stashbot 1,093,780 bytes +5,899 ejegg: updated payments-wiki from 7b8091ba87 to 85dce8f79f

18 August 2019

  • curprev 08:2808:28, 18 August 2019imported>Stashbot 1,087,881 bytes +165 onimisionipe: running `_cluster/reroute?pretty&explain=true&retry_failed` on eqiad production-search cluster to force allocation of shards

16 August 2019

  • curprev 19:4819:48, 16 August 2019imported>Stashbot 1,087,716 bytes +1,009 sbassett: Deployed security patch for T230576 (ex:MobileFrontend)

15 August 2019

  • curprev 23:3523:35, 15 August 2019imported>Stashbot 1,086,707 bytes +7,511 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@b4da6e4]: Rollback blazegraph due to T230588 (duration: 09m 48s)
  • curprev 00:1500:15, 15 August 2019imported>Stashbot 1,079,196 bytes +5,064 robh: scs-ulsfo offline due to networking issues, rob returning tomorrow with fix T230077

13 August 2019

  • curprev 20:4320:43, 13 August 2019imported>Stashbot 1,074,132 bytes +11,563 ejegg: rolled back payments-wiki from 9ed8be0532 to 9533f70fab

12 August 2019

11 August 2019

  • curprev 22:3822:38, 11 August 2019imported>Stashbot 1,056,982 bytes +198 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Temporary make account creation limits more restrictive (T230304) (duration: 00m 50s)

10 August 2019

  • curprev 01:4901:49, 10 August 2019imported>Stashbot 1,056,784 bytes +187 mutante: mwmaint - running (1 of 8, the one for en) refreshLinks maintenance cron manually to verify it works after switching mwscriptwikiset to PHP7.2 (T195392)
  • curprev 00:5200:52, 10 August 2019imported>Stashbot 1,056,597 bytes +3,069 mutante: mwmaint - running update_flaggedrevs_stats - updates the flagged revs statistics table on each wiki

8 August 2019

7 August 2019

  • curprev 00:4800:48, 7 August 2019imported>Stashbot 1,039,482 bytes +10,857 mutante: restarting gerrit to apply config change 528276 to exclude some projects from github replication

5 August 2019

  • curprev 23:3423:34, 5 August 2019imported>Stashbot 1,028,625 bytes +13,466 mutante: mwmaint1002 - remove getJobQueueLengths.php from www-data's crontab (T195392)

4 August 2019

  • curprev 18:4518:45, 4 August 2019imported>Stashbot 1,015,159 bytes +150 krinkle@deploy1001: Synchronized wmf-config/abusefilter.php: labs-only noop - f740f89c594979 (duration: 00m 50s)

3 August 2019

  • curprev 12:0212:02, 3 August 2019imported>Stashbot 1,015,009 bytes +321 gilles: purging ruwiki articles on mwmaint1002
  • curprev 00:4000:40, 3 August 2019imported>Stashbot 1,014,688 bytes +5,626 ejegg: rolled back fundraising python tools from 493a38f9e0 to 2a56e5e283

1 August 2019

  • curprev 23:3223:32, 1 August 2019imported>Stashbot 1,009,062 bytes +8,682 Urbanecm: Evening SWAT done
  • curprev 00:3300:33, 1 August 2019imported>Stashbot 1,000,380 bytes +12,622 krinkle@deploy1001: Synchronized php-1.34.0-wmf.16/resources/Resources.php: acfff6751f3b8f7650 (duration: 00m 54s)

31 July 2019

  • curprev 00:2400:24, 31 July 2019imported>Stashbot 987,758 bytes +10,243 eileen: tools revision changed from 4910f1507c to 2a56e5e283

29 July 2019

  • curprev 23:3723:37, 29 July 2019imported>Stashbot 977,515 bytes +9,325 XioNoX: replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in ams

28 July 2019

  • curprev 15:1315:13, 28 July 2019imported>Stashbot 968,190 bytes +99 arturo: disable 1m load average check in icinga for labstore1007 for 24h

27 July 2019

  • curprev 17:3917:39, 27 July 2019imported>Stashbot 968,091 bytes +223 bd808: Updated profile & images for @wikimediatech twitter account
  • curprev 00:3500:35, 27 July 2019imported>Stashbot 967,868 bytes +4,085 mutante: restbase-dev1006 - starting nagios-nrpe-server

26 July 2019

  • curprev 00:5300:53, 26 July 2019imported>Stashbot 963,783 bytes +13,853 ejegg: re-enabled dedupe_civicrm_contacts and major_gifts_addresses fundraising jobs

24 July 2019

  • curprev 23:4623:46, 24 July 2019imported>Stashbot 949,930 bytes +10,601 nuria@deploy1001: Finished deploy [analytics/refinery@7d93398]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues). Try 2 (duration: 13m 34s)

23 July 2019

  • curprev 23:4423:44, 23 July 2019imported>Stashbot 939,329 bytes +9,022 eileen: civicrm revision changed from 88e9f24893 to 875ab97742, config revision is 4006d3bdc5
  • curprev 00:0600:06, 23 July 2019imported>Stashbot 930,307 bytes +11,599 Urbanecm: slwiki updateCollection.php completed (T208984)

21 July 2019

19 July 2019

  • curprev 22:3622:36, 19 July 2019imported>Stashbot 918,628 bytes +5,701 mutante: phab2001 - switching apache to php-fpm and worker instead of mpm-prefork (to match phab1001) (T190568 T137928 T190572)
  • curprev 00:5200:52, 19 July 2019imported>Stashbot 912,927 bytes +9,770 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Add even more severe rate limits for eswikiquote and some other, smaller wikis (T227416) (duration: 00m 58s)

17 July 2019

  • curprev 23:5123:51, 17 July 2019imported>Stashbot 903,157 bytes +11,329 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Add wmgUseTheWikipediaLibrary (false everywhere, no-op) (duration: 00m 54s)
  • curprev 00:5000:50, 17 July 2019imported>Stashbot 891,828 bytes +10,253 mutante: wikitech-static commented out cert renewal cron job out of caution - still needs fixing but continue tomorrow

16 July 2019

  • curprev 00:2400:24, 16 July 2019imported>Stashbot 881,575 bytes +12,943 krinkle@deploy1001: Synchronized php-1.34.0-wmf.13/includes/cache/LinkCache.php: 4a5f4ca2fd788 (duration: 00m 51s)

14 July 2019

  • curprev 13:1813:18, 14 July 2019imported>Stashbot 868,632 bytes +257 godog: silence mr1-eqsin.oob IPv6 until tomorrow 8 UTC - T227967

13 July 2019

12 July 2019

  • curprev 23:3523:35, 12 July 2019imported>Stashbot 868,307 bytes +3,936 mutante: netmon1003 - shutdown -h now after it's gone from Icinga now
  • curprev 01:0101:01, 12 July 2019imported>Stashbot 864,371 bytes +11,144 Krinkle: mw1342 generated some ~ 11,500 additional PHP errors over a 4 hour period (18:00-22:30 UTC), ref T224491

10 July 2019

  • curprev 23:1623:16, 10 July 2019imported>Stashbot 853,227 bytes +7,815 jforrester@deploy1001: Synchronized php-1.34.0-wmf.13/extensions/CirrusSearch/includes: T227691 RedirectsAndIncomingLinks: succeede or fail, but not both (duration: 01m 02s)

9 July 2019

  • curprev 23:0723:07, 9 July 2019imported>Stashbot 845,412 bytes +8,973 jhuneidi@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.13 refs T220738
  • curprev 01:4701:47, 9 July 2019imported>Stashbot 836,439 bytes +12,820 XioNoX: restart PHP FPM on mwdebug2001

7 July 2019

  • curprev 20:1320:13, 7 July 2019imported>Stashbot 823,619 bytes +210 urandom: decommissioning restbase1017-b -- T222960

6 July 2019

  • curprev 07:5607:56, 6 July 2019imported>Stashbot 823,409 bytes +74 thcipriani: restarting gerrit out of heap space

5 July 2019

  • curprev 17:1817:18, 5 July 2019imported>Stashbot 823,335 bytes +5,803 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@4b7cdf5]: Deploy new versions preparing for reload (duration: 00m 39s)

4 July 2019

  • curprev 21:5021:50, 4 July 2019imported>Stashbot 817,532 bytes +6,053 volans@deploy1001: Finished deploy [debmonitor/deploy@0ee26a3]: Deploy Debmonitor v0.1.10 (duration: 00m 48s)
  • curprev 00:4100:41, 4 July 2019imported>Stashbot 811,479 bytes +18,775 twentyafterfour: phabricator upgrade complete

3 July 2019

  • curprev 00:1400:14, 3 July 2019imported>Stashbot 792,704 bytes +11,829 eileen: process-control config revision is 8e215d07f2 (renable jobs)

1 July 2019

  • curprev 20:3320:33, 1 July 2019imported>Stashbot 780,875 bytes +7,789 milimetric@deploy1001: Finished deploy [analytics/refinery@4e9894c]: minor, just removing hiwikisource from sqoop list (duration: 01m 33s)

30 June 2019

  • curprev 23:2723:27, 30 June 2019imported>Stashbot 773,086 bytes +199 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 45s)

28 June 2019

  • curprev 21:1921:19, 28 June 2019imported>Stashbot 772,887 bytes +6,285 otto@deploy1001: Finished deploy [eventstreams/deploy@2af2719]: Manually blacklisting IP - T226808 (duration: 03m 07s)
  • curprev 01:2201:22, 28 June 2019imported>Stashbot 766,602 bytes +7,465 Krinkle: Killing arclamp-log on webperf1002, no flame graphs for three days, presumably mwlog/redis connection dropped again. T215740

27 June 2019

25 June 2019

  • curprev 22:5022:50, 25 June 2019imported>Stashbot 747,998 bytes +8,261 jforrester@deploy1001: Synchronized php-1.34.0-wmf.11/extensions/Echo/modules/nojs/: T226503 Fix badge icons in Monobook (duration: 00m 56s)
  • curprev 00:1000:10, 25 June 2019imported>Stashbot 739,737 bytes +11,072 reedy@deploy1001: Synchronized wmf-config/flaggedrevs.php: poke at autopromote config (duration: 00m 54s)

21 June 2019

  • curprev 14:5414:54, 21 June 2019imported>Stashbot 728,665 bytes +5,320 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 00:4500:45, 21 June 2019imported>Stashbot 723,345 bytes +15,806 RoanKattouw: Running FlowReserializeRevisionContent.php on testwiki

19 June 2019

  • curprev 18:1018:10, 19 June 2019imported>Stashbot 707,539 bytes +18,219 legoktm: added MatmaRex to extension-VisualEditor-staff Gerrit group

18 June 2019

  • curprev 22:3322:33, 18 June 2019imported>Stashbot 689,320 bytes +17,179 jijiki: pool thumbor1001
  • curprev 00:2200:22, 18 June 2019imported>Stashbot 672,141 bytes +12,375 RoanKattouw: Running populateRevisionSha1.php on dewikivoyage for T219816

16 June 2019

  • curprev 14:2014:20, 16 June 2019imported>Stashbot 659,766 bytes +253 Urbanecm: running mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='AKA MBG' /home/urbanecm/T225886

15 June 2019

  • curprev 20:3820:38, 15 June 2019imported>Stashbot 659,513 bytes +618 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@55174a4]: deploy new pattern for bots (duration: 21m 42s)

14 June 2019

  • curprev 23:2323:23, 14 June 2019imported>Stashbot 658,895 bytes +1,595 ejegg: updated payments-wiki from 75abd71cc1 to 79d1822644
  • curprev 00:1100:11, 14 June 2019imported>Stashbot 657,300 bytes +1,216 SMalyshev: depooled wdqs1005 - let it catch up

13 June 2019

  • curprev 00:4500:45, 13 June 2019imported>Stashbot 656,084 bytes +2,359 paravoid: setting the CPU governor to performance for ms-be1036 (a while ago)

11 June 2019

  • curprev 19:2419:24, 11 June 2019imported>Stashbot 653,725 bytes +610 tzatziki: Removing four (4) files for legal compliance

10 June 2019

9 June 2019

  • curprev 08:3008:30, 9 June 2019imported>Stashbot 650,997 bytes +79 vgutierrez: rebooting lvs4007 after NIC driver crash

8 June 2019

  • curprev 11:5811:58, 8 June 2019imported>Stashbot 650,918 bytes +203 godog: stop swift processes on ms-be1033 - T223518

7 June 2019

  • curprev 18:5618:56, 7 June 2019imported>Stashbot 650,715 bytes +6,617 herron: performing rolling reboots of logstash codfw frontends for security updates
  • curprev 00:0000:00, 7 June 2019imported>Stashbot 644,098 bytes +674 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove unused preference T47877-buster (duration: 00m 47s)

6 June 2019

5 June 2019

  • curprev 22:1522:15, 5 June 2019imported>Stashbot 635,049 bytes +20,643 chaomodus: restarting gerrit on cobalt due to it being down (seems like Java out of heap space)

4 June 2019

  • curprev 00:0900:09, 4 June 2019imported>Stashbot 614,406 bytes +7,967 bstorm_: T223406 repooled labsdb1011 after completing view updates

2 June 2019

  • curprev 20:2820:28, 2 June 2019imported>Stashbot 606,439 bytes +347 onimisionipe: pooled wdqs1007. It caught up on lag

1 June 2019

  • curprev 22:4922:49, 1 June 2019imported>Stashbot 606,092 bytes +604,365 krinkle@deploy1001: Synchronized php-1.34.0-wmf.7/extensions/3D/modules/mmv.3d.js: T224812 / bd4fbfddbe1a0 (duration: 01m 07s)

3 February 2019

  • curprev 20:2520:25, 3 February 2019imported>Stashbot 1,727 bytes +610 elukey: powercycle mw1272 - no ssh, no tty available via com2 - DIMM correctable errors + OEM errors registered in getsel
  • curprev 01:1001:10, 3 February 2019imported>Stashbot 1,117 bytes −57,244 elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred."

13 January 2019

  • curprev 16:3316:33, 13 January 2019imported>Stashbot 58,361 bytes +103 hoo: Updated operations/dumps/dcat (559dee37452..a86285f4e7) on snapshot1008

12 January 2019

11 January 2019

  • curprev 21:5621:56, 11 January 2019imported>Stashbot 57,967 bytes +2,518 jforrester@deploy1001: Finished scap: Full scap sync to update wmf.12 i18n for the weekend Idf2a67860f (duration: 19m 12s)
  • curprev 00:5000:50, 11 January 2019imported>Stashbot 55,449 bytes +11,158 XioNoX: bump prefix limit for AS6939 in eqsin

10 January 2019

  • curprev 01:4101:41, 10 January 2019imported>Stashbot 44,291 bytes +8,082 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make GrowthExperiments config wmf.12-proof (duration: 00m 52s)

9 January 2019

  • curprev 01:1501:15, 9 January 2019imported>Stashbot 36,209 bytes +13,531 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: T213227 Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s)

8 January 2019

6 January 2019

  • curprev 14:5014:50, 6 January 2019imported>Stashbot 15,081 bytes +251 ariel@deploy1001: Finished deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag (duration: 00m 06s)

5 January 2019

4 January 2019

  • curprev 23:0723:07, 4 January 2019imported>Stashbot 14,687 bytes +1,714 mutante: scandium apt-get remove nodejs nodes-legacy ; puppet agent -tv - after merging gerrit:482150 this fixed "you have held broken packages" issue, now we are at a puppet dependecy cycle with apt::pin T201366
  • curprev 00:0700:07, 4 January 2019imported>Stashbot 12,973 bytes +6,055 mutante: an-coord1001 - apt-get clean to free disk space, reacting to Icinga alert for running out of disk

3 January 2019

  • curprev 00:2100:21, 3 January 2019imported>Stashbot 6,918 bytes +6,798 mutante: notebook1004 - started nagios-nrpe-server one more time

1 January 2019

30 December 2018

29 December 2018

  • curprev 14:5314:53, 29 December 2018imported>Stashbot 133,496 bytes +935 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T212667 218371fd35 - Adjust mw.org abusefilter emergency shutoff threshold down to 0.3 (duration: 00m 46s)

28 December 2018

  • curprev 18:1418:14, 28 December 2018imported>Stashbot 132,561 bytes +537 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 97446843a27 T212667 - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s)

27 December 2018

  • curprev 19:3119:31, 27 December 2018imported>Stashbot 132,024 bytes +1,926 mobrovac@deploy1001: Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 (duration: 25m 49s)

26 December 2018

  • curprev 16:3716:37, 26 December 2018imported>Stashbot 130,098 bytes +468 godog: clear logstash persistent queue /var/lib/logstash on logstash100[789]

24 December 2018

23 December 2018

22 December 2018

21 December 2018

20 December 2018

  • curprev 00:0900:09, 20 December 2018imported>Stashbot 119,323 bytes +8,184 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/479911/ (duration: 00m 53s)

19 December 2018

  • curprev 01:2701:27, 19 December 2018imported>Stashbot 111,139 bytes +5,652 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: g 480681 make maintenance script dry run more useful (duration: 00m 52s)

18 December 2018

16 December 2018

  • curprev 09:5209:52, 16 December 2018imported>Stashbot 92,602 bytes +240 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)

15 December 2018

14 December 2018

  • curprev 23:5923:59, 14 December 2018imported>Stashbot 235,055 bytes +4,222 mutante: LDAP: added aezell to wmf group (T211945) for grafana access
  • curprev 01:3901:39, 14 December 2018imported>Stashbot 230,833 bytes +14,006 mutante: install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)

13 December 2018

  • curprev 00:5200:52, 13 December 2018imported>Stashbot 216,827 bytes +9,257 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php: T211806 (duration: 00m 51s)

12 December 2018

11 December 2018

  • curprev 01:0301:03, 11 December 2018imported>Stashbot 197,142 bytes +9,075 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure localized logos for nywiki (T211570) (duration: 01m 36s)

10 December 2018

  • curprev 00:5500:55, 10 December 2018imported>Stashbot 188,067 bytes +620 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

8 December 2018

  • curprev 23:0723:07, 8 December 2018imported>Stashbot 187,447 bytes +460 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)

7 December 2018

6 December 2018

  • curprev 00:5500:55, 6 December 2018imported>Stashbot 177,878 bytes +11,714 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)

5 December 2018

4 December 2018

  • curprev 01:2201:22, 4 December 2018imported>Stashbot 153,921 bytes +10,468 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703

3 December 2018

  • curprev 00:3400:34, 3 December 2018imported>Stashbot 143,453 bytes +402 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

1 December 2018

  • curprev 16:4816:48, 1 December 2018imported>Stashbot 143,051 bytes +106 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904

30 November 2018

29 November 2018

  • curprev 00:0800:08, 29 November 2018imported>Stashbot 131,984 bytes +8,825 catrope@deploy1001: Synchronized wmf-config/throttle.php: T210681 (duration: 01m 04s)

28 November 2018

  • curprev 01:4501:45, 28 November 2018imported>Stashbot 123,159 bytes +10,054 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)

27 November 2018

  • curprev 00:3800:38, 27 November 2018imported>Stashbot 113,105 bytes +8,723 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix weird caching issue maybe for Ian's patch (duration: 00m 46s)

25 November 2018

23 November 2018

22 November 2018

21 November 2018

  • curprev 23:3423:34, 21 November 2018imported>Stashbot 97,893 bytes +6,623 mutante: rsyncing /home from rutherfordium.eqiad to people1001.eqiad (people.wikimedia.org) T210036
  • curprev 00:4800:48, 21 November 2018imported>Stashbot 91,270 bytes +7,660 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474967 Disable FlaggedRevs, enable RC patrol and add rights on srwikinews (duration: 00m 47s)

20 November 2018

  • curprev 00:5500:55, 20 November 2018imported>Stashbot 83,610 bytes +6,935 catrope@deploy1001: Synchronized static/images/project-logos/: Correct logos for Sindhi Wiktionary (duration: 00m 47s)

18 November 2018

17 November 2018

  • curprev 07:0207:02, 17 November 2018imported>Stashbot 76,539 bytes +331 volans|off: 'reset modified attributes' on IcingaUI for db1078 (and mgmt) and all its services

16 November 2018

  • curprev 23:1323:13, 16 November 2018imported>Stashbot 76,208 bytes +2,465 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010 (duration: 00m 23s)
  • curprev 00:5500:55, 16 November 2018imported>Stashbot 73,743 bytes +12,226 mutante: some users reported missing files in home dirs on mwmaint1002, reversed rsyncd/ferm setup and rsynced /home from mwmaint2001 to /root on mwmaint1002, restored individually where requested, rsync is not fully automatic but puppetized with rsync...

15 November 2018

  • curprev 00:5000:50, 15 November 2018imported>Stashbot 61,517 bytes +9,591 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: increase Schema.org page split test to 5% sampling T208755 (duration: 00m 54s)

14 November 2018

12 November 2018

  • curprev 19:2219:22, 12 November 2018imported>Stashbot 44,564 bytes +4,905 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T208663 4ff32d1df - Enable moving files for users with patrol and rollbacker rights on srwiki (duration: 00m 54s)

10 November 2018

  • curprev 01:1001:10, 10 November 2018imported>Stashbot 39,659 bytes +4,598 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s)

9 November 2018

  • curprev 00:0700:07, 9 November 2018imported>Stashbot 35,061 bytes +5,634 ejegg: updated fundraising CiviCRM from 07183ed7cc to 02cc1f80d4

8 November 2018

  • curprev 00:4300:43, 8 November 2018imported>Stashbot 29,427 bytes +10,411 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader: Fail less hard when JSON serialization of config fails I673f59d93 (duration: 00m 53s)

7 November 2018

  • curprev 00:4600:46, 7 November 2018imported>Stashbot 19,016 bytes +4,891 mutante: tegmen - shutting down for renaming and reinstall (T208824)

6 November 2018

  • curprev 00:2600:26, 6 November 2018imported>Stashbot 14,125 bytes +6,053 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s)

4 November 2018

3 November 2018

  • curprev 09:3509:35, 3 November 2018imported>Stashbot 7,058 bytes +112 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2 November 2018

  • curprev 17:0417:04, 2 November 2018imported>Stashbot 6,946 bytes +1,465 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}

1 November 2018

30 October 2018

29 October 2018

  • curprev 21:4421:44, 29 October 2018imported>Stashbot 171,385 bytes +15,633 mutante: contint2001 - installing libtiff upgrade, with the same command from apt/history.log that had shown as failed

28 October 2018

  • curprev 23:3623:36, 28 October 2018imported>Stashbot 155,752 bytes +600 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/Graph: T184128 - I02da92de33 (duration: 00m 58s)
  • curprev 00:0300:03, 28 October 2018imported>Stashbot 155,152 bytes +2,187 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/CirrusSearch: T206967 - Ia23d19cf1e6 (duration: 01m 02s)

27 October 2018

  • curprev 00:0000:00, 27 October 2018imported>Stashbot 152,965 bytes +4,090 mutante: icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

26 October 2018

  • curprev 01:4101:41, 26 October 2018imported>Stashbot 148,875 bytes +9,098 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207900 (b74911f6201) enable csp users with session all group1 wikis (duration: 00m 55s)

24 October 2018

  • curprev 23:2423:24, 24 October 2018imported>Stashbot 139,777 bytes +6,282 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/469495/ (duration: 00m 54s)
  • curprev 01:4601:46, 24 October 2018imported>Stashbot 133,495 bytes +5,751 tstarling@deploy1001: Synchronized php-1.32.0-wmf.26/includes/page/WikiPage.php: fix deletion performance regression T207530 (duration: 00m 55s)

23 October 2018

21 October 2018

20 October 2018

  • curprev 23:0523:05, 20 October 2018imported>Stashbot 116,878 bytes +662 reedy@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/CentralAuth/: Update setEmail (duration: 00m 55s)

19 October 2018

  • curprev 20:3320:33, 19 October 2018imported>Stashbot 116,216 bytes +3,360 twentyafterfour: deployed RCFilters: Fix completely broken highlight circles refs T207472

18 October 2018

  • curprev 22:0022:00, 18 October 2018imported>Stashbot 112,856 bytes +11,112 mutante: lvs1011,lvs1012 - manually editing nagios NRPE config and restarting service (to make monitoring from icinga1001 work and puppet is disabled)
  • curprev 00:3200:32, 18 October 2018imported>Stashbot 101,744 bytes +7,362 twentyafterfour: restarting apache on phab1001 to apply b3bfff1138d1212b318392b5e18ac0bfd6f78108

16 October 2018

  • curprev 22:1122:11, 16 October 2018imported>Stashbot 94,382 bytes +10,318 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 6 (duration: 01m 18s)
  • curprev 00:2300:23, 16 October 2018imported>Stashbot 84,064 bytes +10,295 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@c81dd9e]: Redeploy Updater for removal of props channel (duration: 10m 21s)

15 October 2018

  • curprev 00:4500:45, 15 October 2018imported>Stashbot 73,769 bytes +1,095 krinkle@deploy1001: Synchronized multiversion/MWRealm.php: I79fb3d194a58: use env.php (duration: 00m 49s)

14 October 2018

  • curprev 00:4800:48, 14 October 2018imported>Stashbot 72,674 bytes +412 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/CentralAuth/includes/specials/SpecialGlobalGroupMembership.php: T203767 - If2bfa092b (duration: 00m 50s)

12 October 2018

  • curprev 18:5618:56, 12 October 2018imported>Stashbot 72,262 bytes +1,505 brion: restarted vp9 background transcodes in eqiad, via mwmaint1002

11 October 2018

  • curprev 23:3323:33, 11 October 2018imported>Stashbot 70,757 bytes +8,125 Reedy: ran mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=gomwiki T206741

10 October 2018

  • curprev 23:0823:08, 10 October 2018imported>Stashbot 62,632 bytes +8,966 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/maintenance/resources/foreign-resources.yaml: Ic865e7077d (duration: 00m 49s)
  • curprev 00:1900:19, 10 October 2018imported>Stashbot 53,666 bytes +6,996 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/utils/UIDGenerator.php: T94522 - I2a0c51bea58 (duration: 00m 56s)

9 October 2018

  • curprev 00:0400:04, 9 October 2018imported>Stashbot 46,670 bytes +5,648 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/libs/rdbms/database: T201900 - I8ae754a2518 (duration: 00m 59s)

7 October 2018

  • curprev 16:4016:40, 7 October 2018imported>Stashbot 41,022 bytes +552 dereckson: Reset user email for account "Dominic Mayers" (T206421)

6 October 2018

5 October 2018

  • curprev 23:5023:50, 5 October 2018imported>Stashbot 39,744 bytes +4,624 bblack: <<<<<<< repooling eqiad edge caches, a few days ahead of intended switchback next Weds, to alleviate some traffic engineering concerns over the weekend >>>>>>
  • curprev 00:3100:31, 5 October 2018imported>Stashbot 35,120 bytes +8,957 mutante: LDAP: added user skvjold to group wmf (T204377)

3 October 2018

2 October 2018

  • curprev 16:5316:53, 2 October 2018imported>Stashbot 13,748 bytes −577,114 jynus: setup test s3 replication channel on db1110 (filtered)

18 September 2018

  • curprev 23:2123:21, 18 September 2018imported>Stashbot 590,862 bytes −591,903 twentyafterfour@deploy1001: Synchronized wmf-config/: SWAT: tested on mwdebug1001, now syncing wmf-config settings to the whole cluster (duration: 00m 59s)

17 March 2018

  • curprev 00:1300:13, 17 March 2018imported>Stashbot 1,182,765 bytes +4,645 mutante: running puppet on all cache::misc to rename director bromine to webserver_misc_static (T188163)

16 March 2018

  • curprev 01:4601:46, 16 March 2018imported>Stashbot 1,178,120 bytes +153 XioNoX: librenms IRC bot moved to -operations channel. Doc on how to turn it off is on https://wikitech.wikimedia.org/wiki/LibreNMS#IRC_Alerting
  • curprev 01:0001:00, 16 March 2018imported>Stashbot 1,177,967 bytes +8,937 reedy@tin: Synchronized php-1.31.0-wmf.25/includes/specials/pagers/NewFilesPager.php: Fix T189846 (duration: 00m 58s)

15 March 2018

  • curprev 00:2500:25, 15 March 2018imported>Stashbot 1,169,030 bytes +11,582 tgr@tin: Synchronized php-1.31.0-wmf.24/extensions/Wikibase/client/includes/RecentChanges/ExternalChangeFactory.php: T189320 Use only local part of username when building the RC line (duration: 01m 18s)

14 March 2018

  • curprev 00:0100:01, 14 March 2018imported>Stashbot 1,157,448 bytes +6,839 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: crwiki logo (duration: 01m 15s)

12 March 2018

  • curprev 22:5222:52, 12 March 2018imported>Stashbot 1,150,609 bytes +7,389 eileen: update civicrm revision changed from a819d64d98 to 8652db05f5, config revision is 08b7e6216e - update civicrm.settings.php

11 March 2018

  • curprev 08:5008:50, 11 March 2018imported>Stashbot 1,143,220 bytes +139 elukey: executed sudo rm /etc/logrotate.d/kafkatee-webrequest-analytics on oxygen/rhenium to stop daily cronspam

10 March 2018

9 March 2018

  • curprev 23:2923:29, 9 March 2018imported>Stashbot 1,142,976 bytes +206 tgr@tin: Synchronized php-1.31.0-wmf.24/extensions/ReadingLists/src/Api/ApiQueryReadingListEntries.php: T189272 fix stupid ReadingLists typo breaking production (duration: 00m 54s)
  • curprev 19:4319:43, 9 March 2018imported>Stashbot 1,142,770 bytes +4,654 foks: changed global email for User:Mathmensch
  • curprev 00:4100:41, 9 March 2018imported>Stashbot 1,138,116 bytes +10,852 thcipriani@tin: Synchronized static/images/project-logos: SWAT: Update logos for Banyumasan and Urdu Wikipedias T189155 PART IV (duration: 00m 58s)

8 March 2018

  • curprev 01:1701:17, 8 March 2018imported>Stashbot 1,127,264 bytes +9,869 twentyafterfour: phabricator update completed

7 March 2018

6 March 2018

  • curprev 01:0001:00, 6 March 2018imported>Stashbot 1,106,991 bytes +9,749 tgr@tin: Synchronized wmf-config/InitialiseSettings.php: refresh wmf-config/InitialiseSettings, seems to have stuck in old state on some servers after doing the initial sync in the wrong order (duration: 00m 57s)

4 March 2018

  • curprev 20:1620:16, 4 March 2018imported>Stashbot 1,097,242 bytes +497 tgr: T188721 ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --ignorestatus --logwiki=metawiki 'Erik Fastman' 'Glorious Engine'

3 March 2018

  • curprev 14:1714:17, 3 March 2018imported>Stashbot 1,096,745 bytes +725 akosiaris: 13:56:20 ema: powercycle ganeti1005 T181121
  • curprev 01:2101:21, 3 March 2018imported>Stashbot 1,096,020 bytes +3,349 chasemp: labnodepool1001:~# service nodepool stop

2 March 2018

  • curprev 00:4900:49, 2 March 2018imported>Stashbot 1,092,671 bytes +12,320 ebernhardson@tin: Synchronized wmf-config/flaggedrevs.php: SWAT: T148603: (duration: 00m 57s)

1 March 2018

  • curprev 00:4300:43, 1 March 2018imported>Stashbot 1,080,351 bytes +10,690 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Clean up $wgEchoPerUserBlacklist setting (duration: 01m 14s)

28 February 2018

  • curprev 00:5500:55, 28 February 2018imported>Stashbot 1,069,661 bytes +10,577 demon@tin: Synchronized scap/plugins/wmfbetaautoupdate.py: no-op (duration: 01m 14s)

27 February 2018

  • curprev 00:5500:55, 27 February 2018imported>Stashbot 1,059,084 bytes +7,771 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: add very likely bad faith filter on svwiki (T174560) (duration: 00m 57s)

25 February 2018

24 February 2018

  • curprev 06:1206:12, 24 February 2018imported>Stashbot 1,051,215 bytes +148 marostegui: Reload haproxy on dbproxy1005
  • curprev 01:1801:18, 24 February 2018imported>Stashbot 1,051,067 bytes +5,059 demon@tin: Synchronized docroot/noc/conf/index.php: fix dblist links from listing (duration: 00m 56s)

23 February 2018

  • curprev 01:2001:20, 23 February 2018imported>Stashbot 1,046,008 bytes +9,207 eileen: update civicrm revision changed from aa251f1a93 to a47eafcbad, config revision is c1787646bc

22 February 2018

  • curprev 01:0501:05, 22 February 2018imported>Stashbot 1,036,801 bytes +11,188 anomie: Running cleanupBlocks.php on more wikis for T187834: alswiki bgwiki bhwiki cawiki dewiki elwiki eswiki frwiki hewiki hiwiki huwiki hywiki jawiki jawikibooks jawikinews jawikiquote jawikisource jawiktionary kawiki kowiki mswiki mswiktionary rowi...

21 February 2018

  • curprev 01:1601:16, 21 February 2018imported>Stashbot 1,025,613 bytes +11,639 eileen: update civicrm revision changed from efba904b06 to 43a7641597, config revision is ef884a2c5d

19 February 2018

  • curprev 23:2223:22, 19 February 2018imported>Stashbot 1,013,974 bytes +6,048 eileen: re-enable omnirecipient jobs - process-control config revision is 8c7ce87207

18 February 2018

  • curprev 15:5015:50, 18 February 2018imported>Stashbot 1,007,926 bytes +118 _joe_: rolling restart (1 at a time, staggered by 2 minutes) of 18 api appservers in equiad

17 February 2018

  • curprev 17:3317:33, 17 February 2018imported>Stashbot 1,007,808 bytes +297 twentyafterfour: restarting apache on phab1001 to clear deadlocked workers. refs T182832

16 February 2018

  • curprev 21:1221:12, 16 February 2018imported>Stashbot 1,007,511 bytes +5,314 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3 | T187567
  • curprev 00:1700:17, 16 February 2018imported>Stashbot 1,002,197 bytes +8,436 ebernhardson@tin: Synchronized php-1.31.0-wmf.21/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: SWAT: T187454 fix text selection on #wpTextbox1 (duration: 00m 58s)

15 February 2018

  • curprev 01:2701:27, 15 February 2018imported>Stashbot 993,761 bytes +11,159 twentyafterfour: restarting apache2 on phab1001 to free deadlocked php processes.

14 February 2018

13 February 2018

11 February 2018

10 February 2018

  • curprev 16:5116:51, 10 February 2018imported>Stashbot 960,906 bytes +187 legoktm@tin: Synchronized php-1.31.0-wmf.20/includes/specials/SpecialLog.php: SpecialLog: Fix results when no offender is specified - T186950 (duration: 00m 57s)
  • curprev 01:1101:11, 10 February 2018imported>Stashbot 960,719 bytes +8,233 demon@tin: Finished deploy [gerrit/gerrit@5d5193e]: one last gitiles rebuild before the weekend (duration: 00m 10s)

9 February 2018

  • curprev 00:1800:18, 9 February 2018imported>Stashbot 952,486 bytes +13,814 demon@tin: rebuilt and synchronized wikiversions files: surprise, it broke. revert group1 back to wmf.20

8 February 2018

  • curprev 01:2901:29, 8 February 2018imported>Stashbot 938,672 bytes +10,937 andrew@tin: Started deploy [horizon/deploy@9223ba7]: Now with static content, I hope

7 February 2018

6 February 2018

4 February 2018

  • curprev 22:4122:41, 4 February 2018imported>Stashbot 909,306 bytes +316 elukey: restart aphlict.service on phab1001 to force it to pick up the new logfile (/var/log/aphlict/aphlict.log rather than the .log.1)

3 February 2018

2 February 2018

  • curprev 23:4023:40, 2 February 2018imported>Stashbot 908,907 bytes +4,624 no_justification: gerrit: one last restart to try and force gerrit/phab session restart

1 February 2018

  • curprev 23:3723:37, 1 February 2018imported>Stashbot 904,283 bytes +2,967 mutante: creating new 100GB virtual disk for ganeti VM meitnerium (T186020)

31 January 2018

  • curprev 23:5623:56, 31 January 2018imported>Stashbot 901,316 bytes +4,831 mutante: restarting apache on phabricator server, same pattern as described in T182832
  • curprev 00:5500:55, 31 January 2018imported>Stashbot 896,485 bytes +1,226 krinkle@tin: Synchronized wmf-config: no-op, adding files for beta cluster (duration: 00m 59s)

30 January 2018

  • curprev 00:3200:32, 30 January 2018imported>Stashbot 895,259 bytes +3,803 demon@tin: rebuilt and synchronized wikiversions files: not changing versions, testing something

28 January 2018

26 January 2018

  • curprev 16:3216:32, 26 January 2018imported>Stashbot 891,415 bytes +737 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: BETA: Enable FileImporter on testwiki with open config PT 2/2 (duration: 00m 56s)

25 January 2018

  • curprev 22:2422:24, 25 January 2018imported>Stashbot 890,678 bytes +783 mutante: restarting gerrit service to apply a few small config changes https://gerrit.wikimedia.org/r/#/q/topic:gerrit-trivial-tweaks+(status:open+OR+status:merged)
  • curprev 00:2500:25, 25 January 2018imported>Stashbot 889,895 bytes +645 urandom: bootstrapping restbase2008-a - T184100

24 January 2018

22 January 2018

  • curprev 22:3822:38, 22 January 2018imported>Stashbot 886,795 bytes +5,025 mutante: rebooting the-server-formerly-known-as-amslvs4 to PXE to reinstall it as bast3003. doesnt work

21 January 2018

20 January 2018

19 January 2018

  • curprev 22:5322:53, 19 January 2018imported>Stashbot 881,103 bytes +4,027 matt_flaschen: Ran (time foreachwikiindblist flow.dblist extensions/Flow/maintenance/FlowFixInconsistentBoards.php --force) 2>&1|tee --append ~/FlowFixInconsistentBoards_all_2018-01-19_actual_force.txt
  • curprev 01:2001:20, 19 January 2018imported>Stashbot 877,076 bytes +12,450 TimStarling: attempting to restore home_pmtpa from bacula to bast1001

18 January 2018

  • curprev 01:0801:08, 18 January 2018imported>Stashbot 864,626 bytes +8,658 twentyafterfour: phabricator deployment finished without incident.

17 January 2018

  • curprev 00:2900:29, 17 January 2018imported>Stashbot 855,968 bytes +11,542 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T182616 Remove cirrus AB test config for hewiki (duration: 01m 09s)

15 January 2018

  • curprev 23:4023:40, 15 January 2018imported>Stashbot 844,426 bytes +6,099 demon@tin: Synchronized wmf-config/InitialiseSettings.php: turn educationprogram back on for cs.wikipedia -- turns out there was no consensus and a patch should never have been written 😡 (duration: 01m 13s)

12 January 2018

11 January 2018

10 January 2018

9 January 2018

7 January 2018

  • curprev 20:2520:25, 7 January 2018imported>Stashbot 792,129 bytes +609 demon@tin: Synchronized wmf-config/interwiki.php: auto-sync with my plugin was busted 🙃 (duration: 00m 50s)

6 January 2018

  • curprev 08:0908:09, 6 January 2018imported>Stashbot 791,520 bytes +369 elukey: re-enable eventlogging mysql consumers after database maintenance - T168414

5 January 2018

  • curprev 22:2822:28, 5 January 2018imported>Stashbot 791,151 bytes +4,738 tgr: T184263 ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=eswiki --logwiki=metawiki "Mega849" "Mega809"

4 January 2018

3 January 2018

  • curprev 23:0223:02, 3 January 2018imported>Stashbot 779,920 bytes +9,314 twentyafterfour: restarted apache on phab1001 to clear hung workers (refs T182832)
  • curprev 01:2601:26, 3 January 2018imported>Stashbot 770,606 bytes +9,111 eileen: civicrm updated civicrm revision changed from ffa9d7fc7a to 429a5c5385, config revision is a7b9b58595

1 January 2018

  • curprev 22:4522:45, 1 January 2018imported>Stashbot 761,495 bytes +701 eileen1: re-enable silverpop jobs process-control config revision is 4a4c61ae1b

30 December 2017

29 December 2017

  • curprev 18:3218:32, 29 December 2017imported>Stashbot 760,718 bytes +813 mutante: gerrit: restart service to apply change from Dec 20th, avoid logspam due to unapplied config change

28 December 2017

27 December 2017

26 December 2017

  • curprev 16:3316:33, 26 December 2017imported>Stashbot 759,543 bytes +1,456 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 (duration: 01m 44s)

24 December 2017

23 December 2017

22 December 2017

  • curprev 18:3118:31, 22 December 2017imported>Stashbot 757,594 bytes +1,768 mobrovac@tin: Finished deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557 (duration: 02m 45s)

21 December 2017

  • curprev 19:2519:25, 21 December 2017imported>Stashbot 755,826 bytes +3,470 robh: cp4032 going offline for memory swap
  • curprev 01:1701:17, 21 December 2017imported>Stashbot 752,356 bytes +5,916 eileen: update CiviCRM civicrm revision changed from 9d6ba74f57 to ffa9d7fc7a, config revision is da6fa4cce9

19 December 2017

  • curprev 21:5121:51, 19 December 2017imported>Stashbot 746,440 bytes +5,951 XioNoX: removing local-as AS43821 from ams transits - T167840
  • curprev 00:4400:44, 19 December 2017imported>Stashbot 740,489 bytes +6,658 mutante: einsteinium: sudo systemctl restrart ircecho (alias kick-icinga-wm)

18 December 2017

16 December 2017

  • curprev 17:2717:27, 16 December 2017imported>Stashbot 733,671 bytes +1,212 no_justification: gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover
  • curprev 01:1601:16, 16 December 2017imported>Stashbot 732,459 bytes +4,280 mutante: reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package

15 December 2017

  • curprev 01:2301:23, 15 December 2017imported>Stashbot 728,179 bytes +9,669 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES: Fix broken join conditions (T182936) (duration: 00m 57s)

14 December 2017

  • curprev 00:1200:12, 14 December 2017imported>Stashbot 718,510 bytes +7,734 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T182534 (duration: 01m 08s)

13 December 2017

  • curprev 00:4900:49, 13 December 2017imported>Stashbot 710,776 bytes +14,879 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)

12 December 2017

  • curprev 00:5300:53, 12 December 2017imported>Stashbot 695,897 bytes +7,791 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Remove manual firejailing of Score binaries (T181535) (duration: 00m 56s)

10 December 2017

  • curprev 20:3420:34, 10 December 2017imported>Stashbot 688,106 bytes +317 elukey: execute restart-hhvm on mw1312 - hhvm stuck multiple times queueing requests

9 December 2017

  • curprev 17:0017:00, 9 December 2017imported>Stashbot 687,789 bytes +1,492 apergos: restarted hhvm on mw1276, the same old hang with the same old symptoms
  • curprev 00:1100:11, 9 December 2017imported>Stashbot 686,297 bytes +2,589 Jamesofur: removed 2FA from EVinente after verification T182373

8 December 2017

6 December 2017

  • curprev 23:4523:45, 6 December 2017imported>Stashbot 673,198 bytes +8,263 eileen: update CiviCRM from 85263f625763a454b2664062154c8b791e5a6838 to 2db9c761884c1c48f4e1dd41be416e029b23b481 (improve org merges)
  • curprev 01:0101:01, 6 December 2017imported>Stashbot 664,935 bytes +7,019 eileen: update process-control to 55529ea12d9f16c259fa3 - threshold for $500+ dedupe was too low

5 December 2017

  • curprev 00:2900:29, 5 December 2017imported>Stashbot 657,916 bytes +9,704 mutante: bast4001 - removing ganglia aggregators, package, config...

3 December 2017

2 December 2017

  • curprev 17:5617:56, 2 December 2017imported>Stashbot 647,571 bytes +231 marostegui: Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359
  • curprev 00:0300:03, 2 December 2017imported>Stashbot 647,340 bytes +5,857 mutante: tried one more time on db2028,db2029, both trusty. on db2028: gmond was running as user ganglia-monitor, failed, had to manually kill the process, run puppet again then ok. on db2029, gmond was running as "499" but puppet just ran and removed...

1 December 2017

  • curprev 00:1300:13, 1 December 2017imported>Stashbot 641,483 bytes +8,175 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: HTML5 sections be upon us! (duration: 00m 45s)

30 November 2017

  • curprev 01:1701:17, 30 November 2017imported>Stashbot 633,308 bytes +8,979 ejegg: updated fundraising tools from 6d4b6f388e64ff4dc9a6a501bc8339fc1294fe9f to 6e604fd9f3ca57935e2c35e37b7e85b4ca438d6a

29 November 2017

27 November 2017

  • curprev 22:2922:29, 27 November 2017imported>Stashbot 614,927 bytes +10,005 mutante: labtestnet2001 - re-enabled puppet to decom ganglia, no errors

25 November 2017

24 November 2017

  • curprev 13:5213:52, 24 November 2017imported>Stashbot 603,543 bytes +3,195 moritzm: removing git packages from jessie-wikimedia/experimental (replaced by component/git)
  • curprev 00:1200:12, 24 November 2017imported>Stashbot 600,348 bytes +4,938 reedy@tin: Synchronized docroot/mediawiki/keys/: Add Brian Wolff's key (duration: 00m 45s)

23 November 2017

  • curprev 01:2901:29, 23 November 2017imported>Stashbot 595,410 bytes +7,884 demon@tin: Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthGlobalBlockInterwikiPrefix (duration: 00m 45s)

22 November 2017

  • curprev 01:0901:09, 22 November 2017imported>Stashbot 587,526 bytes +10,099 hoo: Cleaned out remaining T180934 related log blow up on snapshot1007 (dumpwikidatajson-wikidata-20171120-all-0.log)

21 November 2017

  • curprev 00:1900:19, 21 November 2017imported>Stashbot 577,427 bytes +11,039 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: T180863 (duration: 00m 49s)

19 November 2017

18 November 2017

17 November 2017

  • curprev 01:2801:28, 17 November 2017imported>Stashbot 559,627 bytes +11,959 ejegg: updated CiviCRM from 8454e0639eed6d4a0e7b01970dd5a6f9d426cb29 to 272580a4b0511b7eb5392dc087a2d21e56474ab9

16 November 2017

  • curprev 01:2901:29, 16 November 2017imported>Stashbot 547,668 bytes +11,330 twentyafterfour: Phabricator will be offline for a couple of minutes while I apply database migrations.

15 November 2017

  • curprev 00:5900:59, 15 November 2017imported>Stashbot 536,338 bytes +9,146 ejegg: updated payments-wiki from d150287f0b64d13589e1af728be0cb5b89f66699 to 210cb37d9973eabed06a520ac31fe74d30ffde4a

13 November 2017

  • curprev 22:4922:49, 13 November 2017imported>Stashbot 527,192 bytes +7,588 bawolff: deployed patch T119158 (Will affect language converter. -{}- no longer allowed in link urls)

12 November 2017

  • curprev 19:3919:39, 12 November 2017imported>Stashbot 519,604 bytes +153 urandom: Decommissioning Cassandra, restbase1007-b.eqiad.wmnet (T179422)
  • curprev 01:2901:29, 12 November 2017imported>Stashbot 519,451 bytes +214 urandom: Decommissioning Cassandra, restbase1007-a.eqiad.wmnet (T179422)

11 November 2017

  • curprev 00:4800:48, 11 November 2017imported>Stashbot 519,237 bytes +4,117 urandom: Decommissioning Cassandra, restbase2006-b.codfw.wmnet (T179422)

10 November 2017

  • curprev 01:2101:21, 10 November 2017imported>Stashbot 515,120 bytes +4,946 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/db: Use the main stash for LBFactory "memStash" parameter (duration: 00m 47s)

9 November 2017

  • curprev 01:0201:02, 9 November 2017imported>Stashbot 510,174 bytes +12,205 twentyafterfour: Not deploying any phabricator updates this week.

7 November 2017

  • curprev 23:5823:58, 7 November 2017imported>Stashbot 497,969 bytes +12,841 urandom: Decommissioning restbase2001-b.codfw.wmnet (T179422)

6 November 2017

4 November 2017

  • curprev 13:3413:34, 4 November 2017imported>Stashbot 474,166 bytes +274 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
  • curprev 00:1800:18, 4 November 2017imported>Stashbot 473,892 bytes +3,306 Krinkle: Finished whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)

2 November 2017

  • curprev 23:3623:36, 2 November 2017imported>Stashbot 470,586 bytes +6,195 krinkle@tin: Synchronized php-1.31.0-wmf.6/extensions/NavigationTiming/modules/ext.navigationTiming.js: Fix zero values - T178479 (duration: 00m 47s)
  • curprev 00:5200:52, 2 November 2017imported>Stashbot 464,391 bytes +7,145 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/extension.json: SWAT followup: Update SearchSatisfaction eventlogging schema revision id (duration: 00m 50s)

31 October 2017

  • curprev 23:5623:56, 31 October 2017imported>Stashbot 457,246 bytes +7,141 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Adjust cirrussearch interleave configuration for AB test (duration: 00m 50s)

30 October 2017

  • curprev 23:2823:28, 30 October 2017imported>Stashbot 450,105 bytes +11,992 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update cirrussearch MLR model for enwiki (duration: 00m 50s)

29 October 2017

28 October 2017

  • curprev 21:0321:03, 28 October 2017imported>Stashbot 437,356 bytes +1,408 bblack: cp1067 (current target cache): disabling the relatively-new VCL that sets do_stream=false if !CL on applayer fetches...

27 October 2017

26 October 2017

  • curprev 22:4722:47, 26 October 2017imported>Stashbot 433,292 bytes +10,731 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata to wmf.4

25 October 2017

  • curprev 23:4823:48, 25 October 2017imported>Stashbot 422,561 bytes +8,480 legoktm@tin: Synchronized php-1.31.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: MWInternalLinkContextItem: increase specificity to override OOUI changes - T178933 (duration: 00m 50s)

24 October 2017

  • curprev 22:5522:55, 24 October 2017imported>Stashbot 414,081 bytes +5,808 ejegg: updated civicrm from from 0546446640c8b7485c78b4f3836a38630248b44e to 25c7a33e3e3e9b8a5020d826982b3dbb64057614

23 October 2017

  • curprev 23:0923:09, 23 October 2017imported>Stashbot 408,273 bytes +7,033 demon@tin: Synchronized wmf-config/throttle.php: Working Class Movement Library (Salford) throttle rule (duration: 00m 46s)

21 October 2017

  • curprev 08:1608:16, 21 October 2017imported>Stashbot 401,240 bytes +203 hashar: Mass code-review+2 changes made by LibraryUpdater that already had Code-Review:+2 and NOT verified=-1 ( ping legoktm )

20 October 2017

  • curprev 23:4623:46, 20 October 2017imported>Stashbot 401,037 bytes +2,650 demon@tin: Finished deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin (duration: 00m 09s)

19 October 2017

  • curprev 22:2122:21, 19 October 2017imported>Stashbot 398,387 bytes +5,288 addshore: stop running extra change dispatchers on terbium for wikidatawiki
  • curprev 00:0100:01, 19 October 2017imported>Stashbot 393,099 bytes +7,053 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/385106/2 (duration: 00m 50s)

17 October 2017

16 October 2017

15 October 2017

  • curprev 11:1611:16, 15 October 2017imported>Stashbot 367,498 bytes +134 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916

14 October 2017

  • curprev 03:5303:53, 14 October 2017imported>Stashbot 367,364 bytes +148 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)

13 October 2017

  • curprev 23:0423:04, 13 October 2017imported>Stashbot 367,216 bytes +3,525 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)