You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

6 September 2020

  • curprev 19:4519:45, 6 September 2020imported>Stashbot 294,441 bytes +598 marostegui@cumin1001: dbctl commit (dc=all): 'Decrease db2127's weight a bit', diff saved to https://phabricator.wikimedia.org/P12496 and previous config saved to /var/cache/conftool/dbconfig/20200906-194512-marostegui.json

5 September 2020

4 September 2020

  • curprev 01:3501:35, 4 September 2020imported>Stashbot 289,064 bytes +20,066 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

2 September 2020

1 September 2020

  • curprev 22:3922:39, 1 September 2020imported>Stashbot 257,665 bytes +5,601 Urbanecm: [urbanecm@mwmaint2001 ~]$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=sysop_itwiki Pierpao (T261722)
  • curprev 00:0200:02, 1 September 2020imported>Stashbot 252,064 bytes +8,872 mutante: wb2-grrrri was not running and wikibugs had no more Gerrit updates since a while

30 August 2020

29 August 2020

  • curprev 18:0518:05, 29 August 2020imported>Stashbot 243,128 bytes +402 Amir1: end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451)

28 August 2020

  • curprev 21:5321:53, 28 August 2020imported>Stashbot 242,726 bytes +5,179 ryankemper: `sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues
  • curprev 00:0600:06, 28 August 2020imported>Stashbot 237,547 bytes +27,285 eileen: process-control config revision is dd541a25dc

26 August 2020

  • curprev 23:3523:35, 26 August 2020imported>Stashbot 210,262 bytes +10,539 eileen: civicrm revision changed from d2e80f7522 to 481ab742db, config revision is e2ab4d7c1f

25 August 2020

  • curprev 21:5121:51, 25 August 2020imported>Stashbot 199,723 bytes +10,845 mutante: xhgui1001/xhgui2001 - Unpacking xhgui (0.12.0-2-wmf1) over (0.9.0-1-wmf1) (T260397)
  • curprev 00:5700:57, 25 August 2020imported>Stashbot 188,878 bytes +12,712 cdanis: T259621 ❌cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 homer 'cr*' commit 'drain cr2-esams transport link'

23 August 2020

  • curprev 20:3820:38, 23 August 2020imported>Stashbot 176,166 bytes +518 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

22 August 2020

  • curprev 19:3319:33, 22 August 2020imported>Stashbot 175,648 bytes +367 ryankemper: depooled wdqs1006 (still has 2.5 hours to catch up on)

21 August 2020

  • curprev 17:4317:43, 21 August 2020imported>Stashbot 175,281 bytes +1,663 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

20 August 2020

  • curprev 22:3122:31, 20 August 2020imported>Stashbot 173,618 bytes +10,495 eileen: civicrm revision changed from 27d5900f7d to ce28723709, config revision is 706cf3c898
  • curprev 00:5100:51, 20 August 2020imported>Stashbot 163,123 bytes +12,052 mutante: ms-be1039 - started failed ferm service

19 August 2020

  • curprev 01:0501:05, 19 August 2020imported>Stashbot 151,071 bytes +12,992 dpifke@deploy1001: Synchronized wmf-config/profiler.php: Deploying https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620139 (duration: 01m 18s)

18 August 2020

  • curprev 00:5300:53, 18 August 2020imported>Stashbot 138,079 bytes +29,239 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)

16 August 2020

15 August 2020

  • curprev 21:1821:18, 15 August 2020imported>Stashbot 108,768 bytes +123 gehel: depooling wdqs1004 and restarting services, will wait to catch up on lag before repooling

14 August 2020

13 August 2020

  • curprev 23:3923:39, 13 August 2020imported>Stashbot 107,903 bytes +7,860 tzatziki: removing 3 files for legal compliance
  • curprev 00:3700:37, 13 August 2020imported>Stashbot 100,043 bytes +31,681 mutante: removing jenkins_service_running checks from secondary servers where it's stopped, manually from icinga config, running puppet on icinga

11 August 2020

  • curprev 23:4123:41, 11 August 2020imported>Stashbot 68,362 bytes +9,831 Urbanecm: Evening B&C window completed
  • curprev 00:3300:33, 11 August 2020imported>Stashbot 58,531 bytes +11,201 dpifke@deploy1001: Finished deploy [performance/arc-lamp@fc5f1c6]: Deploying latest attempt to fix T259167 (duration: 01m 03s)

9 August 2020

  • curprev 21:5821:58, 9 August 2020imported>Stashbot 47,330 bytes +195 ejegg: updated payments-wiki from cd012f37f1 to 932aacde54

8 August 2020

  • curprev 02:2302:23, 8 August 2020imported>Stashbot 47,135 bytes +243 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload

7 August 2020

  • curprev 16:4216:42, 7 August 2020imported>Stashbot 46,892 bytes +2,081 jforrester@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/DiscussionTools/: T259855 Revert new reply API (duration: 01m 06s)

6 August 2020

  • curprev 23:2123:21, 6 August 2020imported>Stashbot 44,811 bytes +12,330 catrope@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/: Fixes for WelcomeSurvey language question (T232410) (duration: 00m 59s)
  • curprev 00:3500:35, 6 August 2020imported>Stashbot 32,481 bytes +9,981 mutante: wtp2019 - reimaging - parsoid service does not work, unlike on all other wtp*, making sure it's clean

4 August 2020

  • curprev 22:4122:41, 4 August 2020imported>Stashbot 22,500 bytes +12,987 brennen: restarting php7.2-fpm on mw1404 for opcache issues

3 August 2020

  • curprev 23:4323:43, 3 August 2020imported>Stashbot 9,513 bytes +9,109 mutante: mwdebug1001 - temp installing apt-file for debugging an issue on mwmaint

1 August 2020

  • curprev 16:3016:30, 1 August 2020imported>Stashbot 404 bytes −726,353 Amir1: wikiadmin@10.64.32.197(avkwiki)> delete from site_identifiers; (T259122)

31 July 2020

  • curprev 23:4823:48, 31 July 2020imported>Stashbot 726,757 bytes +6,358 ejegg: updated payments-wiki from c365c136d2 to cd012f37f1
  • curprev 00:4700:47, 31 July 2020imported>Stashbot 720,399 bytes +10,664 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: disable lilypond execution again (duration: 01m 10s)

30 July 2020

  • curprev 00:5200:52, 30 July 2020imported>Stashbot 709,735 bytes +18,579 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

29 July 2020

  • curprev 00:5300:53, 29 July 2020imported>Stashbot 691,156 bytes +25,595 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1046.eqiad.wmnet

28 July 2020

  • curprev 00:1900:19, 28 July 2020imported>Stashbot 665,561 bytes +13,224 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephmon1003.eqiad.wmnet

25 July 2020

  • curprev 12:4112:41, 25 July 2020imported>Stashbot 652,337 bytes +620 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1096:3315 into s5 api afte db1082 crashed T258336', diff saved to https://phabricator.wikimedia.org/P12041 and previous config saved to /var/cache/conftool/dbconfig/20200725-124104-marostegui.json
  • curprev 00:4700:47, 25 July 2020imported>Stashbot 651,717 bytes +3,102 mutante: ganeti - removing disk 3 (20G) from webperf1002. the disks are 0-indexed, so the ones actually mounted are 0 (50G) and 1 (300G) (T257931)

24 July 2020

  • curprev 01:1901:19, 24 July 2020imported>Stashbot 648,615 bytes +12,821 ejegg: updated payments-wiki from 31a3de1130 to c365c136d2

23 July 2020

  • curprev 00:2400:24, 23 July 2020imported>Stashbot 635,794 bytes +17,409 legoktm@deploy1001: Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s)

21 July 2020

  • curprev 23:3723:37, 21 July 2020imported>Stashbot 618,385 bytes +16,371 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump cirrus MLR models to latest (duration: 01m 06s)
  • curprev 00:0200:02, 21 July 2020imported>Stashbot 602,014 bytes +20,177 ryankemper: Began Elasticsearch reindex job on index `dewiki_content` across [`eqiad`, `codfw`, `cloudelastic`], on `rkemper@mwmaint1002` under tmux session `reindex`. Should complete in <24 hours

19 July 2020

18 July 2020

  • curprev 21:4121:41, 18 July 2020imported>Stashbot 581,291 bytes +539 shdubsh: restart logstash on logstash200[456]
  • curprev 00:1300:13, 18 July 2020imported>Stashbot 580,752 bytes +4,067 dpifke: Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged.

16 July 2020

  • curprev 22:1622:16, 16 July 2020imported>Stashbot 576,685 bytes +13,808 mutante: testreduce1001 manually git clone 'scandium' branch of integration/visualdiff into /srv/visualdiff (T257906)

15 July 2020

  • curprev 23:5423:54, 15 July 2020imported>Stashbot 562,877 bytes +11,781 eileen: tools revision changed from 7b6018a16e to 711d671600
  • curprev 01:4501:45, 15 July 2020imported>Stashbot 551,096 bytes +15,235 eileen: tools revision changed from a9e7dc1559 to 7b6018a16e

14 July 2020

  • curprev 01:2301:23, 14 July 2020imported>Stashbot 535,861 bytes +9,041 ryankemper: Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002`

11 July 2020

  • curprev 19:1619:16, 11 July 2020imported>Stashbot 526,820 bytes +852 qchris: Restarting Gerrit on gerrit1001 to switch to new gerrit.war and zuul plugin

10 July 2020

  • curprev 21:5221:52, 10 July 2020imported>Stashbot 525,968 bytes +7,988 ryankemper: Started long-running reindex of Elasticsearch indices in `eqiad`, `codfw`, and `dewiki` on `mwmaint1002` under tmux session `reindex` for user `ryankemper`
  • curprev 01:4401:44, 10 July 2020imported>Stashbot 517,980 bytes +10,370 mutante: LDAP - adding coka to wmde and nda (T257038)

9 July 2020

  • curprev 00:5800:58, 9 July 2020imported>Stashbot 507,610 bytes +15,943 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6c1b646e T256395"'

7 July 2020

  • curprev 22:4122:41, 7 July 2020imported>Stashbot 491,667 bytes +21,022 mutante: new Wikimedia Annual Report 2019 now available on annual.wikimedia.org
  • curprev 00:2200:22, 7 July 2020imported>Stashbot 470,645 bytes +11,069 cstone: civicrm revision changed from a48caf0f37 to d73ee2e73f

5 July 2020

  • curprev 21:5021:50, 5 July 2020imported>Stashbot 459,576 bytes +1,640 qchris: Restarting gerrit on gerrit1001 to pick up new war and jars.

4 July 2020

  • curprev 19:2319:23, 4 July 2020imported>Stashbot 457,936 bytes +722 qchris@deploy1001: Finished deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002 (duration: 00m 08s)

3 July 2020

  • curprev 21:4921:49, 3 July 2020imported>Stashbot 457,214 bytes +4,336 reedy@deploy1001: Synchronized php-1.35.0-wmf.39/extensions/Score/: Sync maintenance script (duration: 00m 58s)

2 July 2020

  • curprev 23:2223:22, 2 July 2020imported>Stashbot 452,878 bytes +4,683 jhuneidi@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
  • curprev 00:5300:53, 2 July 2020imported>Stashbot 448,195 bytes +11,373 eileen: tools revision changed from 806e2b4412 to e38f7a83d4

1 July 2020

  • curprev 00:1400:14, 1 July 2020imported>Stashbot 436,822 bytes +12,688 krinkle@deploy1001: Synchronized private/PrivateSettings.php: T254795 - Set $wmgXhguiDBuser and $wmgXhguiDBpasswor (duration: 01m 06s)

29 June 2020

  • curprev 23:2823:28, 29 June 2020imported>Stashbot 424,134 bytes +9,692 eileen: civicrm revision changed from 52a32f2d66 to 391d0fdf75, config revision is f1b4bdb7b7

28 June 2020

  • curprev 21:4321:43, 28 June 2020imported>Stashbot 414,442 bytes +271 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: no-op I56eb4a802 (duration: 00m 58s)

27 June 2020

26 June 2020

  • curprev 18:4218:42, 26 June 2020imported>Stashbot 411,866 bytes +4,536 robh: all ulsfo onsite work completed as of 30 minutes ago
  • curprev 01:4101:41, 26 June 2020imported>Stashbot 407,330 bytes +15,922 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6cc5f3e6 T256395"'

25 June 2020

  • curprev 00:4800:48, 25 June 2020imported>Stashbot 391,408 bytes +12,937 twentyafterfour: restart php-fpm on phab1001 to fix T256343

24 June 2020

  • curprev 00:3500:35, 24 June 2020imported>Stashbot 378,471 bytes +8,585 ejegg: restarted fundraising jobs on main CiviCRM box

22 June 2020

  • curprev 23:4123:41, 22 June 2020imported>Stashbot 369,886 bytes +10,612 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: touch for T247330 (duration: 00m 56s)

20 June 2020

  • curprev 22:5622:56, 20 June 2020imported>Stashbot 359,274 bytes +414 cdanis@cumin2001: dbctl commit (dc=all): 'db1088 seems to have crashed', diff saved to https://phabricator.wikimedia.org/P11611 and previous config saved to /var/cache/conftool/dbconfig/20200620-225624-cdanis.json

19 June 2020

  • curprev 18:1018:10, 19 June 2020imported>Stashbot 358,860 bytes +6,598 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt - T238230 (duration: 00m 59s)

18 June 2020

  • curprev 22:3022:30, 18 June 2020imported>Stashbot 352,262 bytes +9,559 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 56s)

17 June 2020

  • curprev 23:2523:25, 17 June 2020imported>Stashbot 342,703 bytes +17,347 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 0e7079d: Install DiscussionTools on all wikis (attempt 2) (T252264; T253943) (duration: 00m 56s)
  • curprev 01:4301:43, 17 June 2020imported>Stashbot 325,356 bytes +20,465 shdubsh: restart elasticsearch on logstash1011

16 June 2020

  • curprev 00:2800:28, 16 June 2020imported>Stashbot 304,891 bytes +12,502 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: limit HTTP client timeout T245170 (duration: 00m 56s)

14 June 2020

  • curprev 13:5113:51, 14 June 2020imported>Stashbot 292,389 bytes +113 qchris: Disabling puppet on gerrit1002 (test instance) to do some more upgrade testing

13 June 2020

  • curprev 21:1221:12, 13 June 2020imported>Stashbot 292,276 bytes +335 qchris: Enabling puppet on gerrit1002 (test instance). Done with testing for today.

12 June 2020

  • curprev 17:4417:44, 12 June 2020imported>Stashbot 291,941 bytes +6,714 herron: restarting logstash1011 elasticsearch instance
  • curprev 01:0201:02, 12 June 2020imported>Stashbot 285,227 bytes +14,139 ejegg: updated payments-wiki from aceddff8b5 to 5fd4eb1519

10 June 2020

  • curprev 23:5523:55, 10 June 2020imported>Stashbot 271,088 bytes +14,140 catrope@deploy1001: Synchronized php-1.35.0-wmf.36/includes/skins/SkinTemplate.php: T255073 (duration: 01m 07s)

9 June 2020

  • curprev 23:1823:18, 9 June 2020imported>Stashbot 256,948 bytes +12,465 Reedy: run namespaceDupes.php --fix for hiwikibooks T254012
  • curprev 00:3900:39, 9 June 2020imported>Stashbot 244,483 bytes +10,746 ejegg: updated payments-wiki from c1d14a5db7 to aceddff8b5

5 June 2020

  • curprev 16:4516:45, 5 June 2020imported>Stashbot 233,737 bytes +4,799 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0 (duration: 00m 11s)

4 June 2020

  • curprev 23:4523:45, 4 June 2020imported>Stashbot 228,938 bytes +11,913 catrope@deploy1001: Synchronized wmf-config/mc.php: Set coalesceKeys=non-global for WANCache on enwiki (duration: 00m 59s)
  • curprev 00:3600:36, 4 June 2020imported>Stashbot 217,025 bytes +14,843 reedy@deploy1001: Synchronized php-1.35.0-wmf.35/includes/specials/SpecialUserrights.php: T254417 T251534 (duration: 01m 06s)

3 June 2020

  • curprev 01:0001:00, 3 June 2020imported>Stashbot 202,182 bytes +10,631 reedy@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: wgLocalVirtualHosts (duration: 01m 06s)

1 June 2020

  • curprev 20:1420:14, 1 June 2020imported>Stashbot 191,551 bytes +5,297 shdubsh: downgrade mtail to rc5 in ulsfo -- T254192

31 May 2020

  • curprev 09:5609:56, 31 May 2020imported>Stashbot 186,254 bytes +199 Urbanecm: mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Vox Golf' 'Colonel Chicken' (T254068)

29 May 2020

  • curprev 22:3222:32, 29 May 2020imported>Stashbot 186,055 bytes +4,498 bstorm_: updated views on labsdb1010 T252219

28 May 2020

  • curprev 23:4823:48, 28 May 2020imported>Stashbot 181,557 bytes +7,100 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/skins/Vector/resources/skins.vector.styles/Menu.less: T253912 Hotfix: Cannot rename emptyPortlet to empty-portlet yet (duration: 00m 59s)

27 May 2020

  • curprev 23:2023:20, 27 May 2020imported>Stashbot 174,457 bytes +16,759 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add autoreviewrestore right to rollbacker group on hiwiki (T252986) (duration: 01m 05s)

26 May 2020

  • curprev 21:3421:34, 26 May 2020imported>Stashbot 157,698 bytes +8,943 krinkle@deploy1001: Synchronized wmf-config/mc.php: I0fb124b3593 (duration: 01m 05s)

25 May 2020

  • curprev 23:3423:34, 25 May 2020imported>Stashbot 148,755 bytes +7,068 ejegg: re-enabled fundraising queue consumers and job runners, except audits, dedupe, and recurring

24 May 2020

  • curprev 17:3617:36, 24 May 2020imported>Stashbot 141,687 bytes +170 gehel: restarting elasticsearch psi on elastic1052

23 May 2020

  • curprev 19:0419:04, 23 May 2020imported>Stashbot 141,517 bytes +906 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 06s)

22 May 2020

  • curprev 22:4222:42, 22 May 2020imported>Stashbot 140,611 bytes +8,134 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/: Ie19613ef7643a (duration: 01m 06s)

21 May 2020

  • curprev 23:5823:58, 21 May 2020imported>Stashbot 132,477 bytes +8,419 ejegg: updated civicrm from b658fd8233 to 6b1d5902dd

20 May 2020

  • curprev 20:1620:16, 20 May 2020imported>Stashbot 124,058 bytes +10,406 herron: logstash1011:~# kafka-preferred-replica-election --zookeeper conf1004.eqiad.wmnet,conf1005.eqiad.wmnet,conf1006.eqiad.wmnet/kafka/logging-eqiad
  • curprev 00:0500:05, 20 May 2020imported>Stashbot 113,652 bytes +7,654 RoanKattouw: Ran namespaceDupes.php on tiwiki and tiwiktionary for T251287

18 May 2020

  • curprev 23:5023:50, 18 May 2020imported>Stashbot 105,998 bytes +9,614 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

16 May 2020

  • curprev 22:0422:04, 16 May 2020imported>Stashbot 96,384 bytes +1,903 Krinkle: krinkle@mc1022 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
  • curprev 00:5500:55, 16 May 2020imported>Stashbot 94,481 bytes +6,235 krinkle@deploy1001: Synchronized wmf-config/logging.php: I046868190b472 (duration: 01m 13s)

14 May 2020

  • curprev 23:2423:24, 14 May 2020imported>Stashbot 88,246 bytes +6,005 catrope@deploy1001: Synchronized static/images/project-logos/: Revert temporary 20k logo for vecwiki (T252770) (duration: 01m 06s)
  • curprev 00:5400:54, 14 May 2020imported>Stashbot 82,241 bytes +9,400 foks: change password for "Python eggs"

13 May 2020

  • curprev 00:4300:43, 13 May 2020imported>Stashbot 72,841 bytes +10,801 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer

12 May 2020

  • curprev 00:3700:37, 12 May 2020imported>Stashbot 62,040 bytes +11,598 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

10 May 2020

  • curprev 12:1812:18, 10 May 2020imported>Stashbot 50,442 bytes +926 marostegui: Start event scheduler on db1115 after a massive delete - T252324

8 May 2020

  • curprev 21:4521:45, 8 May 2020imported>Stashbot 49,516 bytes +4,853 bstorm_: cleaned up wb_terms_no_longer_updated view for testwikidatawiki and testcommonswiki on labsdb1010 T251598
  • curprev 00:3000:30, 8 May 2020imported>Stashbot 44,663 bytes +11,109 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert all wikis except test to 1.35.0-wmf.30 for T252179

7 May 2020

  • curprev 00:1200:12, 7 May 2020imported>Stashbot 33,554 bytes +5,907 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

5 May 2020

  • curprev 23:4423:44, 5 May 2020imported>Stashbot 27,647 bytes +10,916 catrope@deploy1001: Synchronized wmf-config/flaggedrevs.php: Restore the reviewer group on fawiki (T249643) (duration: 01m 06s)

4 May 2020

  • curprev 23:3823:38, 4 May 2020imported>Stashbot 16,731 bytes +12,007 mstyles@deploy1001: Finished deploy [wdqs/wdqs@6518a8d]: v.0.3.26 (duration: 14m 39s)

3 May 2020

  • curprev 22:5222:52, 3 May 2020imported>Stashbot 4,724 bytes +491 Krinkle: scap pull mwmaint1002 and mw2001 for noc.wm.o. – https://gerrit.wikimedia.org/r/593929

2 May 2020

  • curprev 07:4907:49, 2 May 2020imported>Stashbot 4,233 bytes +460 oblivian@cumin1001: conftool action : set/pooled=yes; selector: name=mw13(49|5[0-9]|6[0-2])\.eqiad\.wmnet

1 May 2020

  • curprev 19:5619:56, 1 May 2020imported>Stashbot 3,773 bytes −815,836 rzl@cumin1001: conftool action : set/pooled=no; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet
  • curprev 01:3601:36, 1 May 2020imported>Stashbot 819,609 bytes +7,834 bmansurov@deploy1001: Finished deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service (duration: 04m 33s)

30 April 2020

29 April 2020

  • curprev 00:5600:56, 29 April 2020imported>Stashbot 802,927 bytes +15,206 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Growth Study QuickSurvey on enwiki (with sample size 0, for testing) (T248421) (duration: 01m 10s)

27 April 2020

  • curprev 23:2523:25, 27 April 2020imported>Stashbot 787,721 bytes +15,220 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Update logos for tiwiki and tiwiktionary (T150618, T249451) (duration: 00m 57s)

26 April 2020

  • curprev 18:0818:08, 26 April 2020imported>Stashbot 772,501 bytes +140 elukey: powercycle puppetmaster1001 - mgmt serial console not usable, no ssh, racadm getsel doesn't show anything

25 April 2020

  • curprev 10:2310:23, 25 April 2020imported>Stashbot 772,361 bytes +271 addshore: going to restart and probably depool for a short time wdqs1005 as it is in a deadlock T242453

24 April 2020

  • curprev 21:2521:25, 24 April 2020imported>Stashbot 772,090 bytes +562 cdanis@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=eqiad

23 April 2020

  • curprev 22:0622:06, 23 April 2020imported>Stashbot 771,528 bytes +300 Urbanecm: Perform timeouting rename at enwiki Wikipedia talk:Introduction --> Wikipedia talk:Introduction (historical) using moveBatch.php (request)

22 April 2020

  • curprev 08:5508:55, 22 April 2020imported>Stashbot 771,228 bytes +1,318 Urbanecm: Move User:Wikipedia:Introduction (historical) --> Wikipedia:Introduction (historical) at enwiki using moveBatch.php, on-wiki interface was time-outing
  • curprev 01:4601:46, 22 April 2020imported>Stashbot 769,910 bytes +5,717 milimetric@deploy1001: Started deploy [analytics/refinery@30facc4]: Analytics: another follow-up on the train, jar version bump

20 April 2020

  • curprev 23:2923:29, 20 April 2020imported>Stashbot 764,193 bytes +9,549 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Update project wordmarks and icons (T249047) (duration: 01m 01s)

19 April 2020

  • curprev 16:1916:19, 19 April 2020imported>Stashbot 754,644 bytes +302 reedy@deploy1001: Synchronized wmf-config/LabsServices.php: labs: Move RB traffic to new stretch host (duration: 01m 11s)

18 April 2020

17 April 2020

  • curprev 19:3319:33, 17 April 2020imported>Stashbot 753,607 bytes +5,333 Krinkle: Depool mw1407.eqiad.wmnet for opcache testing. Do not repool without first reverting https://gerrit.wikimedia.org/r/589674.

16 April 2020

  • curprev 22:3422:34, 16 April 2020imported>Stashbot 748,274 bytes +7,894 maryum: reindexing wikis that failed from previous reindex on mwmain1002

15 April 2020

  • curprev 22:1122:11, 15 April 2020imported>Stashbot 740,380 bytes +9,855 mholloway-shell@deploy1001: Synchronized php-1.35.0-wmf.28/extensions/MachineVision: Fix: Initialize categories array for initial images (T250321) (duration: 01m 07s)

14 April 2020

  • curprev 23:2423:24, 14 April 2020imported>Stashbot 730,525 bytes +9,174 AndyRussG: re-enabled thank-you, onimailing and new recurring charge jobs

13 April 2020

  • curprev 23:2423:24, 13 April 2020imported>Stashbot 721,351 bytes +6,995 mdholloway: re-ran extensions/MachineVision/maintenance/withholdImages.php on commonswiki

12 April 2020

  • curprev 11:1111:11, 12 April 2020imported>Stashbot 714,356 bytes +810 vgutierrez: restart ats-tls on cp5008.eqsin.wmnet - T249335

11 April 2020

  • curprev 19:5219:52, 11 April 2020imported>Stashbot 713,546 bytes +836 cdanis@cumin1001: dbctl commit (dc=all): 'slight deweight to db1111', diff saved to https://phabricator.wikimedia.org/P10960 and previous config saved to /var/cache/conftool/dbconfig/20200411-195235-cdanis.json

10 April 2020

  • curprev 21:1221:12, 10 April 2020imported>Stashbot 712,710 bytes +3,913 cdanis@cumin1001: dbctl commit (dc=all): 'db1111 seems overloaded', diff saved to https://phabricator.wikimedia.org/P10954 and previous config saved to /var/cache/conftool/dbconfig/20200410-211202-cdanis.json

9 April 2020

  • curprev 23:4423:44, 9 April 2020imported>Stashbot 708,797 bytes +5,617 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 58s)

8 April 2020

  • curprev 21:2021:20, 8 April 2020imported>Stashbot 703,180 bytes +8,049 jforrester@deploy1001: Synchronized php-1.35.0-wmf.27/extensions/TemplateData/includes/TemplateDataHooks.php: Restore call to OutputPage::setupOOUI() (duration: 01m 07s)

7 April 2020

  • curprev 20:3920:39, 7 April 2020imported>Stashbot 695,131 bytes +13,653 andrewbogott: correction: briefly downtiming ldap-eqiad-replica0 and ldap-eqiad-replica1. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes
  • curprev 00:0100:01, 7 April 2020imported>Stashbot 681,478 bytes +13,583 addshore@deploy1001: Synchronized php-1.35.0-wmf.26/extensions/Wikibase/repo/includes/Store/Sql/DatabaseSchemaUpdater.php: Do not try to drop things when theres no wb_terms table T208425 T249565 cache bust (duration: 01m 01s)

3 April 2020

  • curprev 21:1721:17, 3 April 2020imported>Stashbot 667,895 bytes +4,748 andrewbogott: ugpraded wikitech-static to 1.34.1
  • curprev 00:4200:42, 3 April 2020imported>Stashbot 663,147 bytes +9,745 catrope@deploy1001: Synchronized php-1.35.0-wmf.26/extensions/FlaggedRevs/: Fix logic for determining if pending edits were null (T249277) (duration: 01m 00s)

1 April 2020

  • curprev 22:4422:44, 1 April 2020imported>Stashbot 653,402 bytes +9,945 volker-e@deploy1001: Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s)
  • curprev 00:3900:39, 1 April 2020imported>Stashbot 643,457 bytes +11,449 reedy@deploy1001: Synchronized docroot/mediawiki.org/xml/: Update http and prot rel links to https, fix link to sitelist in MW Core (duration: 01m 06s)

31 March 2020

  • curprev 00:2600:26, 31 March 2020imported>Stashbot 632,008 bytes +8,786 eileen: civicrm revision changed from cf2e2c11c3 to 524b162174, config revision is 708198a154

29 March 2020

  • curprev 08:2408:24, 29 March 2020imported>Stashbot 623,222 bytes +162 elukey: powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific

28 March 2020

27 March 2020

  • curprev 20:5120:51, 27 March 2020imported>Stashbot 622,744 bytes +6,524 ejegg: updated payments-wiki from db618f429d to 1640f5e21e

26 March 2020

  • curprev 23:0723:07, 26 March 2020imported>Stashbot 616,220 bytes +12,328 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: ce63a4e: Enable wmgUseFooterContactLink for cswiki (T248584; take II) (duration: 00m 57s)
  • curprev 00:0600:06, 26 March 2020imported>Stashbot 603,892 bytes +15,568 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Special:Investigate on testwiki (T247645) (duration: 03m 14s)

24 March 2020

23 March 2020

  • curprev 21:5021:50, 23 March 2020imported>Stashbot 582,770 bytes +7,770 krinkle@deploy1001: Synchronized docroot/noc/css/vector.css: I627a0ddba5 (duration: 01m 02s)

22 March 2020

  • curprev 23:1923:19, 22 March 2020imported>Stashbot 575,000 bytes +225 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: T248274 (duration: 01m 19s)

20 March 2020

  • curprev 23:1623:16, 20 March 2020imported>Stashbot 574,775 bytes +5,506 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

19 March 2020

  • curprev 22:1522:15, 19 March 2020imported>Stashbot 569,269 bytes +4,690 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@794f099]: Update mobileapps to 99869f45 (duration: 05m 13s)
  • curprev 00:3900:39, 19 March 2020imported>Stashbot 564,579 bytes +8,440 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikiws to 1.35.0-wmf.24 refs T233872

18 March 2020

  • curprev 00:3100:31, 18 March 2020imported>Stashbot 556,139 bytes +12,787 Amir1: foreachwikiindblist medium deleteEqualMessages.php --delete (T247562)

17 March 2020

  • curprev 00:5700:57, 17 March 2020imported>Stashbot 543,352 bytes +9,300 krinkle@deploy1001: Synchronized php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Formatters/: Ic77b2c6b33a, T247458 (duration: 01m 12s)

15 March 2020

14 March 2020

  • curprev 08:3308:33, 14 March 2020imported>Stashbot 533,856 bytes +358 elukey: run kafka preferred-replica-election on kafka-jumbo1001 - T247561

13 March 2020

  • curprev 23:1223:12, 13 March 2020imported>Stashbot 533,498 bytes +3,031 bstorm_: rebooting labstore1006 for upgrade to stretch T224583
  • curprev 00:2000:20, 13 March 2020imported>Stashbot 530,467 bytes +13,395 shdubsh: reload prometheus@ops on prometheus1003

12 March 2020

  • curprev 00:5600:56, 12 March 2020imported>Stashbot 517,072 bytes +10,878 ebernhardson@deploy1001: Synchronized php-1.35.0-wmf.22/extensions/CirrusSearch/includes/Maintenance/Reindexer.php: wait around for counts to match up in reindexer before giving up (duration: 01m 08s)

11 March 2020

  • curprev 00:2400:24, 11 March 2020imported>Stashbot 506,194 bytes +18,985 eileen: civicrm revision changed from 35651da117 to c4b81b19b0, config revision is 71c8cda115

9 March 2020

  • curprev 23:2123:21, 9 March 2020imported>Stashbot 487,209 bytes +7,801 catrope@deploy1001: Synchronized wmf-config/throttle.php: Remove expired throttle exemptions (duration: 01m 00s)

8 March 2020

  • curprev 17:5817:58, 8 March 2020imported>Stashbot 479,408 bytes +406 elukey: restart hadoop-yarn-nodemanger on an-worker1087

7 March 2020

  • curprev 12:4812:48, 7 March 2020imported>Stashbot 479,002 bytes +109 reedy@deploy1001: Synchronized wmf-config/throttle.php: T247149 (duration: 01m 07s)
  • curprev 01:3501:35, 7 March 2020imported>Stashbot 478,893 bytes +10,335 reedy@deploy1001: Synchronized wmf-config/throttle.php: tidy up (duration: 00m 56s)

6 March 2020

  • curprev 00:5800:58, 6 March 2020imported>Stashbot 468,558 bytes +21,223 dzahn@cumin1001: START - Cookbook sre.hosts.downtime

5 March 2020

  • curprev 00:5700:57, 5 March 2020imported>Stashbot 447,335 bytes +13,946 twentyafterfour: deploying phabricator-extensions tag release/2020-03-04/1 ( https://phabricator.wikimedia.org/source/phab-extensions/history/wmf%252Fstable/;release/2020-03-04/1 )

4 March 2020

  • curprev 00:2500:25, 4 March 2020imported>Stashbot 433,389 bytes +23,744 ebernhardson@deploy1001: Synchronized php-1.35.0-wmf.21/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: [cirrus] Match fallback config key with the one used in cirrus (duration: 01m 03s)

3 March 2020

  • curprev 00:5300:53, 3 March 2020imported>Stashbot 409,645 bytes +17,902 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T240055 Update special scandium configuration to load from /srv/parsoid-testing (duration: 00m 58s)

1 March 2020

  • curprev 17:5417:54, 1 March 2020imported>Stashbot 391,743 bytes +926 marostegui: Start replication on db1111 new host on s8 - T246447

29 February 2020

  • curprev 12:3712:37, 29 February 2020imported>Stashbot 390,817 bytes +350 reedy@deploy1001: Synchronized wmf-config/config/viwiki.yaml: T246511 (duration: 00m 56s)

28 February 2020

  • curprev 21:3121:31, 28 February 2020imported>Stashbot 390,467 bytes +6,686 mutante: using planet1001 to manually hack APT sources to test new apt1001.wikimedia.org

27 February 2020

26 February 2020

  • curprev 00:4800:48, 26 February 2020imported>Stashbot 354,983 bytes +15,242 eileen: civicrm revision changed from bec2d6ad9f to 62e62e107c, config revision is c0ef31e2fd

25 February 2020

  • curprev 00:5900:59, 25 February 2020imported>Stashbot 339,741 bytes +14,768 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T238803: Drop ability to load SkinPerPage, EUCopyrightCampaign, and EUCopyrightCampaignSkin (duration: 00m 56s)

23 February 2020

22 February 2020

  • curprev 03:4103:41, 22 February 2020imported>Stashbot 324,738 bytes +596 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • curprev 00:4500:45, 22 February 2020imported>Stashbot 324,142 bytes +6,187 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

20 February 2020

19 February 2020

  • curprev 00:4900:49, 19 February 2020imported>Stashbot 291,837 bytes +16,528 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

17 February 2020

15 February 2020

  • curprev 01:0101:01, 15 February 2020imported>Stashbot 273,144 bytes +176 cdanis: ✔️ cdanis@an-coord1001.eqiad.wmnet ~ 🕗🍺 sudo systemctl restart hive-server2.service ; sudo systemctl restart hive-metastore.service

14 February 2020

  • curprev 23:4223:42, 14 February 2020imported>Stashbot 272,968 bytes +4,182 XenoRyet: updated civicrm from cf86495d44 to 8c77e9e915
  • curprev 00:5300:53, 14 February 2020imported>Stashbot 268,786 bytes +18,037 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T245202 cirrus: Move all move_like traffic to codfw (duration: 01m 02s)

13 February 2020

  • curprev 00:5800:58, 13 February 2020imported>Stashbot 250,749 bytes +16,260 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

12 February 2020

  • curprev 00:4400:44, 12 February 2020imported>Stashbot 234,489 bytes +182,486 jforrester@deploy1001: Synchronized php-1.35.0-wmf.19/includes/page/ImageHistoryPseudoPager.php: T244937 ImageHistoryPseudoPager: Update doQuery() for IndexPager changes (duration: 01m 03s)

11 February 2020

  • curprev 00:2600:26, 11 February 2020imported>Stashbot 52,003 bytes +14,854 ebernhardson@deploy1001: Synchronized php-1.35.0-wmf.18/skins/MinervaNeue: SWAT: Revert: Reduce userContributions icon code (duration: 01m 06s)

9 February 2020

8 February 2020

  • curprev 19:1219:12, 8 February 2020imported>Stashbot 37,073 bytes +318 _joe_: set cpufreq governor to performance on mw1328
  • curprev 00:0500:05, 8 February 2020imported>Stashbot 36,755 bytes +12,166 Jeff_Green: switched payments.wikimedia.org to codfw datacenter due to T244610

7 February 2020

  • curprev 00:1300:13, 7 February 2020imported>Stashbot 24,589 bytes +17,865 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

5 February 2020

  • curprev 23:3023:30, 5 February 2020imported>Stashbot 6,724 bytes −158,385 ebernhardson: delete search indices duplicated on multiple clusters for: hywwiki, chrwiktionary, gcrwiki, mnwwiki, noboard_chapterswikimedia nqowiki nrmwiki outreachwiki and srnwiki

4 February 2020

  • curprev 22:3522:35, 4 February 2020imported>Stashbot 165,109 bytes +4,681 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.18 refs T233866
  • curprev 00:1400:14, 4 February 2020imported>Stashbot 160,428 bytes +9,252 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [enwiki] Add Commons as an import source T242884 (duration: 00m 57s)

2 February 2020

1 February 2020

31 January 2020

  • curprev 01:0901:09, 31 January 2020imported>Stashbot 149,050 bytes +2,034 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@322ee4c]: Update mobileapps to 3eec28d (duration: 06m 53s)

29 January 2020

27 January 2020

  • curprev 23:4023:40, 27 January 2020imported>Stashbot 144,743 bytes +1,366 eileen: civicrm revision changed from fbd5c35fb0 to ac730a6bcb, config revision is 837b9d0703

26 January 2020

24 January 2020

  • curprev 22:3122:31, 24 January 2020imported>Stashbot 141,377 bytes +2,319 mutante: ganeti1003 - sudo gnt-instance remove etherpad1001.eqiad.wmnet (T224580)
  • curprev 00:4600:46, 24 January 2020imported>Stashbot 139,058 bytes +9,927 mutante: cp4032 - starting varnishmtail.service

23 January 2020

  • curprev 01:4001:40, 23 January 2020imported>Stashbot 129,131 bytes +8,387 jforrester@deploy1001: Synchronized php-1.35.0-wmf.16/extensions/AbuseFilter/includes/AFComputedVariable.php: T243469 When no registration date is recorded, use 2008-01-15 (duration: 01m 08s)

22 January 2020

  • curprev 01:1301:13, 22 January 2020imported>Stashbot 120,744 bytes +8,885 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: resync, the last sync only took on half the appservers (duration: 01m 05s)

20 January 2020

19 January 2020

  • curprev 12:0212:02, 19 January 2020imported>Stashbot 106,204 bytes +350 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2085:3311, db2085:3318 T243148', diff saved to https://phabricator.wikimedia.org/P10210 and previous config saved to /var/cache/conftool/dbconfig/20200119-120236-marostegui.json
  • curprev 00:4600:46, 19 January 2020imported>Stashbot 105,854 bytes +429 cdanis: T238305 cp3053.mgmt /admin1-> racadm serveraction hardreset

17 January 2020

  • curprev 21:5621:56, 17 January 2020imported>Stashbot 105,425 bytes +5,582 urandom: bootstrapping restbase2023-c — T243000
  • curprev 00:4500:45, 17 January 2020imported>Stashbot 99,843 bytes +13,373 mutante: urldownloaders - rm /etc/logrotate.d/squid3 ; systemctl start logrotate (this fixes failed logrotate because of squid3 vs squid file = duplicate entry, but puppet will recreate it)

16 January 2020

  • curprev 00:4000:40, 16 January 2020imported>Stashbot 86,470 bytes +11,747 bstorm_: set max_connections on db1133 (m5-master) back to 500 since the neutron connections seem fairly stable now T242817

15 January 2020

  • curprev 00:2200:22, 15 January 2020imported>Stashbot 74,723 bytes +6,560 bstorm_: restarted maintain-dbusers on labstore1004 after recovering the m5 DB's connection issue

14 January 2020

  • curprev 01:0301:03, 14 January 2020imported>Stashbot 68,163 bytes +7,915 catrope@deploy1001: Synchronized php-1.35.0-wmf.14/extensions/GrowthExperiments/: Various topic search-related cherry-picks (duration: 00m 57s)

13 January 2020

  • curprev 00:5800:58, 13 January 2020imported>Stashbot 60,248 bytes +1,047 jiji@cumin1001: conftool action : set/pooled=yes; selector: name=cp3061.esams.wmnet

11 January 2020

  • curprev 05:3405:34, 11 January 2020imported>Stashbot 59,201 bytes +242 volker-e@deploy1001: Finished deploy [design/style-guide@6a44c69]: Deploy design/style-guide: (duration: 00m 08s)

10 January 2020

  • curprev 22:3322:33, 10 January 2020imported>Stashbot 58,959 bytes +3,964 mutante: ms-be1026 sudo systemctl reset-failed (failed Session 372989 of user debmonitor)
  • curprev 00:4500:45, 10 January 2020imported>Stashbot 54,995 bytes +8,040 catrope@deploy1001: Synchronized php-1.35.0-wmf.14/extensions/GrowthExperiments/: Expose tasktype/topic API parameter info (T240512) (duration: 01m 01s)

9 January 2020

  • curprev 00:3300:33, 9 January 2020imported>Stashbot 46,955 bytes +13,245 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no-op) set config page for newcomer tasks (T233465) (duration: 01m 05s)

8 January 2020

  • curprev 00:3800:38, 8 January 2020imported>Stashbot 33,710 bytes +15,858 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@024488f]: airflow: set mjolnir dag start date to today (20200108) (duration: 00m 42s)

7 January 2020

  • curprev 00:5900:59, 7 January 2020imported>Stashbot 17,852 bytes +9,482 mutante: ganeti - creating urldownloader1002.wikimedia.org in eqiad_C with 1 CPU, 1 GB RAM, 10 GB disk, public IP (T241979)

6 January 2020

4 January 2020

  • curprev 16:3416:34, 4 January 2020imported>Stashbot 7,863 bytes +313 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

3 January 2020

  • curprev 22:1422:14, 3 January 2020imported>Stashbot 7,550 bytes +3,165 volker-e@deploy1001: Finished deploy [design/style-guide@8054026]: Deploy design/style-guide: (duration: 00m 08s)

2 January 2020

1 January 2020

30 December 2019

  • curprev 16:0016:00, 30 December 2019imported>Stashbot 868,829 bytes +235 ema: cp1083: restart ats-tls and varnish-fe after crashes - T241593

29 December 2019

28 December 2019

  • curprev 22:5422:54, 28 December 2019imported>Stashbot 868,069 bytes +602 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: cirrus: re-sync move more_like traffic to codfw (duration: 00m 54s)

27 December 2019

25 December 2019

  • curprev 10:1210:12, 25 December 2019imported>Stashbot 866,771 bytes +312 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

24 December 2019

  • curprev 15:3315:33, 24 December 2019imported>Stashbot 866,459 bytes +1,211 gehel@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 00:1300:13, 24 December 2019imported>Stashbot 865,248 bytes +2,405 volker-e@deploy1001: Finished deploy [design/style-guide@efc240b]: Deploy design/style-guide: (duration: 00m 07s)

22 December 2019

  • curprev 16:4816:48, 22 December 2019imported>Stashbot 862,843 bytes +468 bd808: Mastodon feed moved to https://mastodon.social/@wikimedia_sal (T52109)
  • curprev 00:1000:10, 22 December 2019imported>Stashbot 862,375 bytes +612 bd808: Wikimedia SAL messages now available on Mastodon! https://fosstodon.org/@wikimedia_sal

21 December 2019

  • curprev 01:4401:44, 21 December 2019imported>Stashbot 861,763 bytes +8,163 reedy@deploy1001: update-interwiki-cache aborted: Update interwiki cache (duration: 00m 07s)

20 December 2019

  • curprev 00:5200:52, 20 December 2019imported>Stashbot 853,600 bytes +18,576 mholloway-shell@deploy1001: Synchronized php-1.35.0-wmf.11/extensions/MachineVision: Fix: Allow a single period in $basePath in maintenance scripts (duration: 00m 54s)

19 December 2019

  • curprev 01:4401:44, 19 December 2019imported>Stashbot 835,024 bytes +20,843 twentyafterfour: deploying phabricator update (tagged release/2019-12-19/1)

18 December 2019

  • curprev 00:2200:22, 18 December 2019imported>Stashbot 814,181 bytes +17,525 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Glent M0 for dewiki, enwiki and frwiki - T237365 (duration: 01m 02s)

16 December 2019

  • curprev 23:2823:28, 16 December 2019imported>Stashbot 796,656 bytes +15,656 ejegg: updated fundraising civicrm from 505c653da1 to b2d0b5d66d

14 December 2019

13 December 2019

  • curprev 22:3522:35, 13 December 2019imported>Stashbot 780,908 bytes +3,189 cdanis: T229686 ✔️ cdanis@mwdebug1001.eqiad.wmnet /srv/mediawiki 🕠🍺 scap pull
  • curprev 00:3200:32, 13 December 2019imported>Stashbot 777,719 bytes +8,893 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Begin "initiation test" for suggested edits (T238888) (duration: 00m 55s)

12 December 2019

  • curprev 01:4201:42, 12 December 2019imported>Stashbot 768,826 bytes +11,852 volker-e@deploy1001: Finished deploy [design/style-guide@481eaf6]: Deploy design/style-guide: (duration: 00m 07s)

11 December 2019

9 December 2019

8 December 2019

  • curprev 20:4020:40, 8 December 2019imported>Stashbot 741,665 bytes +605 ejegg: disabled Ingenico Connect recurring charge job
  • curprev 01:4701:47, 8 December 2019imported>Stashbot 741,060 bytes +2,785 andrew@deploy1001: Started deploy [horizon/deploy@accbbd1]: (no justification provided)

7 December 2019

  • curprev 00:5900:59, 7 December 2019imported>Stashbot 738,275 bytes +8,615 andrew@deploy1001: Started deploy [horizon/deploy@0f70602]: (no justification provided)

6 December 2019

  • curprev 00:4700:47, 6 December 2019imported>Stashbot 729,660 bytes +15,748 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Use PHP serialization with HMAC for Kask session serialization (T222099) (duration: 01m 01s)

5 December 2019

  • curprev 00:4900:49, 5 December 2019imported>Stashbot 713,912 bytes +15,602 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=phab1001-vcs.eqiad.wmnet

4 December 2019

  • curprev 00:4900:49, 4 December 2019imported>Stashbot 698,310 bytes +15,609 ejegg: changed donations queue consumer and thank you mailer to use 3 minute cycles

3 December 2019

  • curprev 01:4201:42, 3 December 2019imported>Stashbot 682,701 bytes +128,943 dzahn@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
  • curprev 01:0001:00, 3 December 2019imported>Stashbot 681,404 bytes +127,646 James_F: mwscript sql.php --wiki=sewikimedia php-1.35.0-wmf.8/extensions/Translate/sql/translate_{…}.sql T239091

19 November 2019

18 November 2019

  • curprev 00:4400:44, 18 November 2019imported>Stashbot 545,042 bytes +199 tstarling@deploy1001: Synchronized php-1.35.0-wmf.5/includes/Rest/Handler/PageHistoryCountHandler.php: fix extremely slow query T238378 (duration: 00m 59s)

16 November 2019

  • curprev 20:2720:27, 16 November 2019imported>Stashbot 544,843 bytes +277 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

15 November 2019

  • curprev 22:1422:14, 15 November 2019imported>Stashbot 544,566 bytes +7,139 jeh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

14 November 2019

  • curprev 23:0323:03, 14 November 2019imported>Stashbot 537,427 bytes +12,247 mutante: restarting gerrit to ncrease defaultThreadPoolSize to 2
  • curprev 00:4100:41, 14 November 2019imported>Stashbot 525,180 bytes +9,904 ebernhardson: T237849 Start CirrusSearch forceSearchIndex.php commonswiki 2019-10-20T00:00:00 - 2019-11-14T01:00:00 pushing into jobqueue

12 November 2019

  • curprev 23:5723:57, 12 November 2019imported>Stashbot 515,276 bytes +11,275 mholloway-shell@deploy1001: Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Fix: Do not return after inserting a single suggestion (duration: 00m 52s)

11 November 2019

  • curprev 22:5122:51, 11 November 2019imported>Stashbot 504,001 bytes +3,806 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3065.esams.wmnet

9 November 2019

  • curprev 20:2520:25, 9 November 2019imported>Stashbot 500,195 bytes +323 reedy@deploy1001: Synchronized langlist-labs: T237823 (duration: 00m 54s)
  • curprev 01:0701:07, 9 November 2019imported>Stashbot 499,872 bytes +7,974 volker-e@deploy1001: Finished deploy [design/style-guide@ef82b69]: Deploy design/style-guide: (duration: 00m 07s)

8 November 2019

  • curprev 01:2301:23, 8 November 2019imported>Stashbot 491,898 bytes +10,234 reedy@deploy1001: Synchronized php-1.35.0-wmf.5/extensions/GlobalBlocking/: Prevent some extra db queries (duration: 00m 53s)

7 November 2019

  • curprev 00:2100:21, 7 November 2019imported>Stashbot 481,664 bytes +9,074 XioNoX: enable interface damping on primary eqsin-codfw link - T236878

6 November 2019

  • curprev 00:0700:07, 6 November 2019imported>Stashbot 472,590 bytes +5,234 mdholloway: created table wikimedia_editor_tasks_edit_streak on x1/wikishared (T234956)

5 November 2019

  • curprev 00:1500:15, 5 November 2019imported>Stashbot 467,356 bytes +6,242 mutante: gerrit - restarting service to re-enable jgit gc (T217497)

3 November 2019

  • curprev 03:5403:54, 3 November 2019imported>Stashbot 461,114 bytes +719 andrew@deploy1001: Finished deploy [horizon/deploy@0c024d4]: one more prefix fix (duration: 03m 35s)

2 November 2019

  • curprev 00:5800:58, 2 November 2019imported>Stashbot 460,395 bytes +3,957 mutante: gerrit-replica - created missing /var/lib/gerrit2/review_site/tmp and restarted service - service back up on buster (T176774)

1 November 2019

  • curprev 01:2101:21, 1 November 2019imported>Stashbot 456,438 bytes +10,859 jforrester@deploy1001: Synchronized php-1.35.0-wmf.4/resources/src/mediawiki.widgets/mw.widgets.UsersMultiselectWidget.js: T236460 mw.widgets.UsersMultiselectWidget: Fix property name (duration: 00m 54s)

31 October 2019

  • curprev 00:4100:41, 31 October 2019imported>Stashbot 445,579 bytes +20,560 jforrester@deploy1001: Synchronized php-1.35.0-wmf.4/extensions/WikiLove/resources/ext.wikiLove.icon.vector.css: T236958 Fix Vector icon after upstream change (duration: 01m 02s)

28 October 2019

  • curprev 23:2323:23, 28 October 2019imported>Stashbot 425,019 bytes +8,544 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy Echo kask migration to officewiki for testing, part 3 (T222851) (duration: 00m 52s)

26 October 2019

25 October 2019

  • curprev 22:5522:55, 25 October 2019imported>Stashbot 416,347 bytes +9,927 mutante: moscovium rm /dev/shm/envoy_shared_memory_0 to revive envoy which failed to run after changing ports and reinstalling it (T180641)
  • curprev 00:5200:52, 25 October 2019imported>Stashbot 406,420 bytes +18,865 krinkle@deploy1001: Synchronized php-1.35.0-wmf.3/extensions/LiquidThreads/classes/View.php: (no justification provided) (duration: 00m 54s)

24 October 2019

  • curprev 00:0300:03, 24 October 2019imported>Stashbot 387,555 bytes +11,289 mutante: restarting gerrit to increase heap_size from 20G to 32G (T225166 T222391)

22 October 2019

  • curprev 23:5723:57, 22 October 2019imported>Stashbot 376,266 bytes +12,691 maxsem@deploy1001: Synchronized php-1.35.0-wmf.3/includes/block/DatabaseBlock.php: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/545373/ (duration: 00m 59s)

21 October 2019

  • curprev 23:1523:15, 21 October 2019imported>Stashbot 363,575 bytes +15,479 thcipriani: ops/puppet:sudo -u gerrit2 git update-ref refs/changes/66/535966/meta d6909e0537b7fe7ca6fb8989e22ed5e1c1e46e89 && sudo -u gerrit2 git update-ref refs/changes/66/535966/meta 8494c28eee163aaa48f6274e899e0e79f3a757f3 on gerrit1001

19 October 2019

  • curprev 08:4108:41, 19 October 2019imported>Stashbot 348,096 bytes +48 XioNoX: add user papaul to fasw-c-eqiad
  • curprev 00:0500:05, 19 October 2019imported>Stashbot 348,048 bytes +12,443 mutante: LDAP - adding verenali to wmde and nda groups, to match raja_wmde (T233807, T231677)

17 October 2019

  • curprev 21:4221:42, 17 October 2019imported>Stashbot 335,605 bytes +9,045 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@d663006]: Update mobileapps to f345673 (duration: 05m 38s)
  • curprev 00:5500:55, 17 October 2019imported>Stashbot 326,560 bytes +13,865 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

16 October 2019

  • curprev 00:0900:09, 16 October 2019imported>Stashbot 312,695 bytes +11,193 mutante: wikitech - make JBond a "content administrator" to give the ability to create server fingerprint pages

14 October 2019

  • curprev 23:2723:27, 14 October 2019imported>Stashbot 301,502 bytes +9,644 Krinkle: Delete 2019-09-01––2019-09-10 arclamp trace logs from webperf1002, and decompress the rest of 2019-09 (this will trigger svg re-generation), T235425

13 October 2019

  • curprev 00:5200:52, 13 October 2019imported>Stashbot 291,858 bytes +984 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ec77b1b515940c73 (duration: 00m 55s)

11 October 2019

10 October 2019

  • curprev 22:2722:27, 10 October 2019imported>Stashbot 287,642 bytes +12,182 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting wgMFMobileFormatterHeadings, unread T232690 (duration: 00m 51s)
  • curprev 00:3900:39, 10 October 2019imported>Stashbot 275,460 bytes +16,804 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)

8 October 2019

  • curprev 23:2823:28, 8 October 2019imported>Stashbot 258,656 bytes +11,791 mutante: phab1001 - replacing tin.eqiad.wmnet with deploy1001.eqiad.wmnet in phabricator/deployment-cache/.config:git_server - wondering if we can ever get rid of tin (T190568)
  • curprev 00:3300:33, 8 October 2019imported>Stashbot 246,865 bytes +11,958 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)

6 October 2019

  • curprev 20:1120:11, 6 October 2019imported>Stashbot 234,907 bytes +365 Urbanecm: mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Racconish /home/urbanecm/T234741 (T234741)

5 October 2019

  • curprev 06:4806:48, 5 October 2019imported>Stashbot 234,542 bytes +151 elukey: force umount/remount of /mnt/hdfs on an-coord1001 - processes stuck in D state, fuser proc consuming a ton of memory

4 October 2019

3 October 2019

  • curprev 23:5023:50, 3 October 2019imported>Stashbot 231,365 bytes +5,107 mutante: gerrit - restarting for replication config tweaks
  • curprev 00:5100:51, 3 October 2019imported>Stashbot 226,258 bytes +6,323 mutante: gerrit1001 - removing wrong IPv6 address from interface, running puppet

2 October 2019

  • curprev 00:5400:54, 2 October 2019imported>Stashbot 219,935 bytes +8,918 ejegg: updated fundraising CiviCRM from 6d90d0cf06 to 12c5727a23

1 October 2019

  • curprev 00:1200:12, 1 October 2019imported>Stashbot 211,017 bytes +7,795 mutante: phabricator - upgrading PHP version to 7.2.22 - T230024

28 September 2019

27 September 2019

  • curprev 22:4422:44, 27 September 2019imported>Stashbot 203,145 bytes +3,427 mutante: phab2001 - apt-get autoremove - remove unused python and ruby packages
  • curprev 00:5300:53, 27 September 2019imported>Stashbot 199,718 bytes +16,143 twentyafterfour: hotfixing phabricator fatal exception refs T233998

25 September 2019

  • curprev 21:5921:59, 25 September 2019imported>Stashbot 183,575 bytes +17,154 bblack: remove GRE MTU hacks on archiva1001 gerrit2001 cobalt install1002 - T232602
  • curprev 01:0001:00, 25 September 2019imported>Stashbot 166,421 bytes +13,033 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 3373247e123b53 - create new file (duration: 01m 05s)

24 September 2019

  • curprev 00:4300:43, 24 September 2019imported>Stashbot 153,388 bytes +8,330 mutante: db2060 - remove PXE flag boot override - set Boot Device to none

22 September 2019

21 September 2019

19 September 2019

  • curprev 23:2323:23, 19 September 2019imported>Stashbot 138,735 bytes +9,963 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

18 September 2019

  • curprev 23:1823:18, 18 September 2019imported>Stashbot 128,772 bytes +12,473 krinkle@deploy1001: Synchronized php-1.34.0-wmf.23/extensions/MobileFrontend/resources/dist/: T233260, 1667ed957a19067 (duration: 01m 04s)

17 September 2019

  • curprev 23:2623:26, 17 September 2019imported>Stashbot 116,299 bytes +14,935 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.23 refs T220748
  • curprev 00:3400:34, 17 September 2019imported>Stashbot 101,364 bytes +15,480 AndyRussG: updated fruec from fb29cb7407 to 97128874bf

15 September 2019

14 September 2019

  • curprev 23:4223:42, 14 September 2019imported>Stashbot 85,524 bytes +133 onimisionipe: force shard allocation (dewiki_content_1566659363[4]) on eqiad cluster
  • curprev 01:1401:14, 14 September 2019imported>Stashbot 85,391 bytes +7,579 ejegg: updated fundraising python tools from 1e405864d7 to e1b81688c6

12 September 2019

11 September 2019

10 September 2019

  • curprev 20:4620:46, 10 September 2019imported>Stashbot 53,851 bytes +9,981 ejegg: updated payments-wiki from 15baf7f58b to 5432f9c3a4
  • curprev 00:5900:59, 10 September 2019imported>Stashbot 43,870 bytes +5,338 krinkle@deploy1001: Finished deploy [performance/navtiming@f2a0863]: (no justification provided) (duration: 00m 05s)

8 September 2019

  • curprev 22:0422:04, 8 September 2019imported>Stashbot 38,532 bytes +153 eileen: civicrm revision changed from c3172daf95 to d1d65f37ea, config revision is 5a6a9c6c03

6 September 2019

5 September 2019

  • curprev 23:1323:13, 5 September 2019imported>Stashbot 33,990 bytes +8,363 ayounsi@deploy1001: Finished deploy [netbox/deploy@367ca84]: test (duration: 00m 42s)
  • curprev 00:5500:55, 5 September 2019imported>Stashbot 25,627 bytes +10,215 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 53s)

4 September 2019

  • curprev 01:0201:02, 4 September 2019imported>Stashbot 15,412 bytes +11,302 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s)

2 September 2019

1 September 2019

31 August 2019

  • curprev 13:3313:33, 31 August 2019imported>Stashbot 747,212 bytes +134 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: 88ba4f8f4d49 (duration: 00m 55s)

30 August 2019

29 August 2019

  • curprev 23:2323:23, 29 August 2019imported>Stashbot 742,673 bytes +4,768 ejegg: updated payments-wiki from 1d5d7503b0 to 51d9ed79b6

28 August 2019

  • curprev 23:1723:17, 28 August 2019imported>Stashbot 737,905 bytes +7,538 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 8bfe43a: Add scielo.br to wgCopyUploadsDomains for commonswiki (T231402) (duration: 00m 55s)

27 August 2019

25 August 2019

  • curprev 13:4613:46, 25 August 2019imported>Stashbot 714,674 bytes +176 volans: uploaded spicerack_0.0.27-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • curprev 00:3500:35, 25 August 2019imported>Stashbot 714,498 bytes +283 herron: set icinga downtimes on flapping cr2-eqiad and cr2-codfw alerts until monday

23 August 2019

  • curprev 23:3423:34, 23 August 2019imported>Stashbot 714,215 bytes +1,363 robh@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)

22 August 2019

  • curprev 23:4123:41, 22 August 2019imported>Stashbot 712,852 bytes +5,069 urbanecm@deploy1001: Synchronized php-1.34.0-wmf.19/extensions/WikibaseLexeme: SWAT: e4a5457: Fix Lexemes RDF generation (T230974) (duration: 00m 49s)
  • curprev 00:3700:37, 22 August 2019imported>Stashbot 707,783 bytes −393,674 XioNoX: run /usr/local/sbin/restart-php7.2-fpm on mwdebug1001/2

20 August 2019

  • curprev 23:5323:53, 20 August 2019imported>Stashbot 1,101,457 bytes +7,677 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@a7bf6cf]: bulk_daemon: Increase bulk request_timeout (duration: 03m 40s)

19 August 2019

  • curprev 21:2121:21, 19 August 2019imported>Stashbot 1,093,780 bytes +5,899 ejegg: updated payments-wiki from 7b8091ba87 to 85dce8f79f

18 August 2019

  • curprev 08:2808:28, 18 August 2019imported>Stashbot 1,087,881 bytes +165 onimisionipe: running `_cluster/reroute?pretty&explain=true&retry_failed` on eqiad production-search cluster to force allocation of shards

16 August 2019

  • curprev 19:4819:48, 16 August 2019imported>Stashbot 1,087,716 bytes +1,009 sbassett: Deployed security patch for T230576 (ex:MobileFrontend)

15 August 2019

  • curprev 23:3523:35, 15 August 2019imported>Stashbot 1,086,707 bytes +7,511 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@b4da6e4]: Rollback blazegraph due to T230588 (duration: 09m 48s)
  • curprev 00:1500:15, 15 August 2019imported>Stashbot 1,079,196 bytes +5,064 robh: scs-ulsfo offline due to networking issues, rob returning tomorrow with fix T230077

13 August 2019

  • curprev 20:4320:43, 13 August 2019imported>Stashbot 1,074,132 bytes +11,563 ejegg: rolled back payments-wiki from 9ed8be0532 to 9533f70fab

12 August 2019

11 August 2019

  • curprev 22:3822:38, 11 August 2019imported>Stashbot 1,056,982 bytes +198 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Temporary make account creation limits more restrictive (T230304) (duration: 00m 50s)

10 August 2019

  • curprev 01:4901:49, 10 August 2019imported>Stashbot 1,056,784 bytes +187 mutante: mwmaint - running (1 of 8, the one for en) refreshLinks maintenance cron manually to verify it works after switching mwscriptwikiset to PHP7.2 (T195392)
  • curprev 00:5200:52, 10 August 2019imported>Stashbot 1,056,597 bytes +3,069 mutante: mwmaint - running update_flaggedrevs_stats - updates the flagged revs statistics table on each wiki

8 August 2019

7 August 2019

  • curprev 00:4800:48, 7 August 2019imported>Stashbot 1,039,482 bytes +10,857 mutante: restarting gerrit to apply config change 528276 to exclude some projects from github replication

5 August 2019

  • curprev 23:3423:34, 5 August 2019imported>Stashbot 1,028,625 bytes +13,466 mutante: mwmaint1002 - remove getJobQueueLengths.php from www-data's crontab (T195392)

4 August 2019

  • curprev 18:4518:45, 4 August 2019imported>Stashbot 1,015,159 bytes +150 krinkle@deploy1001: Synchronized wmf-config/abusefilter.php: labs-only noop - f740f89c594979 (duration: 00m 50s)

3 August 2019

  • curprev 12:0212:02, 3 August 2019imported>Stashbot 1,015,009 bytes +321 gilles: purging ruwiki articles on mwmaint1002
  • curprev 00:4000:40, 3 August 2019imported>Stashbot 1,014,688 bytes +5,626 ejegg: rolled back fundraising python tools from 493a38f9e0 to 2a56e5e283

1 August 2019

  • curprev 23:3223:32, 1 August 2019imported>Stashbot 1,009,062 bytes +8,682 Urbanecm: Evening SWAT done
  • curprev 00:3300:33, 1 August 2019imported>Stashbot 1,000,380 bytes +12,622 krinkle@deploy1001: Synchronized php-1.34.0-wmf.16/resources/Resources.php: acfff6751f3b8f7650 (duration: 00m 54s)

31 July 2019

  • curprev 00:2400:24, 31 July 2019imported>Stashbot 987,758 bytes +10,243 eileen: tools revision changed from 4910f1507c to 2a56e5e283

29 July 2019

  • curprev 23:3723:37, 29 July 2019imported>Stashbot 977,515 bytes +9,325 XioNoX: replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in ams

28 July 2019

  • curprev 15:1315:13, 28 July 2019imported>Stashbot 968,190 bytes +99 arturo: disable 1m load average check in icinga for labstore1007 for 24h

27 July 2019

  • curprev 17:3917:39, 27 July 2019imported>Stashbot 968,091 bytes +223 bd808: Updated profile & images for @wikimediatech twitter account
  • curprev 00:3500:35, 27 July 2019imported>Stashbot 967,868 bytes +4,085 mutante: restbase-dev1006 - starting nagios-nrpe-server

26 July 2019

  • curprev 00:5300:53, 26 July 2019imported>Stashbot 963,783 bytes +13,853 ejegg: re-enabled dedupe_civicrm_contacts and major_gifts_addresses fundraising jobs

24 July 2019

  • curprev 23:4623:46, 24 July 2019imported>Stashbot 949,930 bytes +10,601 nuria@deploy1001: Finished deploy [analytics/refinery@7d93398]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues). Try 2 (duration: 13m 34s)

23 July 2019

  • curprev 23:4423:44, 23 July 2019imported>Stashbot 939,329 bytes +9,022 eileen: civicrm revision changed from 88e9f24893 to 875ab97742, config revision is 4006d3bdc5
  • curprev 00:0600:06, 23 July 2019imported>Stashbot 930,307 bytes +11,599 Urbanecm: slwiki updateCollection.php completed (T208984)

21 July 2019

19 July 2019

  • curprev 22:3622:36, 19 July 2019imported>Stashbot 918,628 bytes +5,701 mutante: phab2001 - switching apache to php-fpm and worker instead of mpm-prefork (to match phab1001) (T190568 T137928 T190572)
  • curprev 00:5200:52, 19 July 2019imported>Stashbot 912,927 bytes +9,770 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Add even more severe rate limits for eswikiquote and some other, smaller wikis (T227416) (duration: 00m 58s)

17 July 2019

  • curprev 23:5123:51, 17 July 2019imported>Stashbot 903,157 bytes +11,329 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Add wmgUseTheWikipediaLibrary (false everywhere, no-op) (duration: 00m 54s)
  • curprev 00:5000:50, 17 July 2019imported>Stashbot 891,828 bytes +10,253 mutante: wikitech-static commented out cert renewal cron job out of caution - still needs fixing but continue tomorrow

16 July 2019

  • curprev 00:2400:24, 16 July 2019imported>Stashbot 881,575 bytes +12,943 krinkle@deploy1001: Synchronized php-1.34.0-wmf.13/includes/cache/LinkCache.php: 4a5f4ca2fd788 (duration: 00m 51s)

14 July 2019

  • curprev 13:1813:18, 14 July 2019imported>Stashbot 868,632 bytes +257 godog: silence mr1-eqsin.oob IPv6 until tomorrow 8 UTC - T227967

13 July 2019

12 July 2019

  • curprev 23:3523:35, 12 July 2019imported>Stashbot 868,307 bytes +3,936 mutante: netmon1003 - shutdown -h now after it's gone from Icinga now
  • curprev 01:0101:01, 12 July 2019imported>Stashbot 864,371 bytes +11,144 Krinkle: mw1342 generated some ~ 11,500 additional PHP errors over a 4 hour period (18:00-22:30 UTC), ref T224491

10 July 2019

  • curprev 23:1623:16, 10 July 2019imported>Stashbot 853,227 bytes +7,815 jforrester@deploy1001: Synchronized php-1.34.0-wmf.13/extensions/CirrusSearch/includes: T227691 RedirectsAndIncomingLinks: succeede or fail, but not both (duration: 01m 02s)

9 July 2019

  • curprev 23:0723:07, 9 July 2019imported>Stashbot 845,412 bytes +8,973 jhuneidi@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.13 refs T220738
  • curprev 01:4701:47, 9 July 2019imported>Stashbot 836,439 bytes +12,820 XioNoX: restart PHP FPM on mwdebug2001

7 July 2019

  • curprev 20:1320:13, 7 July 2019imported>Stashbot 823,619 bytes +210 urandom: decommissioning restbase1017-b -- T222960

6 July 2019

  • curprev 07:5607:56, 6 July 2019imported>Stashbot 823,409 bytes +74 thcipriani: restarting gerrit out of heap space

5 July 2019

  • curprev 17:1817:18, 5 July 2019imported>Stashbot 823,335 bytes +5,803 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@4b7cdf5]: Deploy new versions preparing for reload (duration: 00m 39s)

4 July 2019

  • curprev 21:5021:50, 4 July 2019imported>Stashbot 817,532 bytes +6,053 volans@deploy1001: Finished deploy [debmonitor/deploy@0ee26a3]: Deploy Debmonitor v0.1.10 (duration: 00m 48s)
  • curprev 00:4100:41, 4 July 2019imported>Stashbot 811,479 bytes +18,775 twentyafterfour: phabricator upgrade complete

3 July 2019

  • curprev 00:1400:14, 3 July 2019imported>Stashbot 792,704 bytes +11,829 eileen: process-control config revision is 8e215d07f2 (renable jobs)

1 July 2019

  • curprev 20:3320:33, 1 July 2019imported>Stashbot 780,875 bytes +7,789 milimetric@deploy1001: Finished deploy [analytics/refinery@4e9894c]: minor, just removing hiwikisource from sqoop list (duration: 01m 33s)

30 June 2019

  • curprev 23:2723:27, 30 June 2019imported>Stashbot 773,086 bytes +199 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 45s)

28 June 2019

  • curprev 21:1921:19, 28 June 2019imported>Stashbot 772,887 bytes +6,285 otto@deploy1001: Finished deploy [eventstreams/deploy@2af2719]: Manually blacklisting IP - T226808 (duration: 03m 07s)
  • curprev 01:2201:22, 28 June 2019imported>Stashbot 766,602 bytes +7,465 Krinkle: Killing arclamp-log on webperf1002, no flame graphs for three days, presumably mwlog/redis connection dropped again. T215740

27 June 2019

25 June 2019

  • curprev 22:5022:50, 25 June 2019imported>Stashbot 747,998 bytes +8,261 jforrester@deploy1001: Synchronized php-1.34.0-wmf.11/extensions/Echo/modules/nojs/: T226503 Fix badge icons in Monobook (duration: 00m 56s)
  • curprev 00:1000:10, 25 June 2019imported>Stashbot 739,737 bytes +11,072 reedy@deploy1001: Synchronized wmf-config/flaggedrevs.php: poke at autopromote config (duration: 00m 54s)

21 June 2019

  • curprev 14:5414:54, 21 June 2019imported>Stashbot 728,665 bytes +5,320 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 00:4500:45, 21 June 2019imported>Stashbot 723,345 bytes +15,806 RoanKattouw: Running FlowReserializeRevisionContent.php on testwiki

19 June 2019

  • curprev 18:1018:10, 19 June 2019imported>Stashbot 707,539 bytes +18,219 legoktm: added MatmaRex to extension-VisualEditor-staff Gerrit group

18 June 2019

  • curprev 22:3322:33, 18 June 2019imported>Stashbot 689,320 bytes +17,179 jijiki: pool thumbor1001
  • curprev 00:2200:22, 18 June 2019imported>Stashbot 672,141 bytes +12,375 RoanKattouw: Running populateRevisionSha1.php on dewikivoyage for T219816

16 June 2019

  • curprev 14:2014:20, 16 June 2019imported>Stashbot 659,766 bytes +253 Urbanecm: running mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='AKA MBG' /home/urbanecm/T225886

15 June 2019

  • curprev 20:3820:38, 15 June 2019imported>Stashbot 659,513 bytes +618 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@55174a4]: deploy new pattern for bots (duration: 21m 42s)

14 June 2019

  • curprev 23:2323:23, 14 June 2019imported>Stashbot 658,895 bytes +1,595 ejegg: updated payments-wiki from 75abd71cc1 to 79d1822644
  • curprev 00:1100:11, 14 June 2019imported>Stashbot 657,300 bytes +1,216 SMalyshev: depooled wdqs1005 - let it catch up

13 June 2019

  • curprev 00:4500:45, 13 June 2019imported>Stashbot 656,084 bytes +2,359 paravoid: setting the CPU governor to performance for ms-be1036 (a while ago)

11 June 2019

  • curprev 19:2419:24, 11 June 2019imported>Stashbot 653,725 bytes +610 tzatziki: Removing four (4) files for legal compliance

10 June 2019

9 June 2019

  • curprev 08:3008:30, 9 June 2019imported>Stashbot 650,997 bytes +79 vgutierrez: rebooting lvs4007 after NIC driver crash

8 June 2019

  • curprev 11:5811:58, 8 June 2019imported>Stashbot 650,918 bytes +203 godog: stop swift processes on ms-be1033 - T223518

7 June 2019

  • curprev 18:5618:56, 7 June 2019imported>Stashbot 650,715 bytes +6,617 herron: performing rolling reboots of logstash codfw frontends for security updates
  • curprev 00:0000:00, 7 June 2019imported>Stashbot 644,098 bytes +674 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove unused preference T47877-buster (duration: 00m 47s)

6 June 2019

5 June 2019

  • curprev 22:1522:15, 5 June 2019imported>Stashbot 635,049 bytes +20,643 chaomodus: restarting gerrit on cobalt due to it being down (seems like Java out of heap space)

4 June 2019

  • curprev 00:0900:09, 4 June 2019imported>Stashbot 614,406 bytes +7,967 bstorm_: T223406 repooled labsdb1011 after completing view updates

2 June 2019

  • curprev 20:2820:28, 2 June 2019imported>Stashbot 606,439 bytes +347 onimisionipe: pooled wdqs1007. It caught up on lag

1 June 2019

  • curprev 22:4922:49, 1 June 2019imported>Stashbot 606,092 bytes +604,365 krinkle@deploy1001: Synchronized php-1.34.0-wmf.7/extensions/3D/modules/mmv.3d.js: T224812 / bd4fbfddbe1a0 (duration: 01m 07s)

3 February 2019

  • curprev 20:2520:25, 3 February 2019imported>Stashbot 1,727 bytes +610 elukey: powercycle mw1272 - no ssh, no tty available via com2 - DIMM correctable errors + OEM errors registered in getsel
  • curprev 01:1001:10, 3 February 2019imported>Stashbot 1,117 bytes −57,244 elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred."

13 January 2019

  • curprev 16:3316:33, 13 January 2019imported>Stashbot 58,361 bytes +103 hoo: Updated operations/dumps/dcat (559dee37452..a86285f4e7) on snapshot1008

12 January 2019

11 January 2019

  • curprev 21:5621:56, 11 January 2019imported>Stashbot 57,967 bytes +2,518 jforrester@deploy1001: Finished scap: Full scap sync to update wmf.12 i18n for the weekend Idf2a67860f (duration: 19m 12s)
  • curprev 00:5000:50, 11 January 2019imported>Stashbot 55,449 bytes +11,158 XioNoX: bump prefix limit for AS6939 in eqsin

10 January 2019

  • curprev 01:4101:41, 10 January 2019imported>Stashbot 44,291 bytes +8,082 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make GrowthExperiments config wmf.12-proof (duration: 00m 52s)

9 January 2019

  • curprev 01:1501:15, 9 January 2019imported>Stashbot 36,209 bytes +13,531 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: T213227 Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s)

8 January 2019

6 January 2019

  • curprev 14:5014:50, 6 January 2019imported>Stashbot 15,081 bytes +251 ariel@deploy1001: Finished deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag (duration: 00m 06s)

5 January 2019

4 January 2019

  • curprev 23:0723:07, 4 January 2019imported>Stashbot 14,687 bytes +1,714 mutante: scandium apt-get remove nodejs nodes-legacy ; puppet agent -tv - after merging gerrit:482150 this fixed "you have held broken packages" issue, now we are at a puppet dependecy cycle with apt::pin T201366
  • curprev 00:0700:07, 4 January 2019imported>Stashbot 12,973 bytes +6,055 mutante: an-coord1001 - apt-get clean to free disk space, reacting to Icinga alert for running out of disk

3 January 2019

  • curprev 00:2100:21, 3 January 2019imported>Stashbot 6,918 bytes +6,798 mutante: notebook1004 - started nagios-nrpe-server one more time

1 January 2019

30 December 2018

29 December 2018

  • curprev 14:5314:53, 29 December 2018imported>Stashbot 133,496 bytes +935 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T212667 218371fd35 - Adjust mw.org abusefilter emergency shutoff threshold down to 0.3 (duration: 00m 46s)

28 December 2018

  • curprev 18:1418:14, 28 December 2018imported>Stashbot 132,561 bytes +537 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 97446843a27 T212667 - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s)

27 December 2018

  • curprev 19:3119:31, 27 December 2018imported>Stashbot 132,024 bytes +1,926 mobrovac@deploy1001: Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 (duration: 25m 49s)

26 December 2018

  • curprev 16:3716:37, 26 December 2018imported>Stashbot 130,098 bytes +468 godog: clear logstash persistent queue /var/lib/logstash on logstash100[789]

24 December 2018

23 December 2018

22 December 2018

21 December 2018

20 December 2018

  • curprev 00:0900:09, 20 December 2018imported>Stashbot 119,323 bytes +8,184 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/479911/ (duration: 00m 53s)

19 December 2018

  • curprev 01:2701:27, 19 December 2018imported>Stashbot 111,139 bytes +5,652 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: g 480681 make maintenance script dry run more useful (duration: 00m 52s)

18 December 2018

16 December 2018

  • curprev 09:5209:52, 16 December 2018imported>Stashbot 92,602 bytes +240 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)

15 December 2018

14 December 2018

  • curprev 23:5923:59, 14 December 2018imported>Stashbot 235,055 bytes +4,222 mutante: LDAP: added aezell to wmf group (T211945) for grafana access
  • curprev 01:3901:39, 14 December 2018imported>Stashbot 230,833 bytes +14,006 mutante: install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)

13 December 2018

  • curprev 00:5200:52, 13 December 2018imported>Stashbot 216,827 bytes +9,257 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php: T211806 (duration: 00m 51s)

12 December 2018

11 December 2018

  • curprev 01:0301:03, 11 December 2018imported>Stashbot 197,142 bytes +9,075 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure localized logos for nywiki (T211570) (duration: 01m 36s)

10 December 2018

  • curprev 00:5500:55, 10 December 2018imported>Stashbot 188,067 bytes +620 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

8 December 2018

  • curprev 23:0723:07, 8 December 2018imported>Stashbot 187,447 bytes +460 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)

7 December 2018

6 December 2018

  • curprev 00:5500:55, 6 December 2018imported>Stashbot 177,878 bytes +11,714 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)

5 December 2018

4 December 2018

  • curprev 01:2201:22, 4 December 2018imported>Stashbot 153,921 bytes +10,468 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703

3 December 2018

  • curprev 00:3400:34, 3 December 2018imported>Stashbot 143,453 bytes +402 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

1 December 2018

  • curprev 16:4816:48, 1 December 2018imported>Stashbot 143,051 bytes +106 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904

30 November 2018

29 November 2018

  • curprev 00:0800:08, 29 November 2018imported>Stashbot 131,984 bytes +8,825 catrope@deploy1001: Synchronized wmf-config/throttle.php: T210681 (duration: 01m 04s)

28 November 2018

  • curprev 01:4501:45, 28 November 2018imported>Stashbot 123,159 bytes +10,054 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)

27 November 2018

  • curprev 00:3800:38, 27 November 2018imported>Stashbot 113,105 bytes +8,723 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix weird caching issue maybe for Ian's patch (duration: 00m 46s)

25 November 2018

23 November 2018

22 November 2018

21 November 2018

  • curprev 23:3423:34, 21 November 2018imported>Stashbot 97,893 bytes +6,623 mutante: rsyncing /home from rutherfordium.eqiad to people1001.eqiad (people.wikimedia.org) T210036
  • curprev 00:4800:48, 21 November 2018imported>Stashbot 91,270 bytes +7,660 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474967 Disable FlaggedRevs, enable RC patrol and add rights on srwikinews (duration: 00m 47s)

20 November 2018

  • curprev 00:5500:55, 20 November 2018imported>Stashbot 83,610 bytes +6,935 catrope@deploy1001: Synchronized static/images/project-logos/: Correct logos for Sindhi Wiktionary (duration: 00m 47s)

18 November 2018

17 November 2018

  • curprev 07:0207:02, 17 November 2018imported>Stashbot 76,539 bytes +331 volans|off: 'reset modified attributes' on IcingaUI for db1078 (and mgmt) and all its services

16 November 2018

  • curprev 23:1323:13, 16 November 2018imported>Stashbot 76,208 bytes +2,465 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010 (duration: 00m 23s)
  • curprev 00:5500:55, 16 November 2018imported>Stashbot 73,743 bytes +12,226 mutante: some users reported missing files in home dirs on mwmaint1002, reversed rsyncd/ferm setup and rsynced /home from mwmaint2001 to /root on mwmaint1002, restored individually where requested, rsync is not fully automatic but puppetized with rsync...

15 November 2018

  • curprev 00:5000:50, 15 November 2018imported>Stashbot 61,517 bytes +9,591 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: increase Schema.org page split test to 5% sampling T208755 (duration: 00m 54s)

14 November 2018

12 November 2018

  • curprev 19:2219:22, 12 November 2018imported>Stashbot 44,564 bytes +4,905 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T208663 4ff32d1df - Enable moving files for users with patrol and rollbacker rights on srwiki (duration: 00m 54s)

10 November 2018

  • curprev 01:1001:10, 10 November 2018imported>Stashbot 39,659 bytes +4,598 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s)

9 November 2018

  • curprev 00:0700:07, 9 November 2018imported>Stashbot 35,061 bytes +5,634 ejegg: updated fundraising CiviCRM from 07183ed7cc to 02cc1f80d4

8 November 2018

  • curprev 00:4300:43, 8 November 2018imported>Stashbot 29,427 bytes +10,411 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader: Fail less hard when JSON serialization of config fails I673f59d93 (duration: 00m 53s)

7 November 2018

  • curprev 00:4600:46, 7 November 2018imported>Stashbot 19,016 bytes +4,891 mutante: tegmen - shutting down for renaming and reinstall (T208824)

6 November 2018

  • curprev 00:2600:26, 6 November 2018imported>Stashbot 14,125 bytes +6,053 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s)

4 November 2018

3 November 2018

  • curprev 09:3509:35, 3 November 2018imported>Stashbot 7,058 bytes +112 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2 November 2018

  • curprev 17:0417:04, 2 November 2018imported>Stashbot 6,946 bytes +1,465 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}

1 November 2018

30 October 2018

29 October 2018

  • curprev 21:4421:44, 29 October 2018imported>Stashbot 171,385 bytes +15,633 mutante: contint2001 - installing libtiff upgrade, with the same command from apt/history.log that had shown as failed

28 October 2018

  • curprev 23:3623:36, 28 October 2018imported>Stashbot 155,752 bytes +600 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/Graph: T184128 - I02da92de33 (duration: 00m 58s)
  • curprev 00:0300:03, 28 October 2018imported>Stashbot 155,152 bytes +2,187 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/CirrusSearch: T206967 - Ia23d19cf1e6 (duration: 01m 02s)

27 October 2018

  • curprev 00:0000:00, 27 October 2018imported>Stashbot 152,965 bytes +4,090 mutante: icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)