You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
< Nova Resource:CvnJump to navigation Jump to search
Revision as of 15:31, 14 March 2019 by (Krinkle: Killed 4 parallel /srv/cvn/git/infrastructure/bin/cvndb-pull processes on cvn-apache9)
- 15:31 Krinkle: Killed 4 parallel /srv/cvn/git/infrastructure/bin/cvndb-pull processes on cvn-apache9
- 15:28 Krinkle: Killed 200 parallel cvndb-CVNBot14-publish processes on cvn-app8
- 05:20 rxy: Flags +AV were set on cohaf in #cvn-simplewikis.
- 05:18 rxy: Flags +AV were set on Tulsi in #cvn-sw.
- 18:46 Joan: Restarted CVNBot18 (Last message was received on RCReader 29613.939513 seconds ago).
- 17:52 Joan: Restarted CVNBot18 (Last message was received on RCReader 47252.762304 seconds ago)
- 19:05 Joan: Restarted CVNBot18 (Last message was received on RCReader 64065.614616 seconds ago)
- 18:43 Joan: Flag +V were set on AsimovBot in #cvn-es-scan.
- 18:33 Joan: Flags +AV were set on jem, DLeandroc, wiki1776 and Esteban16 in #cvn-es-scan.
- 18:22 Joan: Restarted CVNBot18 (Last message was received on RCReader 29421.503816 seconds ago)
- 20:11 Joan: Restarted CVNBot18 (Last message was received on RCReader 22761.108125 seconds ago)
- 15:36 rxy: Flags +AV were set on Zppix in #cvn-simplewikis.
- 23:51 Joan: Restarted CVNBot18 (Last message was received on RCReader 95275.811571 seconds ago)
- 21:43 Joan: Restarted CVNBot18 (Last message was received on RCReader 106111.888376 seconds ago)
- 05:12 Joan: Restarted CVNBot18 (Last message was received on RCReader 31518.429916 seconds ago)
- 02:17 Joan: CVNBot18 restarted (Last message was received on RCReader 57307.704279 seconds ago)
- 21:59 rxy: sudo kill $CVNBot1PID
- 22:49 Joan: Restarted CVNBot3 (Last message was received on RCReader 5729.637672 seconds ago)
- 06:20 Joan: Restarted CVNBot18 (Last message was received on RCReader 53393.770933 seconds ago)
- 23:08 Joan: Restarted CVNBot18 (Last message was received on RCReader 131313.161477 seconds ago)
- 00:55 Joan: Restarted CVNBot18 (Last message was received on RCReader 5395.508043 seconds ago)
- 02:55 Joan: CVNBot18 restarted (Last message was received on RCReader 5385.431178 seconds ago)
- 17:51 Joan: Restarted CVNBot18 (Last message was received on RCReader 4033.703116 seconds ago)
- 01:45 Cam11598: Flags +fiotv were set on Operator873 in #cvn-simplewikis.
- 01:43 Cam11598: Flags +AV were set on Operator873 in #cvn-wp-en.
- 00:38 Krinkle: Rebooting cvn-app8
- 00:37 Joan: Restarted CVNBot18 (Last message was received on RCReader 5612.850734 seconds ago)
- 15:50 Cam11598: 7:49:59 AM <@Cam11598> Cameron CVNBot18 restart
- 15:50 Cam11598: 7:49:59 AM <CVNBot18> Last message was received on RCReader 6280.499422 seconds ago
- 08:09 Krinkle: Rebooting cvn-app8 and cvn-app9
- 04:50 Joan: CVNBot18 restarted
- 20:49 AlexZ: restarted SWBot3
- 23:24 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 3624.79361 seconds ago)
- 03:02 Krinkle: Update /srv/cvn/git/infrastructure to latest master on cvn-app8 and cvn-app9
- 02:58 Krinkle: access #cvn-wp-es add Joan@wikimedia/JOAN local_op
- 02:48 Krinkle: Add CVN-ClerkBot_ to #cvn-es-scan
- 02:34 Krinkle: #cvn-es-scan approved for Joan (m:User:JOAN), with wm-bot. Maybe later CVNBot - https://github.com/countervandalism/infrastructure/issues/21
- 00:46 Joan: Restarted CVNBot18 (Last message was received on RCReader 5003.313454 seconds ago)
- 00:27 Joan: Restarted CVNBot18 (Last message was received on RCReader 36500.416889 seconds ago)
- 03:42 rxy: rxy@cvn-app8:~$ sudo kill $CVNBot14_pid
- 03:39 rxy: Flags +AV were set on Vermont in #cvn-mediawiki.
- 03:36 rxy: Flags +AV were set on Zhangj1079 in #cvn-wikivoyage.
- 03:36 rxy: Flags +AV were set on Zhangj1079 in #cvn-wikidata.
- 03:36 rxy: Flags +AV were set on Zhangj1079 in #cvn-mediawiki.
- 03:36 rxy: Flags +AV were set on Zhangj1079 in #cvn-commons-uploads.
- 15:24 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 4003.334833 seconds ago)
- 00:27 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 3666.364954 seconds ago)
- 17:59 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 5893.862965 seconds ago)
- 01:34 AlexZ: Flags +AV were set on Kb03 in #cvn-unifications.
- 23:14 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 3054.046536 seconds ago)
- 19:49 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 2826.897996 seconds ago)
- 18:16 Joan: Restarted CVNBot18 in #cvn-wikidata (Last message was received on RCReader 3021.34062 seconds ago)
- 22:06 Joan: Restart CVNBot18 in #cvn-wikidata (Last message was received on RCReader 6503.673215 seconds ago)
- 22:18 Joan: Restart CVNBot18 in #cvn-wikidata
- 02:57 Cam11598: granted Joan@wikimedia/JOAN local_op on #cvn-wikidata
- 16:40 rxy: Flags +AV were set on LakesideMiners in #cvn-unifications.
- 23:02 AlexZ: restarted CVNBot14, CVNBot18, CVNBot2, CVNBot3, CVNBot5 due to RC lag.
- 15:10 Hauskatze: restarted cvnbot3 on #cvn-wp-es
- 11:11 rxy: Flags +AV were set on shanmugamp7 in #cvn-mediawiki.
- 05:57 Cam11598: restarted cvnbot19
- 05:57 Cam11598: 9:56:34 PM <�25B100+ CVNBot19> Last message was received on RCReader 5202.596163 seconds ago
- 21:14 Cam11598: restarted cvnbot19
- 21:14 Cam11598: 1:13:20 PM <+CVNBot19> Last message was received on RCReader 4397.05632 seconds ago
- 04:40 Oshwah: Received +o in #cvn-wp-en
- 04:38 AlexZ: unban snowycats and +o Oshwah in #CVN-wp-en
- 21:24 Cam11598: restarted cvnbot18
- 21:24 Cam11598: 1:23:04 PM <�25B100+ CVNBot18> Last message was received on RCReader 31391.185322 seconds ago
- 05:41 Cam11598: restarted cvnbot10
- 05:41 Cam11598: 9:40:50 PM <�25B100+ CVNBot10> Last message was received on RCReader 80417.36911 seconds ago
- 05:37 AlexZ: CVNBot3 restarted due to lag
- 01:55 Krinkle: Upgrade CVNBot18 to CVNBot 1.22.0-beta.1 (latest master, xbuild, cvn-app8)
- 01:38 Cam11598: restarted cvnbot18 had no response for over 30000 seconds
- 19:04 Cam11598: CVNBot1 restarted
- 19:04 Cam11598: 11:04:09 AM <+CVNBot1> Last message was received on RCReader 24889.420019 seconds ago
- 18:28 Krinkle: CVNBot19 inactive over 5000 seconds. Restarted
- 18:27 Krinkle: CVNBot 6, 8, 9, 10 were inactive over 30,000 seconds. Restarted
- 01:43 Cam11598: restarted all bots due to 6 minute lag reported by bsadaowski1
- 04:12 AlexZ: restarted CVNBot18 for not responding on rc channel
- 04:11 AlexZ: 8:11:14 PM <+CVNBot18> Last message was received on RCReader 208260.640677 seconds ago
- 23:41 Cam11598: restarted cvnbot19, cvnbot6 cvnbot7
- 23:40 Cam11598: 3:37:56 PM <�25B100+ CVNBot7> Last message was received on RCReader 144095.001034 seconds ago
- 23:40 Cam11598: 3:37:47 PM <�25B100+ CVNBot6> Last message was received on RCReader 143289.512457 seconds ago
- 23:39 Cam11598: 3:37:39 PM <�25B100+ CVNBot19> Last message was received on RCReader 1055.947407 seconds ago
- 00:59 Cam11598: 4:57:54 PM <�25B100+ CVNBot18> Last message was received on RCReader 25099.776314 seconds ago
- 07:55 AlexZ: rebooted cvnbot18 due to no activity in 3d
- 03:30 Cam11598: 7:30:09 PM �25B100<+ CVNBot19> Last message was received on RCReader 2757.868659 seconds ago
- 03:29 Cam11598: 7:29:41 PM <�25B100+ CVNBot18> Last message was received on RCReader 41761.804095 seconds ago
- 03:29 Cam11598: restarted CVNBot1
- 03:28 Cam11598: 7:28:15 PM <�25B100+ CVNBot1> Last message was received on RCReader 40284.618378 seconds ago
- 18:15 Krinkle: access #cvn-wikidata add Esteban16 voiced
- 20:05 Cam11598: 12:03:52 PM <@Cam11598> cvnbot3 restart
- 20:05 Cam11598: 12:03:46 PM <+CVNBot3> Last message was received on RCReader 192160.850344 seconds ago
- 20:01 Cam11598: restarted Cubbie
- 20:01 Cam11598: 12:00:00 PM <+Cubbie> Last message was received on RCReader 191898.12268 seconds ago
- 20:00 Cam11598: 11:59:55 AM <@Cam11598> CVNBot18 restart
- 20:00 Cam11598: 11:59:49 AM <+CVNBot18> Last message was received on RCReader 118066.396796 seconds ago
- 08:05 AlexZ: 12:03:36 AM <+CVNBot1> Last message was received on RCReader 62547.035233 seconds ago
- 08:04 AlexZ: CVNBot1 rebooted due to reporting no edits in 17hrs
- 00:39 AlexZ: rebooted all the bots due to lag
- 16:44 AlexZ: CVNBot3 and CVNBot18 were rebooted
- 21:25 Cam11598: restarted CVNBot19
- 06:12 Cam11598: restarted bot #'s 2, 4, 6, 9, 14, 19 none of them appeared to be in their irc.wikimedia.org channels.
- 06:09 Cam11598: 10:05:45 PM <+CVNBot24> Last message was received on RCReader 566110.673466 seconds ago
- 06:05 Cam11598: 10:04:12 PM <+CVNBot19> Last message was received on RCReader 7186.456569 seconds ago
- 06:03 Cam11598: 10:02:10 PM <+CVNBot14> Last message was received on RCReader 820278.151891 seconds ago
- 05:59 Cam11598: 9:59:35 PM <+CVNBot9> Last message was received on RCReader 30702.16703 seconds ago
- 05:59 Cam11598: 9:59:00 PM <+CVNBot6> Last message was received on RCReader 33346.388139 seconds ago
- 05:58 Cam11598: 9:58:29 PM <+CVNBot4> Last message was received on RCReader 650.202431 seconds ago
- 05:58 Cam11598: 9:57:44 PM <+CVNBot2> Last message was received on RCReader 717807.952057 seconds ago
- 02:28 Cam11598: bots 3 & 18 restarted by rxy at about 22:00 UTC
- 07:33 AlexZ: set voice for Hian in #cvn-commons , #cvn-meta , #cvn-sw-spam , and #cvn-wp-en
- 02:08 Cam11598: set Vermont voiced in cvn-meta
- 03:54 Cam11598: restarted bots 18, 3
- 00:32 Cam11598: 4:32:16 PM <ChanServ> Flags +AV were set on Kb03 in #cvn-wp-en.
- 00:23 Cam11598: restarted CVNBot1
- 00:22 Cam11598: 4:22:35 PM <�25B100+ CVNBot1> Last message was received on RCReader 5768.530484 seconds ago
- 20:44 Cam11598: restarted CVNBots 6, 8, 10, 19 all had greater than 1,000 second delays from RC network
- 07:38 Cam11598: restarted cvnbot18 after this delay in rtrc updates: 11:37:09 PM <�25B100+ CVNBot18> Last message was received on RCReader 16351.91394 seconds ago
- 04:39 Cam11598: 8:39:18 PM <ChanServ> Flags +AV were set on Bradv in #cvn-wp-en.
- 04:34 Cam11598: 8:32:52 PM <ChanServ> Flags +AV were set on SQLDb in #cvn-wp-en. (SQL is an en wiki admin)
- 10:00 AlexZ: restarted bots 6,7,8 and 19 in #cvn-sw
- 16:07 AlexZ: SWBot3 rebooted by Cam11598
- 03:41 Krinkle: CVNBot4, 5 and SWBot3 were missing from broadcast link. Restart fixed it.
- 00:46 Krinkle: Rebooting cvn-app8 and cvn-app9 servers.
- 00:44 Krinkle: Problems with bots are not due to unexpected reboot (there was no such reboot). Instead, the cause is https://phabricator.wikimedia.org/T208916 (wmflabs internal network problems).
- 23:39 Krinkle: It seems connections stopped from cvn-app bot services. Are slowly recovering now. Possible server restart?
- 15:50 rxy: reboot cvn-app9.eqiad1
- 15:50 rxy: reboot cvn-app8.eqiad1
- 03:51 AlexZ: Flags +AV were set on Joan in #cvn-wp-en.
- 09:46 rxy: Flags +AV were set on BRPever in #cvn-commons.
- 15:56 Cam11598: 8:55:44 AM <ChanServ> Flags +AV were set on Tks4Fish in #cvn-simplewikis.
- 19:26 Cam11598: 12:25:33 PM <ChanServ> Flags +AV were set on Operator873 in #cvn-sw.
- 20:22 Krinkle: Upgrade mono packages to latest from mono-project (5.16)
- 20:21 Krinkle: Adding mono-project to apt sources on cvn-app8 and cvn-app9
- 16:48 rxy: rxy@cvn-app9: setup "#cvn-wp-hr" per https://github.com/countervandalism/infrastructure/issues/12
- 22:39 Krinkle: Flags -AV were set on Rodejong in #cvn-wp-da.
- 01:01 Cam11598: 6:00:28 PM <ChanServ> Flags +AV were set on Vermont in #cvn-commons.
- 05:19 rxy: Flags +AV were set on [1997kB] in #cvn-sw.
- 18:25 Krinkle: cs access #cvn-wp-en add Father_of_Lies voiced
- 18:25 Krinkle: cs access #cvn-sw add Father_of_Lies voiced
- 03:17 Cam11598: restarted cvnbot3
- 18:17 Krinkle: access #cvn-simplewikis add CabalBot bot
- 21:17 rxy: Flags +AV were set on Father_Of_Lies in #cvn-wp-nl.
- 17:25 rxy: rxy@cvn-app9:~$ sudo kill $CVNBot25_pid
- 17:24 rxy: rxy@cvn-app9:~$ sudo kill $CVNBot17_pid
- 17:24 rxy: rxy@cvn-app8:~$ sudo kill $CVNBot20_pid
- 17:22 rxy: rxy@cvn-app8:~$ sudo kill $CVNBot5_pid
- 17:22 rxy: rxy@cvn-app8:~$ sudo kill $CVNBot4
- 17:16 rxy: rxy@cvn-app8:~$ sudo kill $SWBot3Pid
- 01:51 AlexZ: 6:50:28 PM <ChanServ> Matiia set flags +AV on irrationalist in #cvn-sw
- 01:06 rxy: Flags +AV were set on Esteban16 in #cvn-meta.
- 22:48 rxy: rxy@cvn-app8:~$ kill $CVNBot21_pid
- 22:46 rxy: rxy@cvn-app8:~$ kill $CVNBot18_pid
- 04:13 rxy: Flags +AV were set on Joanmrl in #cvn-wikidata.
- 04:13 rxy: Flags +AV were set on Joanmrl in #cvn-meta.
- 05:27 Cam11598: Flags +AV were set on PlyrStar93 in #cvn-sw.
- 08:49 rxy: (ChanServ) : Flags +AV were set on BRPever in #cvn-sw.
- 06:01 Cam11598: 10:58:09 PM <ChanServ> Flags +AV were set on Mz7 in #cvn-wp-en.
- 06:00 AlexZ: 10:56:24 PM < ChanServ> Cam11598 (cameron11598) set flags +AV on TheSandDoctor in #cvn-wp-en (en wiki sysop)
- 03:19 Cam11598: Matia set flags +AV on Framawiki in #cvn-sw
- 21:57 Cam11598: 2:52:46 PM <@ ChanServ> AlexZ (Az1568) set flags +AV on AntiComposite in #cvn-wp-en
- 21:56 Cam11598: 2:54:49 PM <@ ChanServ> AlexZ (Az1568) set flags +AV on Operator873 in #cvn-simplewikis
- 06:55 Cam11598: Seahorse, Kylu and Tanvir access removed from all channels. By both Az1568 and Cameron11598.
- 00:17 Cam11598: Flags +AV were set on Tomybrz in #cvn-sw. per metawiki:Special:Diff/18119554
- 22:33 Cam11598: 3:31:30 PM <ChanServ> Flags +AV were set on pepermuntjes in #cvn-wp-nl. Per https://meta.wikimedia.org/w/index.php?title=Countervandalism_Network/Verification&diff=prev&oldid=18116413
- 03:35 Cam11598: ACL updates User:Cameron11598/CVN/Removals/2018/June/09
- 18:49 Cam11598: 11:49:16 AM <@ ChanServ> Cam11598 (cameron11598) set flags +AV on Zhangj1079 in #cvn-sw
- 16:20 rxy: Flags +AV were set on Zhangj1079 in #cvn-simplewikis. per m:Special:Diff/18114842
- 21:23 Krinkle: #cvn-wp-da add Asger1995 local_op
- 01:04 rxy: rxy@cvn-app8: modified CVNBot.ini for SWBot3, and create symlink. per https://github.com/countervandalism/infrastructure/issues/18
- 02:59 Krinkle: Ran `DELETE FROM users WHERE name LIKE "% x=%";` on CVNBot14/Lists.sqlite to purge rows due to https://github.com/countervandalism/CVNBot/issues/44
- 09:12 rxy: rxy@cvn-app8$ echo 'feedFilterUsersReg=1' >> /srv/cvn/services/cvnbot/CVNBot21/CVNBot.ini
- 00:20 Krinkle: Revoked access to 'cvn' Wikimedia Cloud VPS project for 'sactage' and 'seahorseruler' users
- 18:44 rxy: Setup CVNBot25 for #cvn-ko-scan @cvn-app9
- 18:32 rxy: rxy@cvn-app9:/srv/cvn/git/stillalive$ sudo chmod g+w -R .
- 18:02 rxy: rxy@cvn-app9:/srv/cvn/git/CVNBot$ sudo chmod g+w /srv/cvn/services/cvnbot/
- 18:01 rxy: rxy@cvn-app9:/srv/cvn/git/CVNBot$ sudo chmod g+w -R .
- 21:54 rxy: "@cvn-app9:/srv/cvn/services/cvnbot/CVNBot16$ sudo chmod g+w Projects.xml" : error fixing when execute command "batchreload"
- 16:52 Krinkle: Deploying https://github.com/countervandalism/CVNBot/commit/6c60cd4bd1 to all bots on cvn-app8 and cvn-app9
- 21:40 Krinkle: Deleting cvn-apache8 (replaced by cvn-apache9)
- 02:36 Krinkle: Shutdown now-unused cvn-apache8 instance. To be deleted later.
- 02:36 Krinkle: Switch cvn.wmflabs.org web proxy over to new cvn-apache9 backend.
- 02:14 Krinkle: Create and provision cvn-apache9 (Debian 9 Stretch) to replace cvn-apache8 (Ubuntu 14 Trusty)
- 01:41 Krinkle: Deleting cvn-app6 instance
- 01:20 Krinkle: cvn-app6 is now idle. Shutting down from Horizon. To be deleted next week.
- 02:13 Krinkle: Moved CVNBot16 (with Console-js.msgs) to cvn-app9
- 00:58 Krinkle: Voiced NewbieAudioVideo (Dispenser) in #cvn-sw
- 22:26 Krinkle: Provisioning cvn-app9 (Debian Stretch) as [type=appserver] per https://github.com/countervandalism/infrastructure/blob/master/setup.yaml. To be replacemenet of cvn-app6 (Ubuntu Trusty)
- 07:09 Krinkle: Created cvn-app9 (Debian 9) - will replace cvn-app6 (Trusty)
- 07:09 Krinkle: Shutting down cvn-app7 (unused, migrated bots to cvn-app8)
- 20:56 Krinkle: Moved Cubbie to cvn-app8 and upgraded to 1.22.0-alpha
- 15:22 Krinkle: Moved CVNBot18 to cvn-app8 and upgraded to CVNBot v1.22.0-alpha
- 04:58 Krinkle: CVNBot1 (#cvn-wp-en) database maintenance: Purge bad 'users' table entries of names starting with - Ref https://github.com/countervandalism/CVNBot/issues/32
- 04:58 Krinkle: CVNBot12 (#cvn-wp-nl) database maintenance: Purge bad 'users' table entries of names starting with - Ref https://github.com/countervandalism/CVNBot/issues/32
- 18:33 Krinkle: moved CVNBot1 to cvn-app8 and upgraded to v1.22.0-alpha
- 17:54 Krinkle: CVNBot12 is now running v1.22.0-alpha on cvn-app8 (Debian 9, Mono 4)
- 23:29 Krinkle: Disabling /etc/cron.hourly/cvn-collect-log on all servers
- 00:55 Krinkle: Running experimental CVNBot version from https://github.com/countervandalism/CVNBot/commits/mono-4x on cvn-app8, in channel #cvn-sandbox. Discovering and fixing any issues as they come up.
- 20:05 Krinkle: Created and setting up cvn-app8 (Debian 9)
- 20:04 Krinkle: Starting migration from Ubuntu 14 Trusty to Debian 9 Stretch - https://github.com/countervandalism/infrastructure/issues/13
- 21:00 Krinkle: Ran sudo apt-get clean on cvn-apache8 to free up some space.
- 20:47 Krinkle: Soft-rebooting cvn-apache8
- 18:06 Krinkle: Restarted servers to have them re-join #cvn-broadcast following the random K-like from freenode staff
- 18:26 Krinkle: CVNBot12 won't restart. Has been in a once-a-minute restart loop for the past 24 hours (log: "IRC: Closing Link: 220.127.116.11 (Connection timed out)")
- 18:19 Krinkle: CVNBot12 is online, but not in any channel. Process is running. Restarting..
- 21:59 Krinkle: Restarting cvn-app7
- 20:59 rxy: "rxy@cvn-app7:~$ sudo mozroots --import --sync --machine --url http://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt"
- 20:58 rxy: "rxy@cvn-app6:~$ sudo mozroots --import --sync --machine"
- 18:18 rxy: setup CVNBot24 in cvn-app6
- 19:25 Krinkle: Changed e-mail address of Clogger cvnwiki account from my personal gmail to email@example.com
- 03:49 Krinkle: Fix cvndb-pull.cron permissions on cvn-apache8 (caused api to not get updated every 5 minutes)
- 18:54 Krinkle: Deleted cvn-app5 instance
- 01:54 Krinkle: [horizon.wikimedia.org] Retiring cvn-app5 (removing public IP, shutting down instance). To be deleted in a few days.
- 01:49 Krinkle: Restored CVNBot20 database from cvn-app5 (retired) to cvn-app7 (got corrupted in transfer somehow)
- 01:43 Krinkle: CVNBot20 database backup 2 days ago (Nov 16) also corrupt.
- 01:40 Krinkle: CVNBot20 database was corrupted for unknown reasons. Replaced with a copy of CVNBot14's database.
- 00:15 Krinkle: 'sudo chmod 660 */CVNBot.ini' in /srv/cvn/services/cvnbot on cvn-app6 and cvn-app7
- 00:09 Krinkle: 'sudo chmod 775 */CVNBot.exe */Lists.sqlite' in /srv/cvn/services/cvnbot on cvn-app6 and cvn-app7
- 23:06 rxy_: rxy@cvn-app7 "sudo chmod g+w -R /srv/cvn/git/stillalive" for write permission of group members
- 23:05 rxy_: rxy@cvn-app6 "sudo chmod g+w -R /srv/cvn/git/stillalive" for write permission of group members
- 22:56 rxy_: rxy@cvn-app6 "sudo chmod g+w /srv/cvn/services/cvnbot" for write permission of group members
- 22:56 rxy_: rxy@cvn-app6 "sudo chgrp project-cvn /srv/cvn/services/cvnbot" for write permission of group members
- 22:49 rxy_: rxy@cvn-app7 "sudo chmod g+w /srv/cvn/services/cvnbot" for write permission of group members
- 22:48 rxy_: rxy@cvn-app7 "sudo chgrp project-cvn /srv/cvn/services/cvnbot" for write permission of group members
- 02:17 Krinkle: Moving CVNBot1, CVNBot4, CVNBot5, CVNBot12, and CVNBot13 from cvn-app5 to cvn-app7
- 02:02 Krinkle: Moving Cubbie and SWBot3 from cvn-app5 to cvn-app7
- 01:55 Krinkle: Moving CVNBot21 (#cvn-mediawiki) from cvn-app5 to new cvn-app7
- 00:57 Krinkle: Provisioning cvn-app7
- 06:13 Krinkle: Creating cvn-app7 (m1.large; ubuntu-14.04-trusty) - replacement for cvn-app5 (m1.xlarge; ubuntu-12.04-precise)
- 06:00 Krinkle: Decom cvn-app4 (created 2014; ubuntu-12.04-precise) - all bots migrated to cvn-app6 (ubuntu-14.04-trusty)
- 06:00 Krinkle: Moving CVNBot23 from cvn-app4 to cvn-app6
- 05:55 Krinkle: Moving CVNBot9, CVNBot10, and CVNBot19 from cvn-app4 to cvn-app6
- 05:40 Krinkle: Moving CVNBot6, CVNBot7, and CVNBot8 from cvn-app4 to cvn-app6
- 05:17 Krinkle: Moving CVNBot3 from cvn-app4 to cvn-app6
- 11:09 rxy: "rxy@cvn-app4:~$ sudo kill $CVNBot3pid" for restarting.
- 11:09 rxy: "rxy@cvn-app5:~$ sudo kill $CVNBot4pid, $CVNBot14pid, $CVNBot17pid" for restarting.
- 18:58 rxy: rebooting cvn-app4.eqiad.wmflabs
- 00:21 rxy: "sudo kill $CVNBot16pid" for restarting.
- 21:34 Krinkle: cvn-app6 is now up and running. 1/10 bots from cvn-app4 (CVNBot2) has been migrated to it. No other bots are migrated yet as cvn-app6 does not yet have a dedicated IP and therefore is subject to the shared freenode limit for labs.
- 10:47 Krinkle: cvn-app6 (trusty) running stable. Migration from cvn-app4 to continue as planned.
- 10:47 Krinkle: Restarted CVNBot14 (#cvn-commons) - IRC connection was lost
- 02:00 Krinkle: Provision cvn-app6 based on infrastructure setup.yaml
- 17:26 Krinkle: Creating cvn-app6 (Ubuntu 14 Trusty, Size: Large) as migration target for cvn-app4 (*deprecated* Ubuntu 12 Precise, Size: Extra large)
- 11:24 rxy: setup CVNBot23 in cvn-app4
- 05:44 Krinkle: Edit CVNBot19 Projects.xml configuration: Correct projectName for test.wikidata.org to 'testwikidata.wikipedia' (per legacy wmgRC2UDPPrefix override)
- 05:14 Krinkle: rsync'ed CVNBot3 from cvn-app5 to cvn-app4
- 05:14 Krinkle: rsync'ed CVNBot2 from cvn-app5 to cvn-app4
- 05:14 Krinkle: rsync'ed CVNBot19 from cvn-app5 to cvn-app4
- 14:48 rxy: "sudo rm /var/lib/puppet/state/agent_catalog_run.lock" for fix to puppet failure.
- 21:58 Krinkle: Rebooting cvn-app5
- 19:45 Krinkle: CVN-ClerkBot no longer does !log to countervandalism.net. This is now handled by cvn-morebots and writes to the new log at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn/SAL.
- 19:35 rxy: Setting up Cubbie in cvn-app5.
- 23:57 Krinkle: Restore localised messages (nl) for CVNBot12
- 17:00 Krinkle: Unscheduled reboot of cvn-app5. Bots are having trouble writing to local disk.
- 19:03 Krinkle: Restarted cvn-app4. Somehow it became unreachable for over an hour.
- 18:11 Krinkle: Promoted Rxy from member to projectadmin
- 17:13 andrewbogott: rebooted cvn-app5 and cvn-app4 (not at the same time) because I suspect them of running amok
- 15:42 andrewbogott: migrating cvn-app4 to labvirt1001 in an attempt to redistribute load a bit
- 12:47 andrewbogott: migrating cvn-app4 and cvn-app5 to labvirt1005 and labvirt1006, respectively. Seems to be working now.
- 03:00 Krinkle: Archived cvn-apache5 access logs to /data/project/cvn-common/backup/cvn-apache5-accesslogs.tar.gz
- 02:54 Krinkle: https://cvn.wmflabs.org now points to cvn-apache8 (UbuntuTrusty; Apache 2.4). The old cvn-apache5 (UbuntuPrecise; Apache 2.2) will be deleted shortly after archiving web logs to NFS.
- 21:26 Krinkle: Apply role::labs::lvm::srv via Hiera
- 21:01 andrewbogott: cold-migrating cvn-app4 to labvirt1005
- 14:43 andrewbogott: moved cvn-apache5 to virt1007
- 01:51 Krinkle: Disabled role::labs::lvm::mnt, and enabled role::labs::lvm::srv instead (separate changes)
- 01:12 Krinkle: Enable role role::labs::lvm::mnt on all cvn instances
- 18:30 Krinkle: CVNBot1 (feeds #cvn-wp-en; migrated from rcbot) is now hosted in Wikimedia Labs, runs on cvn-app4.
- 04:12 Krinkle: Renamed KrinkleBot2 to CVNBot2 (assigned NickServ account CVN-Bots, shard0)
- 03:50 Krinkle: CVNBot13 (feeds #cvn-wp-da; migrated from KyluBot) is now hosted in Wikimedia Labs, runs on cvn-app5.
- 03:50 Krinkle: CVNBot12 (feeds #cvn-wp-nl; migrated from KrinkleBot7) is now hosted in Wikimedia Labs, runs on cvn-app4.
- 03:50 Krinkle: Installed pidentd package on cnv-app servers.
- 00:00 Krinkle: Migrated web services from pmtpa to eqiad. cvn-apache*.pmtpa.wmflabs can be deleted.
- 19:11 Krinkle: Migrated all applications from pmtpa to eqiad. cvn-app*.pmtpa can be shut down.
- 17:36 Krinkle: Installing packages on new eqiad instances (cvn-app4, cvn-app5, cvn-apache5)
- 05:34 Krinkle: Finished migration of all bots from the old and small app server (cvn-app1) to the large cvn-app2. Instance to be decomissioned.
- 05:26 Krinkle: Created cvn-app3 as the first cvn server in wmflabs-eqiad. Don't use for now (not yet set up with proper packages), but will replace cvn-app1 and cvn-app2 from pmtpa soon.
- 20:45 Krinkle: Setting up cvn-app2 as app server (same packages as on cvn-app1)
- 20:42 Krinkle: Deleted cvn-apache3, re-creating as cvn-apache4 with 'web' security group
- 17:52 Krinkle: Setting up cvn-apache3 as web server
- 23:34 Krinkle: Apache instances keeps going brain dead (time out, no significant spike in ganglia, just gradually going OOM). Rebooted again..
- 23:09 Krinkle: The https://cvn.wmflabs.org entry point now supports HTTPS (thanks YuviPanda!)
- 03:09 Krinkle: Installed the cvn-api web service on the cvn-apache2.wmflabs host http://cvn.wmflabs.org/api.php
- 03:08 Krinkle: The cvn-api web service from CVN at Toolserver is now in source control: https://github.com/countervandalism/cvn-api
- 05:16 Krinkle: Moving CVNBot14 from firstname.lastname@example.org to cvn-app1.wmflabs
- 17:29 Krinkle: Rebooted cvn-app1
- 23:23 Krinkle: Added seahorseruler to cvn access group.
- 14:03 matanya: rebooted by krinkle due to out of memory
- 13:25 matanya: migrating Cluekie from Toolserver to labs
- 21:40 Krinkle: CVNBot-nl and CVNBot-es were down because it was unable to open i18n files. Files restored from git, bots automatically started by stillalive.
- 23:12 Krinkle: Set up symlinks for cvn-config. Bot in #cvn-wp-nl now uses the Dutch msgs file, #cvn-wp-es uses Spanish msgs file.
- 23:07 Krinkle: Setting up checkout of https://github.com/countervandalism/infrastructure in /cvn/externals
- 22:11 Krinkle: cvn-app1 went OOM and kernel failed because of it (/home and /data unavailable). stillalive has auto-started the bots and we're back up.
- 00:19 Krinkle: Installing python-twisted-words and python-mysqldb from apt on cvn-app1 (for cvnclerkbot.py)
- 23:40 Krinkle: Installing mysql-server and mysql-client from apt on cvn-app1 (for cvnclerkbot.py)
- 23:06 Krinkle: Installing subversion from apt on cvn-app1 (for pywikipedia)
- 14:04 Krinkle: gluster had another hickup, /data became unaccessible ("Transport endpoint is not connected"). Bots down for almost 24 hours. Rebooting the instance fixed it.
- 16:55 Krinkle: Pool configuration pushed to github: https://github.com/countervandalism/stillalive/blob/master/localSettings-cvn.json
- 16:54 Krinkle: Add cvn-app1 to application pool.
- 16:12 Krinkle: Depooling cvn-apache1 from application
- 16:11 Krinkle: Installing php5-cli and mono-complete on cvn-app1
- 08:05 Krinkle: cvn-apache2 is getting too small for all the cvn apps (wasn't supposed to run them in the first place, just apache). Created instance cvn-app1 with more cpu/memory. Moving over application pool later today.
- 13:02 Krinkle: Installing package "mono-complete"
Archived at Special:PrefixIndex/Nova_Resource:Cvn/SAL/.