You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Tools/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)

30 November 2022

  • curprev 10:3910:39, 30 November 2022imported>Stashbot 32,271 bytes +428 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (bc3529d) - cookbook ran by arturo@nostromo

29 November 2022

9 November 2022

  • curprev 08:5908:59, 9 November 2022imported>Stashbot 31,763 bytes +124 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

5 November 2022

  • curprev 19:2819:28, 5 November 2022imported>Stashbot 31,639 bytes +396 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.err' -size +1G -exec truncate --size=1G {} \;

4 November 2022

  • curprev 20:4120:41, 4 November 2022imported>Stashbot 31,243 bytes +860 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.err' -not -newermt "Nov 1, 2021" -exec rm {} \;

1 November 2022

  • curprev 09:3709:37, 1 November 2022imported>Stashbot 30,383 bytes +150 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master (T322110) - cookbook ran by dcaro@vulcanus

26 October 2022

  • curprev 08:4508:45, 26 October 2022imported>Stashbot 30,233 bytes +112 dcaro: depooling and rebooting tools-sgeexec-10-22 to get nfs scratch working again

25 October 2022

  • curprev 16:1416:14, 25 October 2022imported>Stashbot 30,121 bytes +233 wm-bot2: Increased quotas by 5120 gigabytes - cookbook ran by fran@wmf3169

20 October 2022

18 October 2022

17 October 2022

14 October 2022

  • curprev 07:5407:54, 14 October 2022imported>Stashbot 28,770 bytes +211 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0cc020e) (T311466) - cookbook ran by taavi@runko

13 October 2022

12 October 2022

  • curprev 23:2523:25, 12 October 2022imported>Stashbot 28,498 bytes +885 bd808: Rebuilding all Toolforge docker images (T278436, T311466, T293552)

11 October 2022

  • curprev 13:5213:52, 11 October 2022imported>Stashbot 27,613 bytes +256 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:8574c36 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8574c36) - cookbook ran by taavi@runko

10 October 2022

  • curprev 19:3019:30, 10 October 2022imported>Stashbot 27,357 bytes +768 taavi: rebooting all k8s worker nodes to clean up labstore1006/7 remains

9 October 2022

  • curprev 17:2917:29, 9 October 2022imported>Stashbot 26,589 bytes +124 taavi: kill 10 idle tmux sessions of user 'hoi' on tools-sgebastion-10 T320352

7 October 2022

  • curprev 13:0213:02, 7 October 2022imported>Stashbot 26,465 bytes +121 taavi: taavi@cloudcontrol1005 ~ $ sudo mark_tool --disable oncall # T320240

6 October 2022

  • curprev 00:3900:39, 6 October 2022imported>Stashbot 26,344 bytes +356 bd808: Image rebuild failing with debian apt repo signature issue. Will investigate tomorrow. (T316554)

3 October 2022

  • curprev 14:3614:36, 3 October 2022imported>Stashbot 25,988 bytes +252 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/volume-admission:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (8da432b) - cookbook ran by taavi@runko

28 September 2022

22 September 2022

  • curprev 12:3012:30, 22 September 2022imported>Stashbot 25,345 bytes +221 taavi: add TheresNoTime to the 'toollabs-trusted' gerrit group T317438

10 September 2022

  • curprev 07:3907:39, 10 September 2022imported>Stashbot 25,124 bytes +105 wm-bot2: removing instance tools-prometheus-03 - cookbook ran by taavi@runko

7 September 2022

  • curprev 10:2210:22, 7 September 2022imported>Stashbot 25,019 bytes +134 dcaro: Pushing the new toolforge builder image based on the new 0.8 buildpacks (T316854)

6 September 2022

  • curprev 08:0608:06, 6 September 2022imported>Stashbot 24,885 bytes +178 dcaro_away: Published new toolforge-bullseye0-run and toolforge-bullseye0-build images for the toolforge buildpack builder (T316854)

25 August 2022

  • curprev 10:4010:40, 25 August 2022imported>Stashbot 24,707 bytes +158 taavi: tagged new version of the python39-web container with a shell implementation of webservice-runner T293552

24 August 2022

  • curprev 12:2012:20, 24 August 2022imported>Stashbot 24,549 bytes +230 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (eba66bc) - cookbook ran by taavi@runko

20 August 2022

18 August 2022

  • curprev 14:4514:45, 18 August 2022imported>Stashbot 23,900 bytes +207 andrewbogott: adding lucaswerkmeister as projectadmin (T314527)

17 August 2022

  • curprev 16:3416:34, 17 August 2022imported>Stashbot 23,693 bytes +210 taavi: kubectl sudo delete cm -n tool-wdml maintain-kubeusers # T315459

16 August 2022

  • curprev 17:2817:28, 16 August 2022imported>Stashbot 23,483 bytes +107 taavi: fail over docker-registry, tools-docker-registry-06->docker-registry-05

11 August 2022

  • curprev 16:5816:58, 11 August 2022imported>Stashbot 23,376 bytes +204 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

5 August 2022

  • curprev 15:0815:08, 5 August 2022imported>Stashbot 23,172 bytes +386 wm-bot2: removing grid node tools-sgewebgen-10-1.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

3 August 2022

  • curprev 15:5115:51, 3 August 2022imported>Stashbot 22,786 bytes +262 dhinus: recreated jobs-api pods to pick up new ConfigMap

20 July 2022

  • curprev 19:3119:31, 20 July 2022imported>Stashbot 22,524 bytes +234 taavi: reboot toolserver-proxy-01 to free up disk space probably held by stale file handles

19 July 2022

  • curprev 17:5317:53, 19 July 2022imported>Stashbot 22,290 bytes +587 wm-bot2: created node tools-sgeexec-10-21.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

17 July 2022

  • curprev 15:5215:52, 17 July 2022imported>Stashbot 21,703 bytes +383 wm-bot2: removing grid node tools-sgeexec-10-10.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

14 July 2022

13 July 2022

  • curprev 12:0912:09, 13 July 2022imported>Stashbot 21,256 bytes +123 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

11 July 2022

  • curprev 16:0616:06, 11 July 2022imported>Stashbot 21,133 bytes +170 wm-bot2: Increased quotas by {self.increases} (T312692) - cookbook ran by nskaggs@x1carbon

7 July 2022

  • curprev 07:3407:34, 7 July 2022imported>Stashbot 20,963 bytes +123 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

28 June 2022

  • curprev 17:3417:34, 28 June 2022imported>Stashbot 20,840 bytes +213 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master (T311538) - cookbook ran by dcaro@vulcanus

27 June 2022

  • curprev 18:1418:14, 27 June 2022imported>Stashbot 20,627 bytes +956 taavi: restart calico, appears to have got stuck after the ca replacement operation

23 June 2022

  • curprev 17:5117:51, 23 June 2022imported>Stashbot 19,671 bytes +1,622 wm-bot2: removing grid node tools-sgeexec-0941.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

22 June 2022

  • curprev 15:5415:54, 22 June 2022imported>Stashbot 18,049 bytes +524 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0917.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

21 June 2022

  • curprev 15:2315:23, 21 June 2022imported>Stashbot 17,525 bytes +524 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

3 June 2022

  • curprev 20:0720:07, 3 June 2022imported>Stashbot 17,001 bytes +2,706 wm-bot2: created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster

2 June 2022

  • curprev 22:2622:26, 2 June 2022imported>Stashbot 14,295 bytes +1,706 bd808: Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler

1 June 2022

  • curprev 11:1811:18, 1 June 2022imported>Stashbot 12,589 bytes +77 taavi: depool and remove tools-sgeexec-09[07-14]

31 May 2022

  • curprev 16:5116:51, 31 May 2022imported>Stashbot 12,512 bytes +106 taavi: delete tools-sgeexec-0904 for T309525 experimentation

30 May 2022

  • curprev 08:2408:24, 30 May 2022imported>Stashbot 12,406 bytes +109 taavi: depool tools-sgeexec-[0901-0909] (7 nodes total) T277653

26 May 2022

  • curprev 15:3915:39, 26 May 2022imported>Stashbot 12,297 bytes +211 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (e6fa299) (T309146) - cookbook ran by taavi@runko

22 May 2022

  • curprev 17:0417:04, 22 May 2022imported>Stashbot 12,086 bytes +245 taavi: failover tools-redis to the updated cluster T278541

16 May 2022

  • curprev 14:0214:02, 16 May 2022imported>Stashbot 11,841 bytes +183 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (7037eca) - cookbook ran by taavi@runko

14 May 2022

  • curprev 10:4710:47, 14 May 2022imported>Stashbot 11,658 bytes +80 taavi: hard reboot unresponsible tools-sgeexec-0940

12 May 2022

10 May 2022

  • curprev 15:1815:18, 10 May 2022imported>Stashbot 11,163 bytes +162 taavi: depool tools-k8s-worker-42 for experiments

6 May 2022

  • curprev 19:4619:46, 6 May 2022imported>Stashbot 11,001 bytes +155 bd808: Rebuilt toolforge-perl532-sssd-base & toolforge-perl532-sssd-web to add liblocale-codes-perl (T307812)

5 May 2022

  • curprev 17:2817:28, 5 May 2022imported>Stashbot 10,846 bytes +89 taavi: deploy tools-webservice 0.83 T307693

3 May 2022

  • curprev 08:2008:20, 3 May 2022imported>Stashbot 10,757 bytes +123 taavi: redis: start replication from the old cluster to the new one (T278541)

2 May 2022

25 April 2022

  • curprev 14:5614:56, 25 April 2022imported>Stashbot 10,547 bytes +180 bd808: Rebuilding all docker images to pick up toolforge-webservice v0.82 (T214343)

23 April 2022

  • curprev 16:5116:51, 23 April 2022imported>Stashbot 10,367 bytes +160 bd808: Built new perl532-sssd/{base,web} images and pushed to registry (T214343)

20 April 2022

  • curprev 16:5816:58, 20 April 2022imported>Stashbot 10,207 bytes +345 taavi: reboot toolserver-proxy-01 to free up disk space from stale file handles(?)

16 April 2022

  • curprev 18:5318:53, 16 April 2022imported>Stashbot 9,862 bytes +187 wm-bot: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics (2c485e9) - cookbook ran by taavi@runko

12 April 2022

  • curprev 21:3221:32, 12 April 2022imported>Stashbot 9,675 bytes +257 bd808: Added komla to Gerrit group 'toollabs-trusted' (T305986)

10 April 2022

  • curprev 18:4318:43, 10 April 2022imported>Stashbot 9,418 bytes +152 taavi: deleted `/tmp/dwl02.out-20210915` on tools-sgebastion-07 (not touched since september, taking up 1.3G of disk space)

9 April 2022

  • curprev 15:3015:30, 9 April 2022imported>Stashbot 9,266 bytes +109 taavi: manually prune user.log on tools-prometheus-03 to free up some space on /

8 April 2022

  • curprev 10:4410:44, 8 April 2022imported>Stashbot 9,157 bytes +90 arturo: disabled debug mode on the k8s jobs-emailer component

5 April 2022

  • curprev 07:5207:52, 5 April 2022imported>Stashbot 9,067 bytes +483 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo

4 April 2022

  • curprev 17:0517:05, 4 April 2022imported>Stashbot 8,584 bytes +529 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo

28 March 2022

  • curprev 09:3209:32, 28 March 2022imported>Stashbot 8,055 bytes +179 wm-bot: cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo

15 March 2022

  • curprev 16:5716:57, 15 March 2022imported>Stashbot 7,876 bytes +410 wm-bot: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (084ee51) - cookbook ran by arturo@nostromo

14 March 2022

  • curprev 11:4411:44, 14 March 2022imported>Stashbot 7,466 bytes +270 arturo: deploy jobs-framework-emailer 9470a5f339fd5a44c97c69ce97239aef30f5ee41 (T286135)

10 March 2022

  • curprev 09:4209:42, 10 March 2022imported>Stashbot 7,196 bytes +99 arturo: cleaned grid queue error state @ tools-sgewebgrid-generic-0902

1 March 2022

  • curprev 13:4113:41, 1 March 2022imported>Stashbot 7,097 bytes +300 dcaro: rebooting tools-sgeexec-0916 to clear any state (T302702)

28 February 2022

17 February 2022

16 February 2022

10 February 2022

9 February 2022

  • curprev 19:2919:29, 9 February 2022imported>Stashbot 4,202 bytes +652 taavi: installed tools-puppetdb-1, not configured on puppetmaster side yet T214427

8 February 2022

  • curprev 15:0115:01, 8 February 2022imported>Nhatminh01 3,550 bytes −239,922 Replaced content with "=== 2022-02-07 === * 17:37 taavi: generated authdns_acmechief ssh key and stored password in a text file in local labs/private repository (T288406) * 12:52 taavi: updated maintain-kubeusers for T301081 === 2022-02-04 === * 22:33 taavi: `root@tools-sgebastion-10:/data/project/ru_monuments/.kube# mv config old_config` # experimenting with T301015 * 21:36 taavi: clear error state from some webgrid nodes === 2022-02-03..."

7 February 2022

  • curprev 17:3717:37, 7 February 2022imported>Stashbot 243,472 bytes +237 taavi: generated authdns_acmechief ssh key and stored password in a text file in local labs/private repository (T288406)

4 February 2022

  • curprev 22:3322:33, 4 February 2022imported>Stashbot 243,235 bytes +220 taavi: `root@tools-sgebastion-10:/data/project/ru_monuments/.kube# mv config old_config` # experimenting with T301015

3 February 2022

  • curprev 09:0609:06, 3 February 2022imported>Stashbot 243,015 bytes +191 taavi: run `sudo apt-get clean` on login-buster/dev-buster to clean up disk space

30 January 2022

  • curprev 14:4114:41, 30 January 2022imported>Stashbot 242,824 bytes +248 taavi: created a neutron port with ip 172.16.2.46 for a service ip for toolforge redis automatic failover T278541

26 January 2022

  • curprev 18:3318:33, 26 January 2022imported>Stashbot 242,576 bytes +1,010 wm-bot: depooled grid node tools-sgeexec-10-10 - cookbook ran by arturo@nostromo

25 January 2022

  • curprev 11:5011:50, 25 January 2022imported>Stashbot 241,566 bytes +226 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

24 January 2022

  • curprev 17:4417:44, 24 January 2022imported>Stashbot 241,340 bytes +209 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

20 January 2022

  • curprev 17:0517:05, 20 January 2022imported>Stashbot 241,131 bytes +205 arturo: drop 9 of the 10 buster exec nodes created earlier. They didn't get DNS records

19 January 2022

  • curprev 17:3417:34, 19 January 2022imported>Stashbot 240,926 bytes +161 andrewbogott: rebooting tools-sgeexec-0913.tools.eqiad1.wikimedia.cloud to recover from (presumed) fallout from the scratch/nfs move

14 January 2022

  • curprev 19:0919:09, 14 January 2022imported>Stashbot 240,765 bytes +131 taavi: set /var/run/lighttpd as world-writable on all lighttpd webgrid nodes, T299243

12 January 2022

  • curprev 11:2711:27, 12 January 2022imported>Stashbot 240,634 bytes +218 arturo: created puppet prefix `tools-sgeweblight`, drop `tools-sgeweblig`

4 January 2022

  • curprev 17:1817:18, 4 January 2022imported>Stashbot 240,416 bytes +154 bd808: tools-acme-chief-01: sudo service acme-chief restart

31 December 2021

  • curprev 19:4819:48, 31 December 2021imported>Stashbot 240,262 bytes +110 taavi: reset grid error status on webgrid-lighttpd@tools-sgewebgrid-lighttpd-0915

28 December 2021

24 December 2021

23 December 2021

20 December 2021

14 December 2021

  • curprev 09:4609:46, 14 December 2021imported>Stashbot 232,768 bytes +126 majavah: testing delete-crashing-pods emailer component with a test tool T292925
(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)