You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Tools/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

16 March 2021

12 March 2021

11 March 2021

10 March 2021

  • curprev 10:5610:56, 10 March 2021imported>Stashbot 198,019 bytes +96 arturo: briefly stopped VM tools-k8s-etcd-7 to disable VMX cpu flag

9 March 2021

  • curprev 13:3113:31, 9 March 2021imported>Stashbot 197,923 bytes +261 arturo: hard-reboot tools-docker-registry-04 because issues related to T276922

5 March 2021

4 March 2021

  • curprev 11:2511:25, 4 March 2021imported>Stashbot 197,523 bytes +219 arturo: rebooted tools-sgewebgrid-generic-0901, repool it again

3 March 2021

  • curprev 15:1715:17, 3 March 2021imported>Stashbot 197,304 bytes +471 arturo: shutting down tools-sgebastion-07 in an attempt to fix nova state and finish hypervisor migration

2 March 2021

  • curprev 15:2415:24, 2 March 2021imported>Stashbot 196,833 bytes +238 bstorm: depooling tools-sgewebgrid-lighttpd-0914.tools.eqiad.wmflabs for reboot. It isn't communicating right

27 February 2021

  • curprev 02:2302:23, 27 February 2021imported>Stashbot 196,595 bytes +252 bstorm: deployed typo fix to maintain-kubeusers in an innocent effort to make the weekend better T275910

26 February 2021

  • curprev 22:0422:04, 26 February 2021imported>Stashbot 196,343 bytes +338 bstorm: cleaned up grid jobs 1230666,1908277,1908299,2441500,2441513

24 February 2021

  • curprev 18:3018:30, 24 February 2021imported>Stashbot 196,005 bytes +212 bd808: `sudo wmcs-openstack role remove --user zfilipin --project tools user` T267313

23 February 2021

  • curprev 23:1123:11, 23 February 2021imported>Stashbot 195,793 bytes +227 bstorm: draining a bunch of k8s workers to clean up after dumps changes T272397

22 February 2021

19 February 2021

  • curprev 12:3112:31, 19 February 2021imported>Stashbot 194,925 bytes +100 arturo: deploying new version of toolforge ingress admission controller

17 February 2021

  • curprev 21:2621:26, 17 February 2021imported>Stashbot 194,825 bytes +118 bstorm: deleted tools-puppetdb-01 since it is unused at this time (and undersized anyway)

4 February 2021

26 January 2021

  • curprev 16:2716:27, 26 January 2021imported>Stashbot 194,636 bytes +110 bd808: Hard reboot of tools-sgeexec-0906 via Horizon for T272978

22 January 2021

  • curprev 09:5909:59, 22 January 2021imported>Stashbot 194,526 bytes +146 dcaro: added the record redis.svc.tools.eqiad1.wikimedia.cloud pointing to tools-redis1003 (T272679)

21 January 2021

19 January 2021

  • curprev 22:5722:57, 19 January 2021imported>Stashbot 194,278 bytes +503 bstorm: truncated 75GB error log /data/project/robokobot/virgule.err T272247

14 January 2021

  • curprev 20:5620:56, 14 January 2021imported>Stashbot 193,775 bytes +367 bstorm: setting bastions to have mostly-uncapped egress network and 40MBps nfs_read for better shared use

13 January 2021

  • curprev 10:0210:02, 13 January 2021imported>Stashbot 193,408 bytes +107 arturo: delete floating IP allocation 185.15.56.245 (T271867)

12 January 2021

  • curprev 18:1618:16, 12 January 2021imported>Stashbot 193,301 bytes +134 bstorm: deleted wedged CSR tool-adhs-wde to get maintain-kubeusers working again T271842

5 January 2021

  • curprev 18:4918:49, 5 January 2021imported>Stashbot 193,167 bytes +134 bstorm: changing the limits on k8s etcd nodes again, so disabling puppet on them T267966

4 January 2021

  • curprev 18:2118:21, 4 January 2021imported>Stashbot 193,033 bytes +191 bstorm: ran 'sudo systemctl stop getty@ttyS1.service && sudo systemctl disable getty@ttyS1.service' on tools-k8s-etcd-5 I have no idea why that keeps coming back.

22 December 2020

  • curprev 18:2218:22, 22 December 2020imported>Stashbot 192,842 bytes +190 bstorm: rebooting the grid master because it is misbehaving following the NFS outage

18 December 2020

17 December 2020

  • curprev 21:4221:42, 17 December 2020imported>Stashbot 192,543 bytes +2,476 bstorm: doing the same procedure to increase the timeouts more T267966

11 December 2020

  • curprev 18:2918:29, 11 December 2020imported>Stashbot 190,067 bytes +1,158 bstorm: certificatesigningrequest.certificates.k8s.io "tool-production-error-tasks-metrics" deleted to stop maintain-kubeusers issues

10 December 2020

8 December 2020

  • curprev 19:0119:01, 8 December 2020imported>Stashbot 187,730 bytes +140 bstorm: pushed updated calico node image (v3.14.0) to internal docker registry as well T269016

7 December 2020

  • curprev 22:5622:56, 7 December 2020imported>Stashbot 187,590 bytes +182 bstorm: pushed updated local copies of the typha, calico-cni and calico-pod2daemon-flexvol images to the tools internal registry T269016

3 December 2020

  • curprev 09:1809:18, 3 December 2020imported>Stashbot 187,408 bytes +312 arturo: restarted kubelet systemd service on tools-k8s-worker-38. Node was NotReady, complaining about 'use of closed network connection'

28 November 2020

  • curprev 23:3523:35, 28 November 2020imported>Stashbot 187,096 bytes +326 Krenair: Re-scheduled 4 continuous jobs from tools-sgeexec-0908 as it appears to be broken, at about 23:20 UTC

24 November 2020

10 November 2020

2 November 2020

29 October 2020

  • curprev 21:3321:33, 29 October 2020imported>Stashbot 186,307 bytes +489 legoktm: published docker-registry.tools.wmflabs.org/toolbeta-test image (T265681)

28 October 2020

  • curprev 23:4223:42, 28 October 2020imported>Stashbot 185,818 bytes +363 bstorm: dramatically elevated the egress cap on tools-k8s-ingress nodes that were affected by the NFS settings T266506

23 October 2020

  • curprev 22:2222:22, 23 October 2020imported>Stashbot 185,455 bytes +115 legoktm: imported pack_0.14.2-1_amd64.deb into buster-tools (T266270)

21 October 2020

  • curprev 17:5817:58, 21 October 2020imported>Stashbot 185,340 bytes +141 legoktm: pushed toolforge-buster0-{build,run}:latest images to docker registry

15 October 2020

  • curprev 22:0022:00, 15 October 2020imported>Stashbot 185,199 bytes +355 bstorm: manually removing nscd from tools-sgebastion-08 and running puppet

14 October 2020

  • curprev 21:0021:00, 14 October 2020imported>Stashbot 184,844 bytes +753 andrewbogott: repooling tools-sgewebgrid-generic-0901 and tools-sgewebgrid-lighttpd-0915

10 October 2020

  • curprev 17:0717:07, 10 October 2020imported>Stashbot 184,091 bytes +123 bstorm: cleared errors on tools-sgeexec-0912.tools.eqiad.wmflabs to get the queue moving again

8 October 2020

  • curprev 17:0717:07, 8 October 2020imported>Stashbot 183,968 bytes +103 bstorm: rebuilding docker images with locales-all T263339

6 October 2020

2 October 2020

  • curprev 21:0921:09, 2 October 2020imported>Stashbot 183,631 bytes +281 bstorm: rebooting tools-k8s-worker-70 because it seems to be unable to recover from an old NFS disconnect

1 October 2020

30 September 2020

23 September 2020

  • curprev 21:3821:38, 23 September 2020imported>Stashbot 182,914 bytes +111 bstorm: ran an 'apt clean' across the fleet to get ahead of the new locale install

18 September 2020

  • curprev 19:4119:41, 18 September 2020imported>Stashbot 182,803 bytes +1,384 andrewbogott: repooling tools-k8s-worker-30, 33, 34, 57, 60
  • curprev 01:0001:00, 18 September 2020imported>Stashbot 181,419 bytes +1,961 andrewbogott: depooling tools-sgeexec-0917, tools-sgeexec-0918, tools-sgeexec-0919, tools-sgeexec-0920 for flavor update

16 September 2020

  • curprev 23:2023:20, 16 September 2020imported>Stashbot 179,458 bytes +512 andrewbogott: repooled tools-sgeexec-0941 and tools-sgeexec-0939 for move to ceph

10 September 2020

9 September 2020

  • curprev 11:1211:12, 9 September 2020imported>Stashbot 178,587 bytes +560 arturo: new ingress nodes added to the cluster, and tainted/labeled per the docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Deploying#ingress_nodes (T250172)

8 September 2020

2 September 2020

31 August 2020

30 August 2020

26 August 2020

  • curprev 21:0821:08, 26 August 2020imported>Stashbot 176,574 bytes +293 bd808: Disabled puppet on tools-proxy-06 to test fixes for a bug in the new T251628 code

25 August 2020

  • curprev 19:3819:38, 25 August 2020imported>Stashbot 176,281 bytes +648 andrewbogott: deleting tools-sgeexec-0943.tools.eqiad.wmflabs, tools-sgeexec-0944.tools.eqiad.wmflabs, tools-sgeexec-0945.tools.eqiad.wmflabs, tools-sgeexec-0946.tools.eqiad.wmflabs, tools-sgeexec-0948.tools.eqiad.wmflabs, tools-sgeexec-0949.tools.eqiad.wmflabs, tools-sgeexec-0953.tools.eqiad.wmflabs — they are broken and we're not very curious why; will retry this exercise when everything is standardized on

19 August 2020

  • curprev 21:2921:29, 19 August 2020imported>Stashbot 175,633 bytes +440 andrewbogott: shutting down and removing tools-k8s-worker-20 through tools-k8s-worker-29; this load can now be handled by new nodes on ceph hosts

18 August 2020

  • curprev 15:2415:24, 18 August 2020imported>Stashbot 175,193 bytes +117 bd808: Rebuilding all Docker containers to pick up newest versions of installed packages

30 July 2020

  • curprev 16:2816:28, 30 July 2020imported>Stashbot 175,076 bytes +152 andrewbogott: added new xlarge ceph-hosted worker nodes: tools-k8s-worker-61, 62, 63, 64, 65, 66. T258663

29 July 2020

  • curprev 23:2423:24, 29 July 2020imported>Stashbot 174,924 bytes +216 bd808: Pushed a copy of docker-registry.wikimedia.org/wikimedia-jessie:latest to docker-registry.tools.wmflabs.org/wikimedia-jessie:latest in preparation for the upstream image going away

24 July 2020

  • curprev 22:3322:33, 24 July 2020imported>Stashbot 174,708 bytes +426 bd808: Removed a few more ancient docker images: grrrit, jessie-toollabs, and nagf

22 July 2020

  • curprev 23:2423:24, 22 July 2020imported>Stashbot 174,282 bytes +1,162 bstorm: created server group 'tools-k8s-worker' to create any new worker nodes in so that they have a low chance of being scheduled together by openstack unless it is necessary T258663

21 July 2020

  • curprev 16:0916:09, 21 July 2020imported>Stashbot 173,120 bytes +212 bstorm: rebooting tools-sgegrid-shadow to remount NFS correctly

17 July 2020

  • curprev 16:4716:47, 17 July 2020imported>Stashbot 172,908 bytes +235 bd808: Enabled Puppet on tools-proxy-06 following successful test (T102367)

15 July 2020

  • curprev 23:1123:11, 15 July 2020imported>Stashbot 172,673 bytes +117 bd808: Removed ssh root key for valhallasw from project hiera (T255697)

9 July 2020

  • curprev 18:5318:53, 9 July 2020imported>Stashbot 172,556 bytes +115 bd808: Updating git-review to 1.27 via clush across cluster (T257496)

8 July 2020

  • curprev 11:1611:16, 8 July 2020imported>Stashbot 172,441 bytes +299 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 -- important change to front-proxy (T234617)

7 July 2020

  • curprev 23:2223:22, 7 July 2020imported>Stashbot 172,142 bytes +655 bd808: Rebuilding all Docker images to pick up webservice v0.73 (T234617, T257229)

6 July 2020

  • curprev 11:5411:54, 6 July 2020imported>Stashbot 171,487 bytes +354 arturo: briefly point DNS tools.wmflabs.org A record to 185.15.56.60 (tools-legacy-redirector) and then switch back to 185.15.56.11 (tools-proxy-05). The legacy redirector does HTTP/307 (T247236)

1 July 2020

  • curprev 11:1911:19, 1 July 2020imported>Stashbot 171,133 bytes +215 arturo: cleanup exim email queue (4 frozen messages)

30 June 2020

  • curprev 11:1811:18, 30 June 2020imported>Stashbot 170,918 bytes +123 arturo: set some hiera keys for mtail in puppet prefix `tools-mail` (T256737)

29 June 2020

25 June 2020

  • curprev 21:5021:50, 25 June 2020imported>Stashbot 170,486 bytes +283 zhuyifei1999_: re-enabling puppet on tools-sgebastion-09 T256426

24 June 2020

  • curprev 12:3612:36, 24 June 2020imported>Stashbot 170,203 bytes +252 arturo: live-hacking puppetmaster with exim prometheus stuff (T175964)

23 June 2020

  • curprev 17:5517:55, 23 June 2020imported>Stashbot 169,951 bytes +237 arturo: killed procs for users `hamishz` and `msyn` which apparently were tools that should be running in the grid / kubernetes instead

17 June 2020

  • curprev 10:4010:40, 17 June 2020imported>Stashbot 169,714 bytes +162 arturo: created VM tools-legacy-redirector, with the corresponding puppet prefix (T247236, T234617)

16 June 2020

  • curprev 23:0123:01, 16 June 2020imported>Stashbot 169,552 bytes +357 bd808: Building new Docker images to pick up webservice 0.72

15 June 2020

  • curprev 21:2821:28, 15 June 2020imported>Stashbot 169,195 bytes +347 bstorm_: cleaned up killgridjobs.sh on the tools bastions T157792

12 June 2020

  • curprev 13:1313:13, 12 June 2020imported>Stashbot 168,848 bytes +192 arturo: live-hacking session in the puppetmaster ended
  • curprev 00:1600:16, 12 June 2020imported>Stashbot 168,656 bytes +227 bstorm_: remounted NFS for tools-k8s-control-3 and tools-acme-chief-01

4 June 2020

  • curprev 13:3213:32, 4 June 2020imported>Stashbot 168,429 bytes +104 bd808: Manually restored /etc/haproxy/conf.d/elastic.cfg on tools-elastic-*

2 June 2020

  • curprev 12:2312:23, 2 June 2020imported>Stashbot 168,325 bytes +441 arturo: renewed TLS cert for k8s metrics-server (T250874) following docs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Certificates#internal_API_access

1 June 2020

  • curprev 23:5123:51, 1 June 2020imported>Stashbot 167,884 bytes +112 bstorm_: refreshed certs for the custom webhook controllers on the k8s cluster T250874
  • curprev 00:3900:39, 1 June 2020imported>Stashbot 167,772 bytes +206 bd808: Ugh. Prior SAL message was about tools-sgeexec-0940

29 May 2020

  • curprev 19:3719:37, 29 May 2020imported>Stashbot 167,566 bytes +160 bstorm_: adding docker image for paws-public docker-registry.tools.wmflabs.org/paws-public-nginx:openresty T252217

28 May 2020

  • curprev 21:1921:19, 28 May 2020imported>Stashbot 167,406 bytes +953 bd808: Killed 7 python processes run by user 'mattho69' on login.toolforge.org

27 May 2020

  • curprev 17:2317:23, 27 May 2020imported>Stashbot 166,453 bytes +160 bstorm_: deleting "tools-k8s-worker-20", "tools-k8s-worker-19", "tools-k8s-worker-18", "tools-k8s-worker-17", "tools-k8s-worker-16"

26 May 2020

  • curprev 18:4518:45, 26 May 2020imported>Stashbot 166,293 bytes +242 bstorm_: upgrading maintain-kubeusers to match what is in toolsbeta T246059 T211096

22 May 2020

  • curprev 20:0020:00, 22 May 2020imported>Stashbot 166,051 bytes +227 bstorm_: rebooted tools-sgebastion-07 to clear up tmp file problems with 10 min warning

21 May 2020

  • curprev 22:4022:40, 21 May 2020imported>Stashbot 165,824 bytes +285 bd808: Rebuilding all Docker containers for tools-webservice 0.70 (T252700)

20 May 2020

  • curprev 09:5909:59, 20 May 2020imported>Stashbot 165,539 bytes +896 arturo: now running tesseract-ocr v4.1.1-2~bpo9+1 in the Toolforge grid (T247422)

19 May 2020

  • curprev 17:0017:00, 19 May 2020imported>Stashbot 164,643 bytes +171 bstorm_: deleting/restarting the paws db-proxy pod because it cannot connect to the replicas...and I'm hoping that's due to depooling and such

13 May 2020

  • curprev 18:1418:14, 13 May 2020imported>Stashbot 164,472 bytes +254 bstorm_: upgrading calico to 3.14.0 with typha enabled in Toolforge K8s T250863

9 May 2020

  • curprev 00:2800:28, 9 May 2020imported>Stashbot 164,218 bytes +332 bstorm_: added nfs.* to ignored_fs_types for the prometheus::node_exporter params in project hiera T252260
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)