You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Tools/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

23 September 2020

  • curprev 21:3821:38, 23 September 2020imported>Stashbot 182,914 bytes +111 bstorm: ran an 'apt clean' across the fleet to get ahead of the new locale install

18 September 2020

  • curprev 19:4119:41, 18 September 2020imported>Stashbot 182,803 bytes +1,384 andrewbogott: repooling tools-k8s-worker-30, 33, 34, 57, 60
  • curprev 01:0001:00, 18 September 2020imported>Stashbot 181,419 bytes +1,961 andrewbogott: depooling tools-sgeexec-0917, tools-sgeexec-0918, tools-sgeexec-0919, tools-sgeexec-0920 for flavor update

16 September 2020

  • curprev 23:2023:20, 16 September 2020imported>Stashbot 179,458 bytes +512 andrewbogott: repooled tools-sgeexec-0941 and tools-sgeexec-0939 for move to ceph

10 September 2020

9 September 2020

  • curprev 11:1211:12, 9 September 2020imported>Stashbot 178,587 bytes +560 arturo: new ingress nodes added to the cluster, and tainted/labeled per the docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Deploying#ingress_nodes (T250172)

8 September 2020

2 September 2020

31 August 2020

30 August 2020

26 August 2020

  • curprev 21:0821:08, 26 August 2020imported>Stashbot 176,574 bytes +293 bd808: Disabled puppet on tools-proxy-06 to test fixes for a bug in the new T251628 code

25 August 2020

  • curprev 19:3819:38, 25 August 2020imported>Stashbot 176,281 bytes +648 andrewbogott: deleting tools-sgeexec-0943.tools.eqiad.wmflabs, tools-sgeexec-0944.tools.eqiad.wmflabs, tools-sgeexec-0945.tools.eqiad.wmflabs, tools-sgeexec-0946.tools.eqiad.wmflabs, tools-sgeexec-0948.tools.eqiad.wmflabs, tools-sgeexec-0949.tools.eqiad.wmflabs, tools-sgeexec-0953.tools.eqiad.wmflabs — they are broken and we're not very curious why; will retry this exercise when everything is standardized on

19 August 2020

  • curprev 21:2921:29, 19 August 2020imported>Stashbot 175,633 bytes +440 andrewbogott: shutting down and removing tools-k8s-worker-20 through tools-k8s-worker-29; this load can now be handled by new nodes on ceph hosts

18 August 2020

  • curprev 15:2415:24, 18 August 2020imported>Stashbot 175,193 bytes +117 bd808: Rebuilding all Docker containers to pick up newest versions of installed packages

30 July 2020

  • curprev 16:2816:28, 30 July 2020imported>Stashbot 175,076 bytes +152 andrewbogott: added new xlarge ceph-hosted worker nodes: tools-k8s-worker-61, 62, 63, 64, 65, 66. T258663

29 July 2020

  • curprev 23:2423:24, 29 July 2020imported>Stashbot 174,924 bytes +216 bd808: Pushed a copy of docker-registry.wikimedia.org/wikimedia-jessie:latest to docker-registry.tools.wmflabs.org/wikimedia-jessie:latest in preparation for the upstream image going away

24 July 2020

  • curprev 22:3322:33, 24 July 2020imported>Stashbot 174,708 bytes +426 bd808: Removed a few more ancient docker images: grrrit, jessie-toollabs, and nagf

22 July 2020

  • curprev 23:2423:24, 22 July 2020imported>Stashbot 174,282 bytes +1,162 bstorm: created server group 'tools-k8s-worker' to create any new worker nodes in so that they have a low chance of being scheduled together by openstack unless it is necessary T258663

21 July 2020

  • curprev 16:0916:09, 21 July 2020imported>Stashbot 173,120 bytes +212 bstorm: rebooting tools-sgegrid-shadow to remount NFS correctly

17 July 2020

  • curprev 16:4716:47, 17 July 2020imported>Stashbot 172,908 bytes +235 bd808: Enabled Puppet on tools-proxy-06 following successful test (T102367)

15 July 2020

  • curprev 23:1123:11, 15 July 2020imported>Stashbot 172,673 bytes +117 bd808: Removed ssh root key for valhallasw from project hiera (T255697)

9 July 2020

  • curprev 18:5318:53, 9 July 2020imported>Stashbot 172,556 bytes +115 bd808: Updating git-review to 1.27 via clush across cluster (T257496)

8 July 2020

  • curprev 11:1611:16, 8 July 2020imported>Stashbot 172,441 bytes +299 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 -- important change to front-proxy (T234617)

7 July 2020

  • curprev 23:2223:22, 7 July 2020imported>Stashbot 172,142 bytes +655 bd808: Rebuilding all Docker images to pick up webservice v0.73 (T234617, T257229)

6 July 2020

  • curprev 11:5411:54, 6 July 2020imported>Stashbot 171,487 bytes +354 arturo: briefly point DNS tools.wmflabs.org A record to 185.15.56.60 (tools-legacy-redirector) and then switch back to 185.15.56.11 (tools-proxy-05). The legacy redirector does HTTP/307 (T247236)

1 July 2020

  • curprev 11:1911:19, 1 July 2020imported>Stashbot 171,133 bytes +215 arturo: cleanup exim email queue (4 frozen messages)

30 June 2020

  • curprev 11:1811:18, 30 June 2020imported>Stashbot 170,918 bytes +123 arturo: set some hiera keys for mtail in puppet prefix `tools-mail` (T256737)

29 June 2020

25 June 2020

  • curprev 21:5021:50, 25 June 2020imported>Stashbot 170,486 bytes +283 zhuyifei1999_: re-enabling puppet on tools-sgebastion-09 T256426

24 June 2020

  • curprev 12:3612:36, 24 June 2020imported>Stashbot 170,203 bytes +252 arturo: live-hacking puppetmaster with exim prometheus stuff (T175964)

23 June 2020

  • curprev 17:5517:55, 23 June 2020imported>Stashbot 169,951 bytes +237 arturo: killed procs for users `hamishz` and `msyn` which apparently were tools that should be running in the grid / kubernetes instead

17 June 2020

  • curprev 10:4010:40, 17 June 2020imported>Stashbot 169,714 bytes +162 arturo: created VM tools-legacy-redirector, with the corresponding puppet prefix (T247236, T234617)

16 June 2020

  • curprev 23:0123:01, 16 June 2020imported>Stashbot 169,552 bytes +357 bd808: Building new Docker images to pick up webservice 0.72

15 June 2020

  • curprev 21:2821:28, 15 June 2020imported>Stashbot 169,195 bytes +347 bstorm_: cleaned up killgridjobs.sh on the tools bastions T157792

12 June 2020

  • curprev 13:1313:13, 12 June 2020imported>Stashbot 168,848 bytes +192 arturo: live-hacking session in the puppetmaster ended
  • curprev 00:1600:16, 12 June 2020imported>Stashbot 168,656 bytes +227 bstorm_: remounted NFS for tools-k8s-control-3 and tools-acme-chief-01

4 June 2020

  • curprev 13:3213:32, 4 June 2020imported>Stashbot 168,429 bytes +104 bd808: Manually restored /etc/haproxy/conf.d/elastic.cfg on tools-elastic-*

2 June 2020

  • curprev 12:2312:23, 2 June 2020imported>Stashbot 168,325 bytes +441 arturo: renewed TLS cert for k8s metrics-server (T250874) following docs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Certificates#internal_API_access

1 June 2020

  • curprev 23:5123:51, 1 June 2020imported>Stashbot 167,884 bytes +112 bstorm_: refreshed certs for the custom webhook controllers on the k8s cluster T250874
  • curprev 00:3900:39, 1 June 2020imported>Stashbot 167,772 bytes +206 bd808: Ugh. Prior SAL message was about tools-sgeexec-0940

29 May 2020

  • curprev 19:3719:37, 29 May 2020imported>Stashbot 167,566 bytes +160 bstorm_: adding docker image for paws-public docker-registry.tools.wmflabs.org/paws-public-nginx:openresty T252217

28 May 2020

  • curprev 21:1921:19, 28 May 2020imported>Stashbot 167,406 bytes +953 bd808: Killed 7 python processes run by user 'mattho69' on login.toolforge.org

27 May 2020

  • curprev 17:2317:23, 27 May 2020imported>Stashbot 166,453 bytes +160 bstorm_: deleting "tools-k8s-worker-20", "tools-k8s-worker-19", "tools-k8s-worker-18", "tools-k8s-worker-17", "tools-k8s-worker-16"

26 May 2020

  • curprev 18:4518:45, 26 May 2020imported>Stashbot 166,293 bytes +242 bstorm_: upgrading maintain-kubeusers to match what is in toolsbeta T246059 T211096

22 May 2020

  • curprev 20:0020:00, 22 May 2020imported>Stashbot 166,051 bytes +227 bstorm_: rebooted tools-sgebastion-07 to clear up tmp file problems with 10 min warning

21 May 2020

  • curprev 22:4022:40, 21 May 2020imported>Stashbot 165,824 bytes +285 bd808: Rebuilding all Docker containers for tools-webservice 0.70 (T252700)

20 May 2020

  • curprev 09:5909:59, 20 May 2020imported>Stashbot 165,539 bytes +896 arturo: now running tesseract-ocr v4.1.1-2~bpo9+1 in the Toolforge grid (T247422)

19 May 2020

  • curprev 17:0017:00, 19 May 2020imported>Stashbot 164,643 bytes +171 bstorm_: deleting/restarting the paws db-proxy pod because it cannot connect to the replicas...and I'm hoping that's due to depooling and such

13 May 2020

  • curprev 18:1418:14, 13 May 2020imported>Stashbot 164,472 bytes +254 bstorm_: upgrading calico to 3.14.0 with typha enabled in Toolforge K8s T250863

9 May 2020

  • curprev 00:2800:28, 9 May 2020imported>Stashbot 164,218 bytes +332 bstorm_: added nfs.* to ignored_fs_types for the prometheus::node_exporter params in project hiera T252260

7 May 2020

  • curprev 21:5121:51, 7 May 2020imported>Stashbot 163,886 bytes +245 bstorm_: rebuilding the docker images for Toolforge k8s

6 May 2020

  • curprev 21:2021:20, 6 May 2020imported>Stashbot 163,641 bytes +509 bd808: Kubectl delete node tools-k8s-worker-[16-20] (T248702)
  • curprev 00:0100:01, 6 May 2020imported>Stashbot 163,132 bytes +444 bd808: Joining tools-k8s-worker-60 to the k8s worker pool

4 May 2020

  • curprev 22:0822:08, 4 May 2020imported>Stashbot 162,688 bytes +346 bstorm_: deleting tools-elastic-01/2/3 T236606

29 April 2020

  • curprev 22:1322:13, 29 April 2020imported>Stashbot 162,342 bytes +452 bstorm_: running a fixup script after fixing a bug T247455

28 April 2020

  • curprev 22:5822:58, 28 April 2020imported>Stashbot 161,890 bytes +131 bstorm_: rebuilding docker-registry.tools.wmflabs.org/maintain-kubeusers:beta T247455

23 April 2020

  • curprev 19:2219:22, 23 April 2020imported>Stashbot 161,759 bytes +92 bd808: Increased Kubernetes services quota for bd808-test tool.

21 April 2020

  • curprev 23:0623:06, 21 April 2020imported>Stashbot 161,667 bytes +386 bstorm_: repooled tools-k8s-worker-38/52, tools-sgewebgrid-lighttpd-0918/9 and tools-sgeexec-0901 T250869

20 April 2020

  • curprev 15:3115:31, 20 April 2020imported>Stashbot 161,281 bytes +607 bd808: Rebuilding Docker containers to pick up tools-webservice v0.68 (T250625)

15 April 2020

  • curprev 23:2023:20, 15 April 2020imported>Stashbot 160,674 bytes +253 bd808: Building ruby25-sssd/base and children (T141388, T250118)

14 April 2020

  • curprev 18:2618:26, 14 April 2020imported>Stashbot 160,421 bytes +316 bstorm_: Deployed new code and RBAC for maintain-kubeusers T246123

10 April 2020

  • curprev 21:3321:33, 10 April 2020imported>Stashbot 160,105 bytes +369 bd808: Rebuilding all Docker images for the Kubernetes cluster (T249843)

9 April 2020

  • curprev 15:1315:13, 9 April 2020imported>Stashbot 159,736 bytes +522 bd808: Rebuilding all stretch and buster Docker images. Jessie is broken at the moment due to package version mismatches
  • curprev 00:2000:20, 9 April 2020imported>Stashbot 159,214 bytes +450 bd808: Docker rebuild failed in toolforge-python2-sssd-base: "zlib1g-dev : Depends: zlib1g (= 1:1.2.8.dfsg-2+b1) but 1:1.2.8.dfsg-2+deb8u1 is to be installed"

7 April 2020

  • curprev 20:0620:06, 7 April 2020imported>Stashbot 158,764 bytes +161 andrewbogott: sss_cache -E on tools-sgebastion-08 and tools-sgebastion-09

6 April 2020

3 April 2020

30 March 2020

  • curprev 18:2818:28, 30 March 2020imported>Stashbot 157,942 bytes +316 bstorm_: Beginning rolling depool, remount, repool of k8s workers for T248702

27 March 2020

  • curprev 21:2221:22, 27 March 2020imported>Stashbot 157,626 bytes +374 bstorm_: removed puppet prefix tools-docker-builder T248703

24 March 2020

  • curprev 11:4411:44, 24 March 2020imported>Stashbot 157,252 bytes +427 arturo: trying to solve a rebase/merge conflict in labs/private.git in tools-puppetmaster-02

18 March 2020

  • curprev 19:0719:07, 18 March 2020imported>Stashbot 156,825 bytes +730 bstorm_: removed role::toollabs::logging::sender from project puppet (it wouldn't work anyway)

17 March 2020

  • curprev 13:2913:29, 17 March 2020imported>Stashbot 156,095 bytes +113 arturo: set `profile::toolforge::bastion::nproc: 200` for tools-sgebastion-08 (T219070)
  • curprev 00:0800:08, 17 March 2020imported>Stashbot 155,982 bytes +357 bstorm_: shut off tools-flannel-etcd-01/02/03 T246689

11 March 2020

6 March 2020

  • curprev 16:2516:25, 6 March 2020imported>Stashbot 155,550 bytes +100 bstorm_: updating maintain-kubeusers image to filter invalid tool names

3 March 2020

  • curprev 18:1618:16, 3 March 2020imported>Stashbot 155,450 bytes +597 jeh: create OpenStack DNS record for elasticsearch.svc.tools.eqiad1.wikimedia.cloud (eqiad1 subdomain change) T236606

2 March 2020

  • curprev 22:2622:26, 2 March 2020imported>Stashbot 154,853 bytes +125 jeh: starting first pass of elasticsearch data migration to new cluster T236606

1 March 2020

  • curprev 01:4801:48, 1 March 2020imported>Stashbot 154,728 bytes +330 bstorm_: old version of kubectl removed. Anyone who needs it can download it with `curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.4.12/bin/linux/amd64/kubectl`

28 February 2020

  • curprev 22:1422:14, 28 February 2020imported>Stashbot 154,398 bytes +2,223 bstorm_: shutting down the old maintain-kubeusers and taking the gloves off the new one (removing --gentle-mode)
  • curprev 00:5000:50, 28 February 2020imported>Stashbot 152,175 bytes +1,873 bstorm_: rebuilt all docker images to include webservice 0.64

27 February 2020

25 February 2020

23 February 2020

21 February 2020

20 February 2020

  • curprev 14:4914:49, 20 February 2020imported>Stashbot 149,127 bytes +117 andrewbogott: moving tools-k8s-worker-19 and tools-k8s-worker-18 to cloudvirt1022 (as part of draining 1014)
  • curprev 00:0400:04, 20 February 2020imported>Stashbot 149,010 bytes +526 Krenair: Shut off tools-puppetmaster-01 - to be deleted in one week T245365

19 February 2020

  • curprev 00:5900:59, 19 February 2020imported>Stashbot 148,484 bytes +424 bd808: Live hacked the "nginx-configuration" ConfigMap for T245426 (done several hours ago, but I forgot to !log it)

17 February 2020

  • curprev 18:5318:53, 17 February 2020imported>Stashbot 148,060 bytes +286 arturo: T168677 created DNS TXT record _psl.toolforge.org. with value `https://github.com/publicsuffix/list/pull/970`

14 February 2020

  • curprev 00:3800:38, 14 February 2020imported>Stashbot 147,774 bytes +1,893 bd808: Added tools-k8s-worker-35 to 2020 Kubernetes cluster (T244791)

12 February 2020

  • curprev 19:2919:29, 12 February 2020imported>Stashbot 145,881 bytes +199 bd808: Rebuilding all Docker images to pick up toollabs-webservice (0.63) (T244954)
  • curprev 00:2000:20, 12 February 2020imported>Stashbot 145,682 bytes +785 bd808: Depooling tools-sgewebgrid-generic-0903 (T244791)

10 February 2020

7 February 2020

  • curprev 10:5510:55, 7 February 2020imported>Stashbot 144,360 bytes +167 arturo: drop jessie VM instances tools-prometheus-{01,02} which were shutdown (T238096)

6 February 2020

  • curprev 10:4410:44, 6 February 2020imported>Stashbot 144,193 bytes +367 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/565556 which is a behavior change to the Toolforge front proxy (T234617)

5 February 2020

  • curprev 11:2211:22, 5 February 2020imported>Stashbot 143,826 bytes +155 arturo: restarting ferm fleet-wide to account for prometheus servers changed IP (but same hostname) (T238096)

4 February 2020

  • curprev 11:3811:38, 4 February 2020imported>Stashbot 143,671 bytes +258 arturo: start again tools-prometheus-01 again to sync data to the new tools-prometheus-03/04 VMs (T238096)

3 February 2020

  • curprev 14:1214:12, 3 February 2020imported>Stashbot 143,413 bytes +471 arturo: move tools-prometheus-04 from cloudvirt1022 to cloudvirt1013

31 January 2020

  • curprev 14:0614:06, 31 January 2020imported>Stashbot 142,942 bytes +411 arturo: leave tools-prometheus-01 as the backend for tools-prometheus.wmflabs.org for the weekend so grafana dashboards keep working (T238096)

30 January 2020

  • curprev 21:0421:04, 30 January 2020imported>Stashbot 142,531 bytes +1,613 andrewbogott: also apt-get install python3-novaclient on tools-prometheus-03 and tools-prometheus-04 to suppress cronspam. Possible real fix for this is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569084/

29 January 2020

  • curprev 20:0720:07, 29 January 2020imported>Stashbot 140,918 bytes +174 bd808: Created {bastion,login,dev}.toolforge.org service names for Toolforge bastions using Horizon & Designate
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)