You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Tools/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

7 May 2020

  • curprev 21:5121:51, 7 May 2020imported>Stashbot 163,886 bytes +245 bstorm_: rebuilding the docker images for Toolforge k8s

6 May 2020

  • curprev 21:2021:20, 6 May 2020imported>Stashbot 163,641 bytes +509 bd808: Kubectl delete node tools-k8s-worker-[16-20] (T248702)
  • curprev 00:0100:01, 6 May 2020imported>Stashbot 163,132 bytes +444 bd808: Joining tools-k8s-worker-60 to the k8s worker pool

4 May 2020

  • curprev 22:0822:08, 4 May 2020imported>Stashbot 162,688 bytes +346 bstorm_: deleting tools-elastic-01/2/3 T236606

29 April 2020

  • curprev 22:1322:13, 29 April 2020imported>Stashbot 162,342 bytes +452 bstorm_: running a fixup script after fixing a bug T247455

28 April 2020

  • curprev 22:5822:58, 28 April 2020imported>Stashbot 161,890 bytes +131 bstorm_: rebuilding docker-registry.tools.wmflabs.org/maintain-kubeusers:beta T247455

23 April 2020

  • curprev 19:2219:22, 23 April 2020imported>Stashbot 161,759 bytes +92 bd808: Increased Kubernetes services quota for bd808-test tool.

21 April 2020

  • curprev 23:0623:06, 21 April 2020imported>Stashbot 161,667 bytes +386 bstorm_: repooled tools-k8s-worker-38/52, tools-sgewebgrid-lighttpd-0918/9 and tools-sgeexec-0901 T250869

20 April 2020

  • curprev 15:3115:31, 20 April 2020imported>Stashbot 161,281 bytes +607 bd808: Rebuilding Docker containers to pick up tools-webservice v0.68 (T250625)

15 April 2020

  • curprev 23:2023:20, 15 April 2020imported>Stashbot 160,674 bytes +253 bd808: Building ruby25-sssd/base and children (T141388, T250118)

14 April 2020

  • curprev 18:2618:26, 14 April 2020imported>Stashbot 160,421 bytes +316 bstorm_: Deployed new code and RBAC for maintain-kubeusers T246123

10 April 2020

  • curprev 21:3321:33, 10 April 2020imported>Stashbot 160,105 bytes +369 bd808: Rebuilding all Docker images for the Kubernetes cluster (T249843)

9 April 2020

  • curprev 15:1315:13, 9 April 2020imported>Stashbot 159,736 bytes +522 bd808: Rebuilding all stretch and buster Docker images. Jessie is broken at the moment due to package version mismatches
  • curprev 00:2000:20, 9 April 2020imported>Stashbot 159,214 bytes +450 bd808: Docker rebuild failed in toolforge-python2-sssd-base: "zlib1g-dev : Depends: zlib1g (= 1:1.2.8.dfsg-2+b1) but 1:1.2.8.dfsg-2+deb8u1 is to be installed"

7 April 2020

  • curprev 20:0620:06, 7 April 2020imported>Stashbot 158,764 bytes +161 andrewbogott: sss_cache -E on tools-sgebastion-08 and tools-sgebastion-09

6 April 2020

3 April 2020

30 March 2020

  • curprev 18:2818:28, 30 March 2020imported>Stashbot 157,942 bytes +316 bstorm_: Beginning rolling depool, remount, repool of k8s workers for T248702

27 March 2020

  • curprev 21:2221:22, 27 March 2020imported>Stashbot 157,626 bytes +374 bstorm_: removed puppet prefix tools-docker-builder T248703

24 March 2020

  • curprev 11:4411:44, 24 March 2020imported>Stashbot 157,252 bytes +427 arturo: trying to solve a rebase/merge conflict in labs/private.git in tools-puppetmaster-02

18 March 2020

  • curprev 19:0719:07, 18 March 2020imported>Stashbot 156,825 bytes +730 bstorm_: removed role::toollabs::logging::sender from project puppet (it wouldn't work anyway)

17 March 2020

  • curprev 13:2913:29, 17 March 2020imported>Stashbot 156,095 bytes +113 arturo: set `profile::toolforge::bastion::nproc: 200` for tools-sgebastion-08 (T219070)
  • curprev 00:0800:08, 17 March 2020imported>Stashbot 155,982 bytes +357 bstorm_: shut off tools-flannel-etcd-01/02/03 T246689

11 March 2020

6 March 2020

  • curprev 16:2516:25, 6 March 2020imported>Stashbot 155,550 bytes +100 bstorm_: updating maintain-kubeusers image to filter invalid tool names

3 March 2020

  • curprev 18:1618:16, 3 March 2020imported>Stashbot 155,450 bytes +597 jeh: create OpenStack DNS record for elasticsearch.svc.tools.eqiad1.wikimedia.cloud (eqiad1 subdomain change) T236606

2 March 2020

  • curprev 22:2622:26, 2 March 2020imported>Stashbot 154,853 bytes +125 jeh: starting first pass of elasticsearch data migration to new cluster T236606

1 March 2020

  • curprev 01:4801:48, 1 March 2020imported>Stashbot 154,728 bytes +330 bstorm_: old version of kubectl removed. Anyone who needs it can download it with `curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.4.12/bin/linux/amd64/kubectl`

28 February 2020

  • curprev 22:1422:14, 28 February 2020imported>Stashbot 154,398 bytes +2,223 bstorm_: shutting down the old maintain-kubeusers and taking the gloves off the new one (removing --gentle-mode)
  • curprev 00:5000:50, 28 February 2020imported>Stashbot 152,175 bytes +1,873 bstorm_: rebuilt all docker images to include webservice 0.64

27 February 2020

25 February 2020

23 February 2020

21 February 2020

20 February 2020

  • curprev 14:4914:49, 20 February 2020imported>Stashbot 149,127 bytes +117 andrewbogott: moving tools-k8s-worker-19 and tools-k8s-worker-18 to cloudvirt1022 (as part of draining 1014)
  • curprev 00:0400:04, 20 February 2020imported>Stashbot 149,010 bytes +526 Krenair: Shut off tools-puppetmaster-01 - to be deleted in one week T245365

19 February 2020

  • curprev 00:5900:59, 19 February 2020imported>Stashbot 148,484 bytes +424 bd808: Live hacked the "nginx-configuration" ConfigMap for T245426 (done several hours ago, but I forgot to !log it)

17 February 2020

  • curprev 18:5318:53, 17 February 2020imported>Stashbot 148,060 bytes +286 arturo: T168677 created DNS TXT record _psl.toolforge.org. with value `https://github.com/publicsuffix/list/pull/970`

14 February 2020

  • curprev 00:3800:38, 14 February 2020imported>Stashbot 147,774 bytes +1,893 bd808: Added tools-k8s-worker-35 to 2020 Kubernetes cluster (T244791)

12 February 2020

  • curprev 19:2919:29, 12 February 2020imported>Stashbot 145,881 bytes +199 bd808: Rebuilding all Docker images to pick up toollabs-webservice (0.63) (T244954)
  • curprev 00:2000:20, 12 February 2020imported>Stashbot 145,682 bytes +785 bd808: Depooling tools-sgewebgrid-generic-0903 (T244791)

10 February 2020

7 February 2020

  • curprev 10:5510:55, 7 February 2020imported>Stashbot 144,360 bytes +167 arturo: drop jessie VM instances tools-prometheus-{01,02} which were shutdown (T238096)

6 February 2020

  • curprev 10:4410:44, 6 February 2020imported>Stashbot 144,193 bytes +367 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/565556 which is a behavior change to the Toolforge front proxy (T234617)

5 February 2020

  • curprev 11:2211:22, 5 February 2020imported>Stashbot 143,826 bytes +155 arturo: restarting ferm fleet-wide to account for prometheus servers changed IP (but same hostname) (T238096)

4 February 2020

  • curprev 11:3811:38, 4 February 2020imported>Stashbot 143,671 bytes +258 arturo: start again tools-prometheus-01 again to sync data to the new tools-prometheus-03/04 VMs (T238096)

3 February 2020

  • curprev 14:1214:12, 3 February 2020imported>Stashbot 143,413 bytes +471 arturo: move tools-prometheus-04 from cloudvirt1022 to cloudvirt1013

31 January 2020

  • curprev 14:0614:06, 31 January 2020imported>Stashbot 142,942 bytes +411 arturo: leave tools-prometheus-01 as the backend for tools-prometheus.wmflabs.org for the weekend so grafana dashboards keep working (T238096)

30 January 2020

  • curprev 21:0421:04, 30 January 2020imported>Stashbot 142,531 bytes +1,613 andrewbogott: also apt-get install python3-novaclient on tools-prometheus-03 and tools-prometheus-04 to suppress cronspam. Possible real fix for this is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569084/

29 January 2020

  • curprev 20:0720:07, 29 January 2020imported>Stashbot 140,918 bytes +174 bd808: Created {bastion,login,dev}.toolforge.org service names for Toolforge bastions using Horizon & Designate

28 January 2020

  • curprev 13:3513:35, 28 January 2020imported>Stashbot 140,744 bytes +289 arturo: `aborrero@tools-clushmaster-02:~$ clush -w @exec-stretch 'for i in $(ps aux | grep [t]ools.j | awk -F" " "{print \$2}") ; do echo "killing $i" ; sudo kill $i ; done || true'` (T243831)

27 January 2020

  • curprev 07:0507:05, 27 January 2020imported>Stashbot 140,455 bytes +329 zhuyifei1999_: wrong package. uninstalled. the correct one is bpfcc-tools and seems only available in buster+. T115231

24 January 2020

  • curprev 20:5820:58, 24 January 2020imported>Stashbot 140,126 bytes +457 bd808: Built tools-k8s-worker-21 to test out build script following openstack client upgrade

23 January 2020

  • curprev 23:3823:38, 23 January 2020imported>Stashbot 139,669 bytes +421 bd808: Halted tools-k8s-worker build script after first instance (tools-k8s-worker-10) stuck in "scheduling" state for 20 minutes

22 January 2020

  • curprev 12:4312:43, 22 January 2020imported>Stashbot 139,248 bytes +200 arturo: for the record, issue with tools-worker-1016 was memory exhaustion apparently

21 January 2020

  • curprev 19:2519:25, 21 January 2020imported>Stashbot 139,048 bytes +398 bstorm_: hard rebooting tools-sgeexec-0913/14/35 because they aren't even on the network

16 January 2020

  • curprev 23:5423:54, 16 January 2020imported>Stashbot 138,650 bytes +432 bstorm_: rebooting tools-docker-builder-06 because there are a couple running containers that don't want to die cleanly

14 January 2020

  • curprev 15:2915:29, 14 January 2020imported>Stashbot 138,218 bytes +216 bstorm_: failed the gridengine master back to the master server from the shadow

13 January 2020

  • curprev 17:4817:48, 13 January 2020imported>Stashbot 138,002 bytes +557 bd808: Running `puppet ca destroy` for each unsigned cert on tools-puppetmaster-01 (T242642)

12 January 2020

11 January 2020

  • curprev 01:3301:33, 11 January 2020imported>Stashbot 137,221 bytes +157 bstorm_: updated toollabs-webservice package to 0.57, which should allow persisting mem and cpu in manifests with burstable qos.

10 January 2020

9 January 2020

  • curprev 23:3523:35, 9 January 2020imported>Stashbot 136,602 bytes +533 bstorm_: depooled tools-sgeexec-0939 because it isn't acting right and rebooting it

7 January 2020

  • curprev 22:4022:40, 7 January 2020imported>Stashbot 136,069 bytes +1,199 bstorm_: rebooted tools-worker-1007 to recover it from disk full and general badness
  • curprev 00:2600:26, 7 January 2020imported>Stashbot 134,870 bytes +1,665 bstorm_: repooled tools-sgewebgrid-lighttpd-0919

4 January 2020

3 January 2020

  • curprev 16:4816:48, 3 January 2020imported>Stashbot 131,428 bytes +586 bstorm_: updated the ValidatingWebhookConfiguration for the ingress admission controller to the working settings
  • curprev 00:1100:11, 3 January 2020imported>Stashbot 130,842 bytes +191 bd808: Rebuiliding all stretch-ssd Docker images to pick up busybox

30 December 2019

  • curprev 05:0205:02, 30 December 2019imported>Stashbot 130,651 bytes +195 andrewbogott: moving tools-worker-1012 to cloudvirt1024 for T241523

29 December 2019

  • curprev 01:3801:38, 29 December 2019imported>Stashbot 130,456 bytes +215 Krenair: Cordoned tools-worker-1012 and deleted pods associated with dplbot and dewikigreetbot as well as my own testing one, host seems to be under heavy load - T241523

27 December 2019

  • curprev 15:0615:06, 27 December 2019imported>Stashbot 130,241 bytes +142 Krenair: Killed a "python parse_page.py outreachy" process by aikochou that was hogging IO on tools-sgebastion-07

25 December 2019

  • curprev 16:0716:07, 25 December 2019imported>Stashbot 130,099 bytes +134 zhuyifei1999_: pkilled 5 `python pwb.py` processes belonging to `tools.kaleem-bot` on tools-sgebastion-07

22 December 2019

  • curprev 20:1320:13, 22 December 2019imported>Stashbot 129,965 bytes +263 bd808: Enabled Puppet on tools-proxy-06.tools.eqiad.wmflabs after nginx config test (T241310)

20 December 2019

  • curprev 22:2822:28, 20 December 2019imported>Stashbot 129,702 bytes +211 bd808: Re-enabled Puppet on tools-sgebastion-09. Reason for disable was "arturo raising systemd limits"

18 December 2019

  • curprev 17:3317:33, 18 December 2019imported>Stashbot 129,491 bytes +310 bstorm_: updated package in aptly for toollabs-webservice to 0.53

17 December 2019

  • curprev 20:2520:25, 17 December 2019imported>Stashbot 129,181 bytes +950 bd808: Fixed https://tools.wmflabs.org/ to redirect to https://tools.wmflabs.org/admin/
  • curprev 00:4500:45, 17 December 2019imported>Stashbot 128,231 bytes +295 bstorm_: enabled encryption at rest on the new k8s cluster

14 December 2019

  • curprev 10:4810:48, 14 December 2019imported>Stashbot 127,936 bytes +153 valhallasw`cloud: re-enabling puppet on tools-sgeexec-0912, likely left-over from NFS maintenance (no reason was specified).

13 December 2019

11 December 2019

  • curprev 18:1318:13, 11 December 2019imported>Stashbot 126,661 bytes +239 bd808: Restarted maintain-dbusers on labstore1004. Process had not logged any account creations since 2019-12-01T22:45:45.

10 December 2019

9 December 2019

  • curprev 11:0611:06, 9 December 2019imported>Stashbot 126,314 bytes +144 andrewbogott: deleting unused security groups: catgraph, devpi, MTA, mysql, syslog, test T91619

4 December 2019

  • curprev 13:4513:45, 4 December 2019imported>Stashbot 126,170 bytes +101 arturo: drop puppet prefix `tools-cron`, deprecated and no longer in use

29 November 2019

19 November 2019

  • curprev 13:4913:49, 19 November 2019imported>Stashbot 124,129 bytes +239 arturo: re-create nginx-ingress pod due to deployment template refresh (T237643)

15 November 2019

13 November 2019

  • curprev 17:2017:20, 13 November 2019imported>Stashbot 123,790 bytes +154 arturo: live-hacking tools-prometheus-01 to test some experimental configs for the new k8s cluster (T237643)

12 November 2019

10 November 2019

  • curprev 02:1702:17, 10 November 2019imported>Stashbot 123,529 bytes +520 bd808: Building new Docker images for T237836 (retrying after cleaning out old images on tools-docker-builder-06)

8 November 2019

  • curprev 22:4722:47, 8 November 2019imported>Stashbot 123,009 bytes +477 bstorm_: adding rsync::server::wrap_with_stunnel: false to the tools-docker-registry-03/4 servers to unbreak puppet

7 November 2019

  • curprev 13:2713:27, 7 November 2019imported>Stashbot 122,532 bytes +616 arturo: deployed registry-admission-webhook and ingress-admission-controller into the new k8s cluster (T236826)

6 November 2019

  • curprev 22:3222:32, 6 November 2019imported>Stashbot 121,916 bytes +804 bstorm_: added rsync::server::wrap_with_stunnel: false to tools-sge-services prefix to fix puppet

5 November 2019

  • curprev 23:0823:08, 5 November 2019imported>Stashbot 121,112 bytes +1,265 Krenair: Dropped 59a77a3, 3830802, and 83df61f from tools-puppetmaster-01:/var/lib/git/labs/private cherry-picks as these are no longer required T206235

4 November 2019

  • curprev 14:4514:45, 4 November 2019imported>Stashbot 119,847 bytes +503 phamhi: Built and pushed ruby25 docker image based on buster (T230961)

1 November 2019

  • curprev 21:0021:00, 1 November 2019imported>Stashbot 119,344 bytes +604 Krenair: Removed tools-checker.wmflabs.org A record to 208.80.155.229 as that target IP is in the old pre-neutron range that is no longer routed

31 October 2019

  • curprev 18:4718:47, 31 October 2019imported>Stashbot 118,740 bytes +783 andrewbogott: deleted and/or truncated a bunch of logfiles on tools-worker-1001. Runaway logfiles filled up the drive which prevented puppet from running. If puppet had run, it would have prevented the runaway logfiles.

30 October 2019

  • curprev 13:5313:53, 30 October 2019imported>Stashbot 117,957 bytes +464 arturo: replacing SSL cert in tools-proxy-x server apparently OK (merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679) T235252

29 October 2019

28 October 2019

  • curprev 16:0616:06, 28 October 2019imported>Stashbot 117,323 bytes +2,583 arturo: delete VM instance `tools-test-proxy-01` and the puppet prefix `tools-test-proxy`
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)