You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Revision history of "Nova Resource:Tools/SAL"

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)
  • curprev 17:46, 28 November 2021imported>Stashbot 232,192 bytes +165 andrewbogott: moving tools-k8s-etcd-13 to cloudvirt1020; cloudvirt1018 (its old host) has a degraded raid which is affecting performance
  • curprev 13:16, 19 November 2021imported>Stashbot 232,027 bytes +97 majavah: manually add 3 project members after ldap issues were fixed
  • curprev 12:31, 16 November 2021imported>Stashbot 231,930 bytes +197 majavah: uploading calico 3.21.0 to the internal docker registry T292698
  • curprev 10:50, 11 November 2021imported>Stashbot 231,733 bytes +107 arturo: add user `srv-networktests` as project user (T294955)
  • curprev 19:18, 5 November 2021imported>Stashbot 231,626 bytes +74 majavah: deploying registry-admission changes
  • curprev 23:58, 29 October 2021imported>Stashbot 231,552 bytes +129 andrewbogott: deleting all files older than 14 days in /srv/tools/shared/tools/project/.shared/cache
  • curprev 12:42, 28 October 2021imported>Stashbot 231,423 bytes +122 arturo: set `allow-snippet-annotations: "false"` for ingress-nginx (T294330)
  • curprev 18:00, 26 October 2021imported>Stashbot 231,301 bytes +238 majavah: deleting legacy ingresses for tools.wmflabs.org urls
  • curprev 14:33, 25 October 2021imported>Stashbot 231,063 bytes +262 majavah: copy nginx-ingress controller v1.0.4 to internal registry T292771
  • curprev 15:35, 22 October 2021imported>Stashbot 230,801 bytes +240 majavah: remove "^tools-k8s-master-[0-9]+\.tools\.eqiad\.wmflabs$" from authorized_regexes for the main certificate
  • curprev 09:48, 21 October 2021imported>Stashbot 230,561 bytes +73 majavah: deploying toolforge-webservice 0.79
  • curprev 15:41, 20 October 2021imported>Stashbot 230,488 bytes +276 majavah: removing toollabs-webservice from grid exec and master nodes where it's not needed and not managed by puppet
  • curprev 15:01, 15 October 2021imported>Stashbot 230,212 bytes +129 arturo: add updated ingress-nginx docker image in the registry (v1.0.1) for T293472
  • curprev 09:13, 7 October 2021imported>Stashbot 230,083 bytes +247 majavah: disabling settings api, now that all pod presets are gone T279106
  • curprev 06:46, 6 October 2021imported>Stashbot 229,836 bytes +154 majavah: taavi@toolserver-proxy-01:~$ sudo systemctl restart apache2.service # see if it helps with toolserver.org ssl alerts
  • curprev 21:31, 3 October 2021imported>Stashbot 229,682 bytes +254 bstorm: rebuilding buster containers since they are also affected T291387 T292355
  • curprev 21:59, 1 October 2021imported>Stashbot 229,428 bytes +347 bd808: clush -w @all -b 'sudo sed -i "s#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#" /etc/ca-certificates.conf && sudo update-ca-certificates' for T292289
  • curprev 22:39, 29 September 2021imported>Stashbot 229,081 bytes +265 bstorm: finished deploy of the toollabs-webservice 0.77 and updating labels across the k8s cluster to match
  • curprev 16:19, 27 September 2021imported>Stashbot 228,816 bytes +257 majavah: deploy volume-admission fix for containers for some volumes mounted
  • curprev 17:20, 23 September 2021imported>Stashbot 228,559 bytes +118 majavah: deploying new maintain-kubeusers for lack of podpresets T279106
  • curprev 18:06, 22 September 2021imported>Stashbot 228,441 bytes +257 bstorm: launching tools-nfs-test-client-01 to run a "fair" test battery against T291406
  • curprev 12:44, 20 September 2021imported>Stashbot 228,184 bytes +130 majavah: deploying volume-admission to tools, should not affect anything yet T279106
  • curprev 08:08, 15 September 2021imported>Stashbot 228,054 bytes +67 majavah: update tools-manifest to 0.24
  • curprev 10:36, 14 September 2021imported>Stashbot 227,987 bytes +104 arturo: add toolforge-jobs-framework-cli v5 to aptly buster-tools/toolsbeta
  • curprev 08:57, 13 September 2021imported>Stashbot 227,883 bytes +291 arturo: cleared grid queues error states (T290844)
  • curprev 08:51, 11 September 2021imported>Stashbot 227,592 bytes +63 majavah: depool tools-sgeexec-0907
  • curprev 23:26, 10 September 2021imported>Stashbot 227,529 bytes +359 bstorm: cleared error state for tools-sgeexec-0907.tools.eqiad.wmflabs
  • curprev 16:20, 9 September 2021imported>Stashbot 227,170 bytes +155 arturo: 70017ec0ac root@tools-k8s-control-3:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
  • curprev 15:27, 7 September 2021imported>Stashbot 227,015 bytes +178 majavah: rolling out python3-prometheus-client updates
  • curprev 16:31, 6 September 2021imported>Stashbot 226,837 bytes +132 arturo: deploying jobs-framework-cli v4
  • curprev 22:36, 3 September 2021imported>Stashbot 226,705 bytes +148 bstorm: backfilling quotas in screen for T286784
  • curprev 01:02, 2 September 2021imported>Stashbot 226,557 bytes +140 bstorm: deployed new version of maintain-kubeusers with new count quotas for new tools T286784
  • curprev 19:10, 20 August 2021imported>Stashbot 226,417 bytes +236 majavah: rebuilding node12-sssd/{base,web} to use debian packaged npm 7
  • curprev 21:32, 18 August 2021imported>Stashbot 226,181 bytes +203 bstorm: rebooted tools-sgecron-01 due to a ram filling up and killing everything
  • curprev 17:00, 16 August 2021imported>Stashbot 225,978 bytes +316 majavah: remove and re-add toollabs-webservice 0.75 on stretch-toolsbeta repository
  • curprev 17:30, 15 August 2021imported>Stashbot 225,662 bytes +546 majavah: deploying update jobs-framework-api container list to include bullseye images
  • curprev 16:59, 12 August 2021imported>Stashbot 225,116 bytes +377 bstorm: deployed updated manifest for ingress-admission
  • curprev 05:59, 7 August 2021imported>Stashbot 224,739 bytes +134 majavah: restart nginx on toolserver-proxy-01 if that helps with flapping icinga certificate expiry check
  • curprev 16:17, 6 August 2021imported>Stashbot 224,605 bytes +104 bstorm: failed over to tools-docker-registry-06 (which has more space) T288229
  • curprev 00:43, 6 August 2021imported>Stashbot 224,501 bytes +430 bstorm: set up sync between the new registry host and the existing one T288229
  • curprev 18:04, 29 July 2021imported>Stashbot 224,071 bytes +133 majavah: reset sul account mapping on striker for developer account "Derek Zax" T287369
  • curprev 21:33, 28 July 2021imported>Stashbot 223,938 bytes +111 majavah: add mdipietro as projectadmin and to sudo policy T287287
  • curprev 16:20, 27 July 2021imported>Stashbot 223,827 bytes +84 bstorm: built new php images with python2 on board T287421
  • curprev 00:04, 27 July 2021imported>Stashbot 223,743 bytes +381 bstorm: deploy a version of the php3.7 web image that includes the python2 package with tag :testing T287421
  • curprev 07:15, 23 July 2021imported>Stashbot 223,362 bytes +109 majavah: restart nginx on tools-static-14 to see if it helps with fontcdn issues
  • curprev 23:35, 22 July 2021imported>Stashbot 223,253 bytes +336 bstorm: deleted tools-sgebastion-09 since it has been shut off since March anyway
  • curprev 20:01, 21 July 2021imported>Stashbot 222,917 bytes +817 bstorm: deployed new maintain-kubeusers to toolforge T285011
  • curprev 18:42, 20 July 2021imported>Stashbot 222,100 bytes +451 majavah: deploying systemd security tools on toolforge public stretch machines T287004
  • curprev 23:24, 19 July 2021imported>Stashbot 221,649 bytes +248 bstorm: applied matchPolicy: equivalent to tools ingress validation controller T280360
  • curprev 14:04, 16 July 2021imported>Stashbot 221,401 bytes +352 arturo: deployed jobs-framework-api 42b7a885a5bc1bf00c300e8d77bd92e1430a8327 (T286132)
  • curprev 16:12, 15 July 2021imported>Stashbot 221,049 bytes +417 arturo: deploy toolforge-jobs-framework-api git version d85d93ee1c5d4be6a526cf83e806b2679dde3875 (T285944, T286107, T285979, T286485, T286107)
  • curprev 23:29, 14 July 2021imported>Stashbot 220,632 bytes +250 bstorm: mounted nfs on tools-services-05 and backing up aptly to NFS dir T286003
  • curprev 16:56, 12 July 2021imported>Stashbot 220,382 bytes +143 bstorm: deleted job 4720371 due to LDAP failure
  • curprev 18:46, 2 July 2021imported>Stashbot 220,239 bytes +99 bstorm: cleared error state for tools-sgeexec-0940.tools.eqiad.wmflabs
  • curprev 22:08, 1 July 2021imported>Stashbot 220,140 bytes +445 bstorm: releasing webservice 0.75
  • curprev 21:58, 29 June 2021imported>Stashbot 219,695 bytes +594 bstorm: clearing one errored queue and a stack of discarded jobs
  • curprev 19:02, 15 June 2021imported>Stashbot 219,101 bytes +181 bstorm: cleared error status from a few queues
  • curprev 22:21, 14 June 2021imported>Stashbot 218,920 bytes +229 bstorm: push docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web:testing to test staged os.execv (and other patches) using toolsbeta toollabs-webservice version 0.75 T282975
  • curprev 08:15, 13 June 2021imported>Stashbot 218,691 bytes +124 majavah: clear grid error state from tools-sgeexec-0907, tools-sgeexec-0916, tools-sgeexec-0940
  • curprev 14:39, 12 June 2021imported>Stashbot 218,567 bytes +267 majavah: remove nonexistent tools-prometheus-04 and add tools-prometheus-05 to hiera key "prometheus_nodes"
  • curprev 17:38, 10 June 2021imported>Stashbot 218,300 bytes +104 majavah: clear error state from tools-sgeexec-0907, task@tools-sgeexec-0939
  • curprev 13:57, 9 June 2021imported>Stashbot 218,196 bytes +135 majavah: clear error state from exec nodes tools-sgeexec-0913, tools-sgeexec-0936, task@tools-sgeexec-0940
  • curprev 18:39, 7 June 2021imported>Stashbot 218,061 bytes +334 bstorm: cleaning up more error conditions on grid queues
  • curprev 21:30, 4 June 2021imported>Stashbot 217,727 bytes +193 bstorm: deleting "tools-k8s-ingress-3", "tools-k8s-ingress-2", "tools-k8s-ingress-1" T264221
  • curprev 18:27, 3 June 2021imported>Stashbot 217,534 bytes +181 majavah: renew prometheus kubernetes certificate T280301
  • curprev 10:10, 1 June 2021imported>Stashbot 217,353 bytes +238 majavah: properly clean up deleted vms tools-k8s-haproxy-[1,2], tools-checker-03 from puppet after using the wrong fqdn first time
  • curprev 18:58, 30 May 2021imported>Stashbot 217,115 bytes +75 majavah: clear grid error state from 14 queues
  • curprev 18:03, 27 May 2021imported>Stashbot 217,040 bytes +283 bstorm: adjusted profile::wmcs::kubeadm::etcd_latency_ms from 30 back to the default (10)
  • curprev 10:36, 24 May 2021imported>Stashbot 216,757 bytes +230 arturo: rebased labs/private.git after merge conflict
  • curprev 14:47, 22 May 2021imported>Stashbot 216,527 bytes +389 majavah: manually remove jeh admin certificates and from maintain-kubeusers configmap T282725
  • curprev 17:06, 21 May 2021imported>Stashbot 216,138 bytes +626 majavah: unpool tooks-k8s-ingress-[4-6]
  • curprev 17:05, 20 May 2021imported>Stashbot 215,512 bytes +488 Majavah: pool tools-k8s-ingress-5 as an ingress node, depool ingress-1 T264221
  • curprev 12:15, 19 May 2021imported>Stashbot 215,024 bytes +263 Majavah: rollback ingress-nginx-gen2
  • curprev 16:52, 16 May 2021imported>Stashbot 214,761 bytes +136 Majavah: clear error state from tools-sgeexec-0905 tools-sgeexec-0907 tools-sgeexec-0936 tools-sgeexec-0941
  • curprev 19:18, 14 May 2021imported>Stashbot 214,625 bytes +379 bstorm: adjusting the rate limits for bastions nfs_write upward a lot to make NFS writes faster now that the cluster is finally using 10Gb on the backend and frontend T218338
  • curprev 19:45, 12 May 2021imported>Stashbot 214,246 bytes +384 bstorm: cleared error state from some queues
  • curprev 17:17, 11 May 2021imported>Stashbot 213,862 bytes +593 Majavah: shutdown and delete tools-checker-03 T278540
  • curprev 22:58, 10 May 2021imported>Stashbot 213,269 bytes +755 bstorm: cleared error state on a grid queue
  • curprev 06:55, 9 May 2021imported>Stashbot 212,514 bytes +79 Majavah: clear error state from tools-sgeexec-0916
  • curprev 10:57, 8 May 2021imported>Stashbot 212,435 bytes +214 Majavah: import docker image k8s.gcr.io/ingress-nginx/controller:v0.46.0 to local registry as docker-registry.tools.wmflabs.org/nginx-ingress-controller:v0.46.0 T264221
  • curprev 18:07, 7 May 2021imported>Stashbot 212,221 bytes +665 Majavah: generate and add k8s haproxy keepalived password (profile::toolforge::k8s::haproxy::keepalived_password) to private puppet repo
  • curprev 14:43, 6 May 2021imported>Stashbot 211,556 bytes +296 Majavah: clear error states from all currently erroring exec nodes
  • curprev 19:27, 5 May 2021imported>Stashbot 211,260 bytes +120 andrewbogott: adding taavi as a sudo root to project toolforge for T278390
  • curprev 15:23, 4 May 2021imported>Stashbot 211,140 bytes +151 arturo: upgrading exim4-daemon-heavy in tools-mail-03
  • curprev 16:24, 3 May 2021imported>Stashbot 210,989 bytes +360 dcaro: started tools-sgeexec-0907, was stuck on initramfs due to an unclean fs (/dev/vda3, root), ran fsck manually fixing all the errors and booted up correctly after (T280641)
  • curprev 18:23, 29 April 2021imported>Stashbot 210,629 bytes +178 bstorm: removing one more etcd node via cookbook T279723
  • curprev 16:40, 27 April 2021imported>Stashbot 210,451 bytes +170 bstorm: deleted all the errored out grid jobs stuck in queue wait
  • curprev 12:17, 26 April 2021imported>Stashbot 210,281 bytes +110 arturo: allowing more tools into the legacy redirector (T281003)
  • curprev 08:44, 22 April 2021imported>Stashbot 210,171 bytes +207 Krenair: Removed yuvipanda from roots sudo policy
  • curprev 22:20, 20 April 2021imported>Stashbot 209,964 bytes +818 bd808: `clush -w @all -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"`
  • curprev 10:53, 19 April 2021imported>Stashbot 209,146 bytes +205 dcaro: reverting setting prometheus data source in grafana to 'server', can't connect,
  • curprev 23:15, 16 April 2021imported>Stashbot 208,941 bytes +622 bstorm: cleaned up all source files for the grid with the old domain name to enable future node creation T277653
  • curprev 13:26, 13 April 2021imported>Stashbot 208,319 bytes +513 dcaro: upgrade puppet and python-wmflib on tools-prometheus-03
  • curprev 16:07, 11 April 2021imported>Stashbot 207,806 bytes +194 bstorm: cleared E state from tools-sgeexec-0917 tools-sgeexec-0933 tools-sgeexec-0934 tools-sgeexec-0937 from failures of jobs 761759, 815031, 815056, 855676, 898936
  • curprev 18:25, 8 April 2021imported>Stashbot 207,612 bytes +706 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for tools-sgegrid-master and tools-sgegrid-shadow using the old fqdns T277653
  • curprev 04:35, 7 April 2021imported>Stashbot 206,906 bytes +182 andrewbogott: replacing the mx record '10 mail.tools.wmcloud.org' with '10 mail.tools.wmcloud.org.' — trying to fix axfr for the tools.wmcloud.org zone
  • curprev 15:16, 6 April 2021imported>Stashbot 206,724 bytes +1,295 bstorm: cleared queue state since a few had "errored" for failed jobs.
  • curprev 17:02, 5 April 2021imported>Stashbot 205,429 bytes +205 bstorm: chowned the data volume for the docker registry to docker-registry:docker-registry
  • curprev 20:43, 1 April 2021imported>Stashbot 205,224 bytes +555 bstorm: cleared error state from the grid queues caused by unspecified job errors
  • curprev 15:57, 31 March 2021imported>Stashbot 204,669 bytes +891 arturo: rebooting `tools-mail-03` after enabling NFS (T267082, T278538)
(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)