You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Revision history of "Nova Resource:Toolsbeta/SAL"

Jump to navigation Jump to search

Diff selection: Mark the radio boxes of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

  • curprev 17:45, 28 November 2021imported>Stashbot 72,317 bytes +174 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance
  • curprev 12:37, 16 November 2021imported>Stashbot 72,143 bytes +90 majavah: testing calico 3.21 upgrade T292698
  • curprev 19:07, 5 November 2021imported>Stashbot 72,053 bytes +72 majavah: testing registry-admission changes
  • curprev 12:48, 28 October 2021imported>Stashbot 71,981 bytes +107 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`
  • curprev 14:41, 25 October 2021imported>Stashbot 71,874 bytes +141 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image T292771
  • curprev 12:15, 20 October 2021imported>Stashbot 71,733 bytes +119 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories
  • curprev 07:47, 16 October 2021imported>Stashbot 71,614 bytes +118 majavah: deployed cert-manager and wave as a test for automating T292238
  • curprev 15:02, 14 October 2021imported>Stashbot 71,496 bytes +470 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
  • curprev 11:18, 13 October 2021imported>Stashbot 71,026 bytes +434 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool (T292465) - cookbook ran by dcaro@vulcanus
  • curprev 16:10, 12 October 2021imported>Stashbot 70,592 bytes +442 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
  • curprev 15:24, 11 October 2021imported>Stashbot 70,150 bytes +365 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
  • curprev 14:21, 7 October 2021imported>Stashbot 69,785 bytes +940 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
  • curprev 10:36, 6 October 2021imported>Stashbot 68,845 bytes +595 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
  • curprev 17:07, 4 October 2021imported>Stashbot 68,250 bytes +427 bstorm: reboot everything T291406
  • curprev 12:34, 1 October 2021imported>Stashbot 67,823 bytes +205 arturo: rebooting toolsbeta-sgebastion-04 (T292289)
  • curprev 22:13, 29 September 2021imported>Stashbot 67,618 bytes +131 bstorm: ran label fix script to use new label format
  • curprev 10:32, 28 September 2021imported>Stashbot 67,487 bytes +111 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api
  • curprev 16:13, 27 September 2021imported>Stashbot 67,376 bytes +107 majavah: testing volume-admission fix for containers with some volumes mounted
  • curprev 17:14, 23 September 2021imported>Stashbot 67,269 bytes +101 majavah: testing new maintain-kubeusers release T279106
  • curprev 18:07, 22 September 2021imported>Stashbot 67,168 bytes +137 bstorm: launching toolsbeta-nfs-test-client-01 to run a "fair" test battery against T291406
  • curprev 08:04, 15 September 2021imported>Stashbot 67,031 bytes +83 majavah: tools-manifest 0.24, T290325
  • curprev 15:45, 14 September 2021imported>Stashbot 66,948 bytes +391 majavah: disable podpreset admission plugin in toolsbeta T279106
  • curprev 15:44, 13 September 2021imported>Stashbot 66,557 bytes +112 majavah: deploy volume-admission-controller in background; T279106
  • curprev 17:36, 9 September 2021imported>Stashbot 66,445 bytes +305 bstorm: deploying a base tekton triggers setup T267374
  • curprev 00:26, 9 September 2021imported>Stashbot 66,140 bytes +104 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp
  • curprev 22:34, 3 September 2021imported>Stashbot 66,036 bytes +83 bstorm: backfilled quotas for T286784
  • curprev 23:23, 30 August 2021imported>Stashbot 65,953 bytes +94 bstorm: deleting toolsbeta-workflow-test T289709
  • curprev 00:18, 21 August 2021imported>Stashbot 65,859 bytes +251 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse T289390
  • curprev 16:55, 12 August 2021imported>Stashbot 65,608 bytes +178 bstorm: deployed updated manifest for ingress-admission
  • curprev 08:01, 30 July 2021imported>Stashbot 65,430 bytes +108 majavah: replace toolsbeta-sgeexec-1002 with -1004 for T287666
  • curprev 14:08, 29 July 2021imported>Stashbot 65,322 bytes +174 majavah: add mdipietro as projectadmin T287287
  • curprev 13:31, 23 July 2021imported>Stashbot 65,148 bytes +102 majavah: upgrading toolsbeta to kubernetes 1.19, T280340
  • curprev 15:32, 22 July 2021imported>Stashbot 65,046 bytes +78 arturo: re-deploying toolforge-jobs-framework-api
  • curprev 11:58, 21 July 2021imported>Stashbot 64,968 bytes +591 arturo: deploying jobs-framework-api 07346d715d17585db9c16dd152cc91ef0bea33c3 (T286108)
  • curprev 21:18, 20 July 2021imported>Stashbot 64,377 bytes +282 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 T287037
  • curprev 19:53, 16 July 2021imported>Stashbot 64,095 bytes +227 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta T280360
  • curprev 15:39, 15 July 2021imported>Stashbot 63,868 bytes +136 arturo: deploy toolforge-jobs-framework-api git version d85d93ee1c5d4be6a526cf83e806b2679dde3875
  • curprev 09:05, 14 July 2021imported>Stashbot 63,732 bytes +92 majavah: testing calico 3.18 upgrade - T280342
  • curprev 11:42, 12 July 2021imported>Stashbot 63,640 bytes +82 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues
  • curprev 09:48, 7 July 2021imported>Stashbot 63,558 bytes +131 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work
  • curprev 17:01, 1 July 2021imported>Stashbot 63,427 bytes +111 majavah: updating jobs-framework-api
  • curprev 09:28, 29 June 2021imported>Stashbot 63,316 bytes +2,414 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. (T267140) - cookbook ran by dcaro@vulcanus
  • curprev 14:46, 28 June 2021imported>Stashbot 60,902 bytes +2,516 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
  • curprev 15:27, 25 June 2021imported>Stashbot 58,386 bytes +1,970 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
  • curprev 15:52, 24 June 2021imported>Stashbot 56,416 bytes +958 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
  • curprev 18:24, 22 June 2021imported>Stashbot 55,458 bytes +115 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18
  • curprev 11:44, 17 June 2021imported>Stashbot 55,343 bytes +148 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again
  • curprev 15:53, 16 June 2021imported>Stashbot 55,195 bytes +142 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100
  • curprev 16:10, 15 June 2021imported>Stashbot 55,053 bytes +82 majavah: set toolsbeta-bastion-05 as grid submit host
  • curprev 21:29, 14 June 2021imported>Stashbot 54,971 bytes +292 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 T282975
  • curprev 14:42, 12 June 2021imported>Stashbot 54,679 bytes +84 majavah: sync hiera key prometheus_nodes to match tools
  • curprev 15:25, 11 June 2021imported>Stashbot 54,595 bytes +159 majavah: undeploy nginx-ingress-jobs from kubernetes
  • curprev 15:11, 8 June 2021imported>Stashbot 54,436 bytes +501 majavah: updating k8s worker nodes to 1.18 T280299
  • curprev 16:55, 3 June 2021imported>Stashbot 53,935 bytes +198 majavah: renew ingress-admission-controller certificates T280301
  • curprev 17:14, 25 May 2021imported>Stashbot 53,737 bytes +344 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
  • curprev 19:40, 24 May 2021imported>Stashbot 53,393 bytes +93 andrewbogott: replacing existing etcd nodes with localdisk nodes
  • curprev 11:35, 19 May 2021imported>Stashbot 53,300 bytes +159 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
  • curprev 07:52, 15 May 2021imported>Stashbot 53,141 bytes +164 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map T262562
  • curprev 11:22, 14 May 2021imported>Stashbot 52,977 bytes +340 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] (T153810)
  • curprev 08:07, 13 May 2021imported>Stashbot 52,637 bytes +154 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / T153810
  • curprev 19:42, 10 May 2021imported>Stashbot 52,483 bytes +370 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
  • curprev 10:42, 8 May 2021imported>Stashbot 52,113 bytes +110 Majavah: create new ingress node toolsbeta-k8s-ingress-3 T264221
  • curprev 17:00, 7 May 2021imported>Stashbot 52,003 bytes +743 bstorm: deleted "toolsbeta-test-k8s-haproxy-2", "toolsbeta-test-k8s-haproxy-1" when the dns caches finally dropped T282227
  • curprev 16:25, 5 May 2021imported>Stashbot 51,260 bytes +122 Majavah: add self to sudo policy `roots`
  • curprev 10:47, 4 May 2021imported>Stashbot 51,138 bytes +89 arturo: rebase & resolve merge conflicts in labs/private.git
  • curprev 13:23, 3 May 2021imported>Stashbot 51,049 bytes +153 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 (T278109)
  • curprev 18:10, 29 April 2021imported>Stashbot 50,896 bytes +67 bstorm: added and removed an etcd node
  • curprev 17:24, 23 April 2021imported>Stashbot 50,829 bytes +117 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was "notready" for some reason
  • curprev 19:01, 20 April 2021imported>Stashbot 50,712 bytes +153 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244
  • curprev 16:41, 13 April 2021imported>Stashbot 50,559 bytes +418 arturo: create VM toolsbeta-sgeexec-1002 (T277653)
  • curprev 18:27, 8 April 2021imported>Stashbot 50,141 bytes +229 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns T277653
  • curprev 13:11, 6 April 2021imported>Stashbot 49,912 bytes +160 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number (T267082)
  • curprev 15:17, 1 April 2021imported>Stashbot 49,752 bytes +194 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
  • curprev 18:22, 31 March 2021imported>Stashbot 49,558 bytes +155 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo T275478
  • curprev 12:17, 24 March 2021imported>Stashbot 49,403 bytes +445 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
  • curprev 11:22, 23 March 2021imported>Stashbot 48,958 bytes +214 arturo: drop and build again the VM toolsbeta-sgregrid-master (T277653)
  • curprev 18:55, 18 March 2021imported>Stashbot 48,744 bytes +1,082 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
  • curprev 12:39, 17 March 2021imported>Stashbot 47,662 bytes +444 arturo: created VM toolsbeta-buster-gridmaster (T277653)
  • curprev 12:32, 16 March 2021imported>Stashbot 47,218 bytes +184 arturo: added packages jobutils / misctools v1.41 to {stretch,buster}-toolsbeta aptly repository in tools-sge-services-03
  • curprev 12:33, 11 March 2021imported>Stashbot 47,034 bytes +155 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for T275865
  • curprev 16:48, 10 March 2021imported>Stashbot 46,879 bytes +104 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor
  • curprev 20:39, 26 February 2021imported>Stashbot 46,775 bytes +705 andrewbogott: rebooting all hosts
  • curprev 12:42, 19 February 2021imported>Stashbot 46,070 bytes +495 arturo: deploying new version of the ingress admission controller
  • curprev 12:34, 10 February 2021imported>Stashbot 45,575 bytes +382 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 (T274139)
  • curprev 11:48, 8 February 2021imported>Stashbot 45,193 bytes +112 arturo: trying to introduce TLS support in the front proxy T274123
  • curprev 00:36, 5 February 2021imported>Stashbot 45,081 bytes +106 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing
  • curprev 15:29, 21 January 2021imported>Stashbot 44,975 bytes +137 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo T271847
  • curprev 14:10, 13 January 2021imported>Stashbot 44,838 bytes +384 dcaro: dcaro doing puppet tests, puppet runs might break
  • curprev 10:48, 22 December 2020imported>Stashbot 44,454 bytes +97 arturo: rebase & resolve ugly git merge conflict in labs/private.git
  • curprev 10:52, 18 December 2020imported>Stashbot 44,357 bytes +160 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 (T267966)
  • curprev 19:27, 14 December 2020imported>Stashbot 44,197 bytes +210 bstorm: create temporary instance toolsbeta-test-io-unthrottled T267966
  • curprev 23:31, 11 December 2020imported>Stashbot 43,987 bytes +157 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts
  • curprev 08:58, 10 December 2020imported>Stashbot 43,830 bytes +133 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) (T267412)
  • curprev 15:30, 9 December 2020imported>Stashbot 43,697 bytes +113 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) (T267412)
  • curprev 11:17, 4 December 2020imported>Stashbot 43,584 bytes +255 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity (T267412)
  • curprev 14:17, 23 November 2020imported>Stashbot 43,329 bytes +735 dcaro: All control nodes re-imaged (T267140)
  • curprev 11:46, 18 November 2020imported>Stashbot 42,594 bytes +202 dcaro_: Modifying the security groupts to mirror tools (T267140)
  • curprev 15:32, 17 November 2020imported>Stashbot 42,392 bytes +625 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster (T267140)
  • curprev 11:44, 16 November 2020imported>Stashbot 41,767 bytes +272 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster (T267140)
  • curprev 19:42, 10 November 2020imported>Stashbot 41,495 bytes +642 bstorm: safelisted "argocd" namespace with namespaceSelector for registry-admission controller
  • curprev 23:44, 6 November 2020imported>Stashbot 40,853 bytes +593 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 T263284
  • curprev 16:40, 5 November 2020imported>Stashbot 40,260 bytes +211 dcaro: Moving active proxy from proxy-1 to proxy-3 (T267140)
  • curprev 15:42, 4 November 2020imported>Stashbot 40,049 bytes +598 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try (T267140)
  • curprev 16:04, 3 November 2020imported>Stashbot 39,451 bytes +369 arturo: add dcaro to the toolsbeta.admin LDAP group (T266068)
  • curprev 13:35, 2 November 2020imported>Stashbot 39,082 bytes +98 arturo: added dcaro as projectadmin & user (T266068)
  • curprev 22:20, 29 October 2020imported>Stashbot 38,984 bytes +111 legoktm: switched test tool over to use buildpack image (T265681)
  • curprev 18:58, 28 October 2020imported>Stashbot 38,873 bytes +106 andrewbogott: deleting toolsbeta-puppetmaster-03 — seems broken and unused
  • curprev 16:22, 22 October 2020imported>Stashbot 38,767 bytes +87 bstorm: created buildpack psp for T265557
  • curprev 09:17, 10 September 2020imported>Stashbot 38,680 bytes +446 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
  • curprev 11:50, 9 September 2020imported>Stashbot 38,234 bytes +721 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
  • curprev 21:35, 15 July 2020imported>Stashbot 37,513 bytes +118 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion T257945
  • curprev 22:28, 14 July 2020imported>Stashbot 37,395 bytes +95 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing
  • curprev 11:08, 8 July 2020imported>Stashbot 37,300 bytes +154 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 (T234617)
  • curprev 12:12, 26 June 2020imported>Stashbot 37,146 bytes +156 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 (T120210)
  • curprev 12:55, 24 June 2020imported>Stashbot 36,990 bytes +564 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 (T120225)
  • curprev 13:10, 23 June 2020imported>Stashbot 36,426 bytes +668 arturo: added herron to the test tool for email testing
  • curprev 22:54, 16 June 2020imported>Stashbot 35,758 bytes +60 bd808: Building webservice 0.72
  • curprev 21:54, 15 June 2020imported>Stashbot 35,698 bytes +149 bstorm_: removed killgridjobs.sh from toolsbeta bastion T157792
  • curprev 19:41, 12 June 2020imported>Stashbot 35,549 bytes +132 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test T127559
  • curprev 12:42, 11 June 2020imported>Stashbot 35,417 bytes +1,241 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
  • curprev 21:06, 4 June 2020imported>Stashbot 34,176 bytes +89 andrewbogott: added krenair to toolsbeta.admin group in ldap
  • curprev 11:27, 28 May 2020imported>Stashbot 34,087 bytes +324 arturo: cleanup livehackings
  • curprev 12:02, 27 May 2020imported>Stashbot 33,763 bytes +850 arturo: the k8s cluster is now running v1.16.10 (T246122)
  • curprev 16:17, 26 May 2020imported>Stashbot 32,913 bytes +909 bstorm_: fix incorrect volume name in kubeadm-config T246122
  • curprev 22:14, 21 May 2020imported>Stashbot 32,004 bytes +92 bd808: Building tools-webservice 0.70 via wmcs-package-build.py
  • curprev 12:20, 19 May 2020imported>Stashbot 31,912 bytes +208 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 (T247422)
  • curprev 20:48, 15 May 2020imported>Stashbot 31,704 bytes +243 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now T246059
  • curprev 12:09, 14 May 2020imported>Stashbot 31,461 bytes +195 arturo: created puppet prefix toolsbeta-acme-chief in horizon (T252762)
  • curprev 18:35, 12 May 2020imported>Stashbot 31,266 bytes +418 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network T250863
  • curprev 12:48, 8 May 2020imported>Stashbot 30,848 bytes +251 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` (T120225)
  • curprev 16:33, 7 May 2020imported>Stashbot 30,597 bytes +495 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 (T251297 and T250866)
  • curprev 19:11, 6 May 2020imported>Stashbot 30,102 bytes +233 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
  • curprev 10:04, 5 May 2020imported>Stashbot 29,869 bytes +204 arturo: add herron as user and projectadmin, we will work on the email setup (T120225)
  • curprev 13:02, 4 May 2020imported>Stashbot 29,665 bytes +203 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project
  • curprev 19:48, 29 April 2020imported>Stashbot 29,462 bytes +122 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta T247455
  • curprev 14:47, 20 April 2020imported>Stashbot 29,340 bytes +435 arturo: added joakino to toolsbeta.admin LDAP group
  • curprev 19:32, 10 April 2020imported>Stashbot 28,905 bytes +170 bstorm_: deployed webservice 0.67 T249843
  • curprev 00:40, 10 April 2020imported>Stashbot 28,735 bytes +91 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up
  • curprev 01:10, 8 April 2020imported>Stashbot 28,644 bytes +105 bstorm_: upgrade toollabs-webservice to 0.66 for qa T249390
  • curprev 23:39, 31 March 2020imported>Stashbot 28,539 bytes +84 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta
  • curprev 10:35, 30 March 2020imported>Stashbot 28,455 bytes +256 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
  • curprev 18:45, 24 March 2020imported>Stashbot 28,199 bytes +154 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) T243327
  • curprev 23:18, 19 March 2020imported>Stashbot 28,045 bytes +253 Krenair: Shut down toolsbeta-puppet(db-01|master-02) - T241719
  • curprev 21:38, 16 March 2020imported>Stashbot 27,792 bytes +1,212 bstorm_: removed lots of hiera related to the legacy k8s cluster T246689
  • curprev 21:32, 11 March 2020imported>Stashbot 26,580 bytes +92 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta
  • curprev 13:11, 9 March 2020imported>Stashbot 26,488 bytes +167 arturo: created VM `toolsbeta-legacy-redirector` (T247236)
  • curprev 16:22, 6 March 2020imported>Stashbot 26,321 bytes +100 bstorm_: updating maintain-kubeusers image to filter invalid tool names
  • curprev 21:22, 5 March 2020imported>Stashbot 26,221 bytes +118 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test
  • curprev 19:19, 27 February 2020imported>Stashbot 26,103 bytes +247 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
  • curprev 20:35, 26 February 2020imported>Stashbot 25,856 bytes +128 bstorm_: hard rebooting the grid master for toolsbeta
  • curprev 23:20, 18 February 2020imported>Stashbot 25,728 bytes +194 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists
  • curprev 21:19, 10 February 2020imported>Stashbot 25,534 bytes +204 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 T244293 T244289 T234617 T156626
  • curprev 23:07, 7 February 2020imported>Stashbot 25,330 bytes +304 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 T244611
  • curprev 03:14, 23 January 2020imported>Stashbot 25,026 bytes +248 bd808: Demoted projectadmins not listed in the "roots" sudoer policy to project members just to avoid random confusion
  • curprev 11:59, 22 January 2020imported>Stashbot 24,778 bytes +142 arturo: remove toolviews scripts from toolsbeta-proxy-{1,2}, source of cronspam
  • curprev 12:49, 21 January 2020imported>Stashbot 24,636 bytes +345 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
  • curprev 12:52, 17 January 2020imported>Stashbot 24,291 bytes +295 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 (T234617)
  • curprev 23:13, 16 January 2020imported>Stashbot 23,996 bytes +245 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
  • curprev 02:15, 14 January 2020imported>Stashbot 23,751 bytes +122 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch
  • curprev 16:41, 13 January 2020imported>Stashbot 23,629 bytes +160 bstorm_: There was a filesystem unclean and other problems on the "old cluster" worker node 1001. Rebooting it in case that helps.
  • curprev 21:05, 10 January 2020imported>Stashbot 23,469 bytes +93 bstorm_: updated toollabs-webservice package to 0.55 for testing
  • curprev 15:51, 7 January 2020imported>Stashbot 23,376 bytes +171 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster T242067
  • curprev 21:42, 6 January 2020imported>Stashbot 23,205 bytes +101 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things
  • curprev 17:46, 3 January 2020imported>Stashbot 23,104 bytes +242 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
  • curprev 22:37, 2 January 2020imported>Stashbot 22,862 bytes +286 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
  • curprev 00:14, 17 December 2019imported>Stashbot 22,576 bytes +196 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes
  • curprev 13:02, 4 December 2019imported>Stashbot 22,380 bytes +200 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
  • curprev 10:38, 2 December 2019imported>Stashbot 22,180 bytes +447 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
  • curprev 14:46, 15 November 2019imported>Stashbot 21,733 bytes +109 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 T237643
  • curprev 10:32, 14 November 2019imported>Stashbot 21,624 bytes +173 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug
  • curprev 19:36, 8 November 2019imported>Stashbot 21,451 bytes +324 bstorm_: rebooted the proxy server just in case that fixes something.
  • curprev 21:55, 7 November 2019imported>Stashbot 21,127 bytes +131 bstorm_: killed pods for ingress admission controller to upgrade to new image T215531
  • curprev 22:39, 6 November 2019imported>Stashbot 20,996 bytes +813 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster T215531
  • curprev 22:50, 5 November 2019imported>Stashbot 20,183 bytes +136 bstorm_: deployed the new maintain-kubeusers to toolsbeta T215531 T228499
  • curprev 23:41, 25 October 2019imported>Stashbot 20,047 bytes +281 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster T215531 T215678 T234231
  • curprev 12:04, 23 October 2019imported>Stashbot 19,766 bytes +486 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` T236074
  • curprev 17:43, 22 October 2019imported>Stashbot 19,280 bytes +1,024 arturo: re-create VM `toolsbeta-test-k8s-control-1` T236074
  • curprev 15:13, 21 October 2019imported>Stashbot 18,256 bytes +343 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers T236074
  • curprev 16:33, 18 October 2019imported>Stashbot 17,913 bytes +451 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
  • curprev 12:26, 14 October 2019imported>Stashbot 17,462 bytes +214 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
  • curprev 11:59, 9 October 2019imported>Stashbot 17,248 bytes +114 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster (T235059)
  • curprev 14:14, 8 October 2019imported>Stashbot 17,134 bytes +240 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to T234037
  • curprev 19:12, 7 October 2019imported>Stashbot 16,894 bytes +157 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue
  • curprev 23:31, 25 September 2019imported>Stashbot 16,737 bytes +123 bd808: Updated user list for "roots" sudoer policy
  • curprev 15:09, 5 September 2019imported>Stashbot 16,614 bytes +221 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam
  • curprev 20:40, 12 August 2019imported>Stashbot 16,393 bytes +112 phamhi: toolsbeta-test-puppet-sandbox instance created for T230147
  • curprev 10:51, 9 August 2019imported>Stashbot 16,281 bytes +136 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003
  • curprev 20:48, 24 July 2019imported>Stashbot 16,145 bytes +240 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container T228887 T215531
  • curprev 16:04, 18 July 2019imported>Stashbot 15,905 bytes +422 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
  • curprev 09:51, 17 July 2019imported>Stashbot 15,483 bytes +214 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster T215531
  • curprev 12:29, 15 July 2019imported>Stashbot 15,269 bytes +162 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
  • curprev 10:49, 3 July 2019imported>Stashbot 15,107 bytes +349 arturo: recreate `toolsbeta-test-k8s-master-1` VM (T215531)
  • curprev 17:24, 2 July 2019imported>Stashbot 14,758 bytes +253 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` (T215531)
  • curprev 17:13, 1 July 2019imported>Stashbot 14,505 bytes +536 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for T215531
  • curprev 19:10, 28 June 2019imported>Stashbot 13,969 bytes +160 bstorm_: T215531 removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm
  • curprev 10:35, 25 June 2019imported>Stashbot 13,809 bytes +208 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for T215531
  • curprev 11:42, 21 June 2019imported>Stashbot 13,601 bytes +139 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in T226098
  • curprev 10:39, 19 June 2019imported>Stashbot 13,462 bytes +110 arturo: add myself to the `toolsbeta.admin` LDAP group (T225303)
  • curprev 16:24, 14 June 2019imported>Stashbot 13,352 bytes +465 bstorm_: Manually failed "back" to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
  • curprev 12:14, 6 June 2019imported>Stashbot 12,887 bytes +283 arturo: T215531 create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
  • curprev 12:40, 5 June 2019imported>Stashbot 12,604 bytes +812 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
  • curprev 11:49, 24 May 2019imported>Stashbot 11,792 bytes +558 arturo: T224273 create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
  • curprev 10:22, 7 May 2019imported>Stashbot 11,234 bytes +286 arturo: T219362 drop the `toolsbeta-exec` puppet prefix
  • curprev 04:17, 25 April 2019imported>Stashbot 10,948 bytes +892 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver: toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master
  • curprev 14:58, 20 February 2019imported>Stashbot 10,056 bytes +204 andrewbogott: moving toolsbeta-grid-master and toolsbeta-puppetmaster-02 to labvirt1003
  • curprev 18:43, 4 December 2018imported>Stashbot 9,852 bytes +132 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/
  • curprev 13:26, 26 November 2018imported>Stashbot 9,720 bytes +186 arturo: T210098 VM=toolsbeta-sgebastion-03
  • curprev 10:01, 8 November 2018imported>Stashbot 9,534 bytes +136 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically T207970)
  • curprev 21:21, 22 October 2018imported>Stashbot 9,398 bytes +86 bstorm_: launched a stretch/sonofgridengine master server
  • curprev 20:11, 19 September 2018imported>Stashbot 9,312 bytes +449 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- T200557
  • curprev 14:28, 22 July 2018imported>Stashbot 8,863 bytes +3,719 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug T156626
  • curprev 18:07, 14 March 2018imported>Stashbot 5,144 bytes +125 chicocvenancio: updated paws-beta k8s cluster and nodes to v1.9.4 for T189680
  • curprev 19:33, 5 March 2018imported>Stashbot 5,019 bytes +80 chicocvenancio: added Zhuyifei1999 as project admin
  • curprev 01:11, 9 February 2018imported>Stashbot 4,939 bytes +96 bd808: Removed Yuvipanda at user request (T186289)
  • curprev 14:09, 7 August 2017imported>Stashbot 4,843 bytes +87 andrewbogott: deleted etcd-k8s-CTEST and k8s-master-CTEST
  • curprev 15:38, 26 April 2017imported>Stashbot 4,756 bytes +4,756 madhuvishy: add Madhuvishy as projectadmin