You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Toolhub.wikimedia.org: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(→‎Testing deployment: fix gerrit link)
imported>BryanDavis
(→‎How to deploy: add info on attaching to container with admin rights)
 
(4 intermediate revisions by the same user not shown)
Line 12: Line 12:
* [[Url-downloader|url-downloader.{codfw,eqiad}.wikimedia.org]] outbound HTTP proxy
* [[Url-downloader|url-downloader.{codfw,eqiad}.wikimedia.org]] outbound HTTP proxy
* [[Varnish|text-lb.{codfw,eqiad}.wikimedia.org]] for MediaWiki API access without [[Envoy#Services_Proxy|envoy proxy + custom Host headers]]
* [[Varnish|text-lb.{codfw,eqiad}.wikimedia.org]] for MediaWiki API access without [[Envoy#Services_Proxy|envoy proxy + custom Host headers]]
=== Monitoring ===
* [https://grafana.wikimedia.org/d/wJHvm8Ank/toolhub Grafana dashboard of Prometheus metrics]
* [https://logstash.wikimedia.org/app/dashboards#/view/d2def360-dd33-11ec-a18d-b1811c123c8d OpenSearch dashboard of log events]
=== How to deploy ===
{{See also|Kubernetes/Deployments}}
# Create a patch to update main_app.version in {{Gitweb|project=operations/deployment-charts|file=helmfile.d/services/toolhub/values.yaml|text=helmfile.d/services/toolhub/values.yaml}} to the desired tag of the [https://docker-registry.wikimedia.org/wikimedia/wikimedia-toolhub/tags/ wikimedia/wikimedia-toolhub container]. ([[gerrit:771070|example]])
# CR+2 the helmfile.d patch in gerrit to start the merge process.
# Ssh to the currently active [[Deployment server|deployment server]] (''deployment.eqiad.wmnet'').
# ''cd /srv/deployment-charts/helmfile.d/services/toolhub''
# Verify that your changes from step 1 are checked out. The git clone is updated once a minute by a timer, so if your changes are not present yet double check that zuul has merged the change and then wait patiently for the update to be fetched.
# Deploy the chart to the staging cluster: ''helmfile -e staging -i apply''
# Deploy the chart to the codfw cluster: ''helmfile -e codfw -i apply''
# Deploy the chart to the eqiad cluster: ''helmfile -e eqiad -i apply''
If your deployment includes database or elasticsearch schema changes, you will need to apply them manually.
* If you happen to have root rights on the deployment server you can use <code>kube_env admin eqiad</code> to get credentials that will allow you to use <code>kubectl exec -n toolhub -it <pod name> toolhub-main -- /bin/bash</code> to attach to a running container.
* If you are not a root you will need to invent your own environment for performing these post-deploy tasks until [[phab:T290357|T290357: Maintenance environment needed for running one-off commands]] is resolved. See [[phab:T290357#7578866|T290357#7578866]] for the script that {{U|BryanDavis|bd808}} uses to run a container on his laptop connected to the production database and elasticsearch instance via ssh tunnels.
Typical migration commands:
* ''DB_USER=toolhub_admin DB_PASSWORD=<toolhub_admin's password> poetry run ./manage.py showmigrations''
* ''DB_USER=toolhub_admin DB_PASSWORD=<toolhub_admin's password> poetry run ./manage.py migrate''
* ''poetry run ./manage.py search_index --rebuild''


== Testing deployment ==
== Testing deployment ==
https://toolhub-demo.wmcloud.org in the [[Nova Resource:Toolhub|Toolhub Cloud VPS project]] hosts a testing deployment that is used for beta testing and demonstrations. It is deployed using docker-compose. See {{Gitweb|project=wikimedia/toolhub|branch=main|file=contrib/demo-server/README.md|text=contrib/demo-server/README.md}} in the project's git repo for more information.
https://toolhub-demo.wmcloud.org in the [[Nova Resource:Toolhub|Toolhub Cloud VPS project]] hosts a testing deployment that is used for beta testing and demonstrations. It is deployed using docker-compose. See {{Gitweb|project=wikimedia/toolhub|branch=main|file=contrib/demo-server/README.md|text=contrib/demo-server/README.md}} in the project's git repo for more information.
=== Updating the demo build ===
{{Codesample|scheme=light|lang=shell-session|code=
$ ssh toolhub-demo01.toolhub.eqiad1.wikimedia.cloud
$ cd /srv/toolhub/demo/
$ make pull restart tail
  watch things happen until the search indices are rebuilt
  ^c
$ make web-shell
$ poetry run ./manage.py crawl --quiet
$ exit
$ exit
}}


== Local development ==
== Local development ==
Local development is done with docker-compose. See {{Gitweb|project=wikimedia/toolhub|file=docs/CONTRIBUTING.rst|text=docs/CONTRIBUTING.rst}} in the project's git repo for more information.
Local development is done with docker-compose. See {{Gitweb|project=wikimedia/toolhub|branch=main|file=docs/CONTRIBUTING.rst|text=docs/CONTRIBUTING.rst}} in the project's git repo for more information.


== Visualizations related to Tools and Account creation ==
== Visualizations related to Tools and Account creation ==

Latest revision as of 21:11, 1 June 2022

Toolhub logo

toolhub.wikimedia.org is a catalog of tools used by the members of the Wikimedia movement. See the Toolhub project page on meta for additional details about the application's features.

Production deployment

Toolhub runs in the "eqiad" Kubernetes cluster. The service is current not active-active in the "codfw" cluster (T288685).

The helm chart and helmfile configuration for the service configure connectivity to its dependencies:

Monitoring

How to deploy

  1. Create a patch to update main_app.version in helmfile.d/services/toolhub/values.yaml to the desired tag of the wikimedia/wikimedia-toolhub container. (example)
  2. CR+2 the helmfile.d patch in gerrit to start the merge process.
  3. Ssh to the currently active deployment server (deployment.eqiad.wmnet).
  4. cd /srv/deployment-charts/helmfile.d/services/toolhub
  5. Verify that your changes from step 1 are checked out. The git clone is updated once a minute by a timer, so if your changes are not present yet double check that zuul has merged the change and then wait patiently for the update to be fetched.
  6. Deploy the chart to the staging cluster: helmfile -e staging -i apply
  7. Deploy the chart to the codfw cluster: helmfile -e codfw -i apply
  8. Deploy the chart to the eqiad cluster: helmfile -e eqiad -i apply

If your deployment includes database or elasticsearch schema changes, you will need to apply them manually.

  • If you happen to have root rights on the deployment server you can use kube_env admin eqiad to get credentials that will allow you to use kubectl exec -n toolhub -it <pod name> toolhub-main -- /bin/bash to attach to a running container.
  • If you are not a root you will need to invent your own environment for performing these post-deploy tasks until T290357: Maintenance environment needed for running one-off commands is resolved. See T290357#7578866 for the script that bd808 uses to run a container on his laptop connected to the production database and elasticsearch instance via ssh tunnels.

Typical migration commands:

  • DB_USER=toolhub_admin DB_PASSWORD=<toolhub_admin's password> poetry run ./manage.py showmigrations
  • DB_USER=toolhub_admin DB_PASSWORD=<toolhub_admin's password> poetry run ./manage.py migrate
  • poetry run ./manage.py search_index --rebuild

Testing deployment

https://toolhub-demo.wmcloud.org in the Toolhub Cloud VPS project hosts a testing deployment that is used for beta testing and demonstrations. It is deployed using docker-compose. See contrib/demo-server/README.md in the project's git repo for more information.

Updating the demo build

$ ssh toolhub-demo01.toolhub.eqiad1.wikimedia.cloud
$ cd /srv/toolhub/demo/
$ make pull restart tail
  watch things happen until the search indices are rebuilt
  ^c
$ make web-shell
$ poetry run ./manage.py crawl --quiet
$ exit
$ exit

Local development

Local development is done with docker-compose. See docs/CONTRIBUTING.rst in the project's git repo for more information.

Visualizations related to Tools and Account creation