You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

User:Aklapper/Sandbox: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Aklapper
No edit summary
imported>Aklapper
Line 29: Line 29:


=== Toolforge ===
=== Toolforge ===
TODO: Merge all those different definitions down here:


[[File:Toolforge_logo.svg|right|frameless|60px|alt=Toolforge|link=Portal:Toolforge]]
[[File:Toolforge_logo.svg|right|frameless|60px|alt=Toolforge|link=Portal:Toolforge]]


[[Portal:Toolforge|Toolforge]] provides a shared hosting/platform-as-a-service environment for running bots, webservices, scheduled jobs, and data analysis.
''[[Portal:Toolforge|Toolforge]]'' is one of the projects hosted by Wikimedia Cloud VPS. It is a shared hosting (platform as a service) environment for volunteers to develop and run [https://admin.toolforge.org/tools tools], [[mw:Manual:Creating a bot|continuous bots]], web services, scheduled jobs, and data analysis.
 
---
 
'''Toolforge''' is a hosting environment for volunteers to develop and operate [https://admin.toolforge.org/tools tools], [[mw:Manual:Creating_a_bot |continuous bots]], web services, and scheduled tasks.
 
Tool Accounts can (and should) have multiple, active co-maintainers. Anyone can create or join a Toolforge account. Toolforge accounts cannot be deleted.
 
To use Toolforge you will need a Wikimedia login and Wikimedia Developer account. You will need some programming knowledge, an understanding of Unix command line, and version control via Gerritt and Git. Kubernetes.
 
Acts as a standard web server hosting web-based tools, command-line tools, account required.


---
To use Toolforge you will need some programming knowledge, an understanding of Unix command line, and version control via Gerrit and Git.


Toolforge is one of the projects hosted by Wikimedia Cloud VPS. The Toolforge administrators manage a pool of virtual servers that provide a shared project hosting environment that can be used by Toolforge users. These resources include [[Help:Toolforge/Web|web servers]], [[Help:Toolforge/Database|databases]] and [[Help:Toolforge#Redis|other data storage]], and a [[Help:Toolforge/Grid|distributed job processing system]]. These services provide a reliable and scalable hosting environment for volunteers to develop and operate their tools and bots.
Users of the Toolforge project create so-called "tool" accounts (technically ''service groups'') which allow one or more users to collaborate to manage the software source code, configuration, and jobs for that tool or bot.


Users of the Toolforge project create "tool" accounts (technically [[#What_is_a_Service_group.3F|service groups]]) which allow one or more users to collaborate to manage the software source code, configuration, and jobs for that tool or bot.
The Toolforge administrators manage a pool of virtual servers that provide a shared project hosting environment that can be used by Toolforge users. These resources include [[Help:Toolforge/Web|web servers]], [[Help:Toolforge/Database|databases]] and [[Help:Toolforge#Redis|other data storage]], and a [[Help:Toolforge/Grid|distributed job processing system]]. These services provide a reliable and scalable hosting environment for volunteers to develop and operate their tools and bots.


For additional documentation and help with Toolforge, see [[Portal:Toolforge]].
For additional documentation and help with Toolforge, see [[Portal:Toolforge]].


=== Cloud VPS ===
=== Cloud VPS ===
TODO: Merge all those different definitions down here:


[[File:Wikimedia_Cloud_Services_logo.svg|right|frameless|60px|alt=Cloud Services|link=Portal:Cloud VPS]]
[[File:Wikimedia_Cloud_Services_logo.svg|right|frameless|60px|alt=Cloud Services|link=Portal:Cloud VPS]]


''[[Portal:Cloud VPS|Cloud VPS]]'' provides collaboratively owned collections of virtual private servers where users develop and maintain software projects that help the Wikimedia movement.
''[[Portal:Cloud VPS|Cloud VPS]]'' (Virtual Private Server) is a [[:w:Cloud computing|cloud computing]] environment powered by [[:w:OpenStack|OpenStack]]. It offers collaboratively owned collections of virtual private servers. You can use this infrastructure to create and maintain open source software projects that help the [[:meta:Wikimedia movement|Wikimedia movement]].


---
The environment includes access to a variety of data services. Cloud VPS allows developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the Toolforge environment.


''Cloud VPS'' (Virtual Private Server) provides a cloud computing infrastructure powered by OpenStack for projects related to the Wikimedia movement. The environment includes access to a variety of data services. Cloud VPS is meant to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the Toolforge environment.
Cloud VPS is for the advanced users to get involved in Wikimedia operations and software development. Cloud VPS contains [https://openstack-browser.toolforge.org/project/ many projects], each of which uses one or more instances.
 
Cloud VPS is for the advanced users / volunteers to get involved in Wikimedia operations and software development. Cloud VPS contains [https://openstack-browser.toolforge.org/project/ many projects], each of which uses one or more instances.
 
Cloud VPS projects can have multiple co-administrators. It uses Horizon.


Cloud VPS instances must go through a request and approval processes. Instances are not permanent and are reviewed periodically for potential deletion/removal. Cloud VPS instances are resource intensive. Before requesting, explore whether Toolforge or another service will adequately meet your needs.
Cloud VPS instances must go through a request and approval processes. Instances are not permanent and are reviewed periodically for potential deletion/removal. Cloud VPS instances are resource intensive. Before requesting, explore whether Toolforge or another service will adequately meet your needs.
---
Wikimedia ''Cloud VPS'' is a [[:w:Cloud computing|cloud computing]] infrastructure powered by [[:w:OpenStack|OpenStack]] for projects related to the [[:meta:Wikimedia movement|Wikimedia movement]]. Volunteer contributors can use this infrastructure to create and maintain open source software projects.
Cloud VPS is for the volunteers to get involved in Wikimedia operations and software development. Cloud VPS contains many projects, each of which uses one or more instances. [[Nova_Resource:Tools|Toolforge]] is one of the most important projects in Wikimedia Cloud VPS.


==== How is Cloud VPS organized? ====
==== How is Cloud VPS organized? ====
Line 88: Line 64:
====How does Cloud VPS work? ====
====How does Cloud VPS work? ====


Cloud VPS is a virtualization cluster and hosts various virtual machines (called instances) using [http://www.openstack.org/software/openstack-compute OpenStack Compute]. This is slightly different from your normal servers that you ssh to (i.e. Toolserver), as virtual machines do not exist physically, but reside inside a much bigger machine called the host machine.  More details about the physical setup of Cloud VPS can be found under [[Portal:Cloud_VPS/Infrastructure]].
Cloud VPS is a virtualization cluster and hosts various virtual machines (called instances) using [http://www.openstack.org/software/openstack-compute OpenStack Compute]. This is slightly different from your normal servers that you ssh to (i.e. Toolserver), as virtual machines do not exist physically, but reside inside a much bigger machine called the host machine.  More details about the physical setup of Cloud VPS can be found under [[Portal:Cloud VPS/Infrastructure]].


=== What is the difference between Cloud VPS and Toolforge? ===
=== What is the difference between Cloud VPS and Toolforge? ===
TODO: Remove this section! Is this sufficiently covered by "what service do I want?" section?


Cloud VPS is an [[:w:Cloud_computing#Infrastructure_as_a_service_.28IaaS.29|Infrastructure as a service (IaaS)]] solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.
Cloud VPS is an [[:w:Cloud_computing#Infrastructure_as_a_service_.28IaaS.29|Infrastructure as a service (IaaS)]] solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.
Line 233: Line 207:
* '''[[Help:Create a Wikimedia developer account|Wikimedia developer account]]''' — this account is used to log into this wiki, Toolforge, Cloud VPS, [[mw:Gerrit|Gerrit]] (our code review system for patches) and other protected Wikimedia Services. When you create your Wikimedia developer account, you will create a username (sometimes called LDAP username), UNIX shell username, and password.
* '''[[Help:Create a Wikimedia developer account|Wikimedia developer account]]''' — this account is used to log into this wiki, Toolforge, Cloud VPS, [[mw:Gerrit|Gerrit]] (our code review system for patches) and other protected Wikimedia Services. When you create your Wikimedia developer account, you will create a username (sometimes called LDAP username), UNIX shell username, and password.
** Note that while [[mw:Gerrit/GitHub|GitHub]] contains many of our public repos, you can only make pull requests for Cloud Services projects via Gerrit. Other wiki projects may use GitHub exclusively.
** Note that while [[mw:Gerrit/GitHub|GitHub]] contains many of our public repos, you can only make pull requests for Cloud Services projects via Gerrit. Other wiki projects may use GitHub exclusively.
* '''Gerrit''': Once you have set up the two accounts above, including your UNIX shell username, [[mw:Gerrit/Tutorial#Set Up_ SSH Keys in Gerrit|set up your SSH keys in Gerrit]].
* '''Gerrit''' Once you have set up the two accounts above, including your UNIX shell username, [[mw:Gerrit/Tutorial#Set Up_ SSH Keys in Gerrit|set up your SSH keys in Gerrit]].


You may also want to [[mw:Phabricator/Help|create an account]] in '''[[mw:Phabricator|Phabricator]]''', our project management system for tasks and bug reports.
You may also want to [[mw:Phabricator/Help|create an account]] in '''[[mw:Phabricator|Phabricator]]''', our project management system for tasks and bug reports.
Line 272: Line 246:


== Communication and support ==
== Communication and support ==
We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.
Please reach out with questions and join the conversation:


{{ContentGrid
{{ContentGrid
|content=
|content=
{{Colored box
{{Colored box
|title = Discuss and receive general support via real-time chat
|title = Discuss and receive general support
|content = Use the [[Help:IRC|IRC channel]] {{Irc|wikimedia-cloud}}, the bridged [https://t.me/wmcloudirc Telegram channel], or the bridged [https://chat.wmcloud.org/wikimedia/channels/wikimedia-cloud Mattermost channel]
|content = * Chat in real time in the [[Help:IRC|IRC channel]] {{Irc|wikimedia-cloud}}, the bridged [https://t.me/wmcloudirc Telegram channel], or the bridged [https://chat.wmcloud.org/wikimedia/channels/wikimedia-cloud Mattermost channel]
}}
* Discuss via email after you subscribed to the [[mail:cloud|cloud@]] [[meta:Special:MyLanguage/Mailing lists|mailing list]]
{{Colored box
|title = Discuss and receive general support via email, and receive information about ongoing initiatives
|content = Subscribe to the [[mail:cloud|cloud@]] [[meta:Special:MyLanguage/Mailing lists|mailing list]]
}}
}}
{{Colored box
{{Colored box
Line 290: Line 261:
{{Colored box
{{Colored box
|title = Track work tasks and [[mw:Special:MyLanguage/How to report a bug|report bugs]]
|title = Track work tasks and [[mw:Special:MyLanguage/How to report a bug|report bugs]]
|content = Use the [[mw:Special:MyLanguage/Phabricator|Phabricator]] workboard [[phab:project/profile/832/|#Cloud-Services]]
|content = Use the [[mw:Special:MyLanguage/Phabricator|Phabricator]] workboard [[phab:project/profile/832/|#Cloud-Services]] for bug reports and feature requests about the Cloud VPS infrastructure itself
}}
}}
{{Colored box
{{Colored box
|title = Read about major near-term plans
|title = Learn about major near-term plans
|content = See the [[News]] wiki page
|content = Read the [[News]] wiki page
}}
}}
{{Colored box
{{Colored box
|title = Read news and stories about our work
|title = Read news and stories about our work
|content = See the [[phab:phame/blog/view/5/|Cloud Services Blog]] (For the broader Wikimedia movement, see the [https://techblog.wikimedia.org/ Wikimedia Technical Blog])
|content = Read the [[phab:phame/blog/view/5/|Cloud Services Blog]] (for the broader Wikimedia movement, see the [https://techblog.wikimedia.org/ Wikimedia Technical Blog])
}}
}}
}}
}}
Line 306: Line 277:


WMCS is a computing ecosystem built on [[w:OpenStack|OpenStack]], [[w:Oracle Grid Engine|GridEngine]], and [[w:Kubernetes|Kubernetes]].
WMCS is a computing ecosystem built on [[w:OpenStack|OpenStack]], [[w:Oracle Grid Engine|GridEngine]], and [[w:Kubernetes|Kubernetes]].
Cloud VPS projects use [[Help:Horizon FAQ|Horizon]].


== Learn more ==
== Learn more ==
Line 311: Line 283:
* [[Portal:Cloud VPS|Cloud VPS Portal]] — Information about Cloud VPS and links to help and technical documentation.
* [[Portal:Cloud VPS|Cloud VPS Portal]] — Information about Cloud VPS and links to help and technical documentation.
* [[Portal:Data Services|Data Services Portal]] — Information about Data Services and links to help and technical documentation.
* [[Portal:Data Services|Data Services Portal]] — Information about Data Services and links to help and technical documentation.
* Some terms are specific to Toolforge and Cloud VPS. See the [[Help:Glossary|Glossary]] for detailed definitions.
* See the [[Help:Glossary|Glossary]] for detailed definitions of terms which are specific to Toolforge and Cloud VPS.


== Historical information ==
== Historical information ==

Revision as of 00:57, 29 January 2022





Poster-format overview

Overview

Wikimedia Cloud Services (WMCS) provides tools, services, and support for technical collaborators who want to contribute to Wikimedia software projects.

It is available for use by anyone connected with the Wikimedia movement without charge. Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers.

Find out more:

(TODO) Examples of what you can do by running software on Cloud Services include:

  • Build a tool that suggests wiki pages that need editing/maintenance
  • Find out which articles on a Wikipedia mention cats

Services

Decide which service is best for your needs.

Toolforge

Toolforge

Toolforge is one of the projects hosted by Wikimedia Cloud VPS. It is a shared hosting (platform as a service) environment for volunteers to develop and run tools, continuous bots, web services, scheduled jobs, and data analysis.

To use Toolforge you will need some programming knowledge, an understanding of Unix command line, and version control via Gerrit and Git.

Users of the Toolforge project create so-called "tool" accounts (technically service groups) which allow one or more users to collaborate to manage the software source code, configuration, and jobs for that tool or bot.

The Toolforge administrators manage a pool of virtual servers that provide a shared project hosting environment that can be used by Toolforge users. These resources include web servers, databases and other data storage, and a distributed job processing system. These services provide a reliable and scalable hosting environment for volunteers to develop and operate their tools and bots.

For additional documentation and help with Toolforge, see Portal:Toolforge.

Cloud VPS

Cloud Services

Cloud VPS (Virtual Private Server) is a cloud computing environment powered by OpenStack. It offers collaboratively owned collections of virtual private servers. You can use this infrastructure to create and maintain open source software projects that help the Wikimedia movement.

The environment includes access to a variety of data services. Cloud VPS allows developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the Toolforge environment.

Cloud VPS is for the advanced users to get involved in Wikimedia operations and software development. Cloud VPS contains many projects, each of which uses one or more instances.

Cloud VPS instances must go through a request and approval processes. Instances are not permanent and are reviewed periodically for potential deletion/removal. Cloud VPS instances are resource intensive. Before requesting, explore whether Toolforge or another service will adequately meet your needs.

How is Cloud VPS organized?

Cloud VPS is divided into projects. Each project has separate members and administrators who can create and maintain virtual machines ("instances") for use by that project. Each project can have own its own access policies, DNS records, etc.

What is a Cloud VPS project?

A project is a unit of privilege separation inside the Cloud VPS environment. Each project has separate management of membership, virtual machines, HTTPS proxies, firewall rules, etc. Examples of projects include Toolforge and the Beta Cluster.

How does Cloud VPS work?

Cloud VPS is a virtualization cluster and hosts various virtual machines (called instances) using OpenStack Compute. This is slightly different from your normal servers that you ssh to (i.e. Toolserver), as virtual machines do not exist physically, but reside inside a much bigger machine called the host machine. More details about the physical setup of Cloud VPS can be found under Portal:Cloud VPS/Infrastructure.

What is the difference between Cloud VPS and Toolforge?

Cloud VPS is an Infrastructure as a service (IaaS) solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.

Toolforge is a Platform as a service (PaaS) solution. It provides web servers, databases and other data storage, and a distributed job processing system as managed services that can be used by tools and their maintainers.

Data Services

Data Services are a collection of products including private-information-redacted copies of Wikimedia's production wiki databases and access to Wikimedia Dumps. Use data services to create replicas of the production databases and other data for analysis and experimentation.

There are also services to interact with data in a web browser:

Quarry

Quarry

Quarry is a public querying interface for Wiki Replicas, a set of live replica SQL databases of public Wikimedia Wikis. Quarry is designed to make running queries against Wiki Replicas easy. Quarry also provides a means for researchers to share and review each other's queries.

Quarry queries are run by individual users. They can be saved and published and forked by other users.

To use Quarry you need only a Wikimedia login and a web browsers. Quarry can be used by individuals with understanding along the technical spectrum. A basic understanding of SQL is recommended. Learn about SQL queries.

PAWS

PAWS

PAWS is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running bots using Pywikibot, and more.

PAWS notebooks are maintained by a single user. They can be downloaded and forked by other users.

To use PAWS you need only a Wikimedia login and a web browser. PAWS can be used by individuals with understanding along the technical spectrum. A knowledge of Python is helpful, but not required.

Which service is right for you?

Activity / Needs Quarry (DaaS) PAWS (DaaS) Toolforge (PaaS) Cloud VPS (IaaS)
Browser based
Terminal based
Write queries against replica databases
Run database dumps
Write and run bots
Run webservices
Build tools to improve Wikimedia projects
Schedule or run continuous jobs
Administrate your own virtual server
Need your own subdomain
Write documentation and create tutorials
Work with co-maintainers and co-admins
User knowledge curious—advanced curious—advanced intermediate—advanced advanced
Service concept Data as a service Data as a service Platform as a service Infrastructure as a service

Get started

Make sure to review and agree to our terms and conditions. Account Holders who plan to use WMCS resources and products must read and agree to the following:

Please pay close attention to the following terms for Toolforge and Cloud VPS:

Set up your accounts

  • Wikimedia account — this is the single user login (SUL) account you use to contribute to Wikipedia and its sister projects. When you create your Wikimedia account, you will create a username and password.
  • Wikimedia developer account — this account is used to log into this wiki, Toolforge, Cloud VPS, Gerrit (our code review system for patches) and other protected Wikimedia Services. When you create your Wikimedia developer account, you will create a username (sometimes called LDAP username), UNIX shell username, and password.
    • Note that while GitHub contains many of our public repos, you can only make pull requests for Cloud Services projects via Gerrit. Other wiki projects may use GitHub exclusively.
  • Gerrit — Once you have set up the two accounts above, including your UNIX shell username, set up your SSH keys in Gerrit.

You may also want to create an account in Phabricator, our project management system for tasks and bug reports.

Get started with Toolforge

{{#lst:Portal:Toolforge/Quickstart|quickstart}}

Get started with Cloud VPS projects

Join an existing project

  1. Choose a project to join with OpenStack browser.
  2. Request membership by creating a Phabricator task and assigning it directly to the project administrator(s).
    • You can find the list of project admins by going to https://openstack-browser.toolforge.org/project/<project-name>.

Create a new project

  1. Follow the instructions on the "Cloud-VPS (Project-requests)" phabricator task.

Add members and admin users to a project

  1. Project admins can add new members or grant administrative permissions to members via https://horizon.wikimedia.org/project/member/
  2. Log your actions in #wikimedia-cloud connect that you added/granted admin permissions to the member.

Access an instance

See Help:Accessing Cloud VPS instances.

Learn about project instances

To learn more about project instances, read the project instances documentation.

Log your actions

It is best practice to log changes to all instances of your project. Wikimedia Cloud Services provides a Server Admin Log for users to log their project server administration actions on.

You can add a log entry in the #wikimedia-cloud connect channel on Libera Chat by sending a message like: !log <projectname> <message>

Communication and support

Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about our work
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

(TODO: Remixed from Help:Cloud Services communication; merge back into template if acceptable)

Technology stack

WMCS is a computing ecosystem built on OpenStack, GridEngine, and Kubernetes. Cloud VPS projects use Horizon.

Learn more

  • Toolforge Portal — Information about Toolforge and links to help and technical documentation.
  • Cloud VPS Portal — Information about Cloud VPS and links to help and technical documentation.
  • Data Services Portal — Information about Data Services and links to help and technical documentation.
  • See the Glossary for detailed definitions of terms which are specific to Toolforge and Cloud VPS.

Historical information

From 2011 until early 2017, Wikimedia Cloud Services was known as Wikimedia Labs. However, the term Labs was used for several different things.

Since 2017, the former Wikimedia Foundation Labs team and Tool Labs Support team merged into the Wikimedia Cloud Services team.