You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Help:Cloud Services introduction: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>JJMC89
m (Reverted edits by Hatta2 (talk) to last revision by Majavah)
imported>Majavah
(5 intermediate revisions by 5 users not shown)
Line 3: Line 3:
[[File:What is Cloud Services? poster.pdf|alt=A Poster showing some Wikimedia Colud services statistics and services|thumb|upright=1.3|Poster-format overview]]
[[File:What is Cloud Services? poster.pdf|alt=A Poster showing some Wikimedia Colud services statistics and services|thumb|upright=1.3|Poster-format overview]]


'''Wikimedia Cloud Services''' ('''WMCS''') provides tools, services, and support for technical collaborators who want to contribute to Wikimedia software projects. Use Cloud Services to host your software tools for the [[meta:Special:MyLanguage/Wikimedia movement|Wikimedia movement]], without charge.
[[Wikimedia_Cloud_Services_team|'''Wikimedia Cloud Services''' ('''WMCS''')]] provides tools, services, and support for technical collaborators who want to contribute in Wikimedia software projects. Use Cloud Services to host your software tools for the [[meta:Special:MyLanguage/Wikimedia movement|Wikimedia movement]], without charge.


== Service concepts ==
== Service concepts ==
Line 59: Line 59:
=== How is Cloud VPS organized? ===
=== How is Cloud VPS organized? ===


Cloud VPS is divided into projects. Each project has separate members and administrators who can create and maintain virtual machines ("instances") for use by that project. Each project can have own its own access policies, DNS records, etc.
Cloud VPS is divided into projects. Each project has separate members and administrators who can create and maintain virtual machines ("instances") for use by that project. Each project can have its own access policies, DNS records, etc.


=== What is a Cloud VPS project? ===
=== What is a Cloud VPS project? ===
Line 95: Line 95:
[[File:PAWS.svg|right|frameless|200px|alt=PAWS|link=PAWS]]
[[File:PAWS.svg|right|frameless|200px|alt=PAWS|link=PAWS]]


''PAWS'' is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running bots using Pywikibot, and more.  
''[[PAWS]]'' is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running bots using Pywikibot, and more.  


PAWS notebooks are maintained by a single user. They can be downloaded and forked by other users.
A single PAWS notebook is maintained by a single user, but they can be downloaded and forked by other users.


To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.
To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.
Line 105: Line 105:
{| class="wikitable"
{| class="wikitable"
|+
|+
!Activity / Needs
! rowspan="2" |Activity / Needs
!PAWS '''(DaaS)'''
!Quarry (DaaS)
!Quarry (DaaS)
!PAWS (DaaS)
!Toolforge (PaaS)
!Toolforge (PaaS)
!Cloud VPS (IaaS)
!Cloud VPS (IaaS)
|-
|-
|Browser based
|Data as a service
|
|Data as a service
|Platform as a service
|Infrastructure as a service
|-
|Write documentation and create tutorials
|✔
|✔
|
|
|
|
|
|-
|-
|Terminal based
|Browser based
|✔
|✔
|
|
|
|
|✔
|✔
|-
|-
|Write queries against replica databases
|Write queries against replica databases
Line 130: Line 135:
|-
|-
|Run database dumps
|Run database dumps
|✔
|
|
|✔
|✔
|✔
|
|
|-
|-
|Write and run bots
|Write and run bots
|✔
|
|
|✔
|✔
|✔
|
|
Line 159: Line 164:
|
|
|-
|-
|Administer your own virtual server
|Terminal based
|
|
|
|
|
|✔
|✔
|✔
|-
|-
Line 171: Line 176:
|✔
|✔
|-
|-
|Write documentation and create tutorials
|Work with co-maintainers and co-admins
|
|
|
|✔
|✔
|✔
|-
|Administer your own virtual server
|
|
|
|-
|Work with co-maintainers and co-admins
|
|
|
|
|✔
|✔
|✔
|-
|-
Line 188: Line 193:
|intermediate—advanced
|intermediate—advanced
|advanced
|advanced
|-
!Service concept
|Data as a service
|Data as a service
|Platform as a service
|Infrastructure as a service
|}
|}


Line 205: Line 204:


WMCS is a computing ecosystem built on [[w:OpenStack|OpenStack]], [[w:Oracle Grid Engine|GridEngine]], and [[w:Kubernetes|Kubernetes]].
WMCS is a computing ecosystem built on [[w:OpenStack|OpenStack]], [[w:Oracle Grid Engine|GridEngine]], and [[w:Kubernetes|Kubernetes]].
Cloud VPS projects use [[Help:Horizon FAQ|Horizon]].


== Learn more ==
== Learn more ==

Revision as of 09:10, 30 August 2022

A Poster showing some Wikimedia Colud services statistics and services
Poster-format overview

Wikimedia Cloud Services (WMCS) provides tools, services, and support for technical collaborators who want to contribute in Wikimedia software projects. Use Cloud Services to host your software tools for the Wikimedia movement, without charge.

Service concepts

Data as a service

Quarry and PAWS empower technically curious to advanced users to query wiki replicas and create scripts, tutorials, and data visualizations to analyze and improve Wikimedia projects.

See #Data Services below.
Platform as a service

Toolforge is for intermediate to advanced users working on tools, bots, webservices that support Wikimedia projects.

See #Toolforge below.
Infrastructure as a service

Cloud VPS is for advanced users who need to administer their own servers for Wikimedia operations and software development.

See #Cloud VPS below.

Are you unsure? Check #Which service is right for you? below.

Toolforge

Toolforge

Toolforge is one of the projects hosted by Wikimedia Cloud VPS. It is a shared hosting (platform as a service) environment for volunteers to develop and run tools, continuous bots, web services, scheduled jobs, and data analysis.

To use Toolforge you will need some programming knowledge, an understanding of Unix command line, and version control via Gerrit and Git.

Users of the Toolforge project create so-called "tool" accounts (technically service groups). These accounts allow one or more users to collaborate to manage the software source code, configuration, and jobs for that tool or bot.

The Toolforge administrators manage a pool of virtual servers which provide a shared project hosting environment that can be used by Toolforge users. These resources include web servers, databases and other data storage, and a distributed job processing system. These services provide a reliable and scalable hosting environment for volunteers to develop and operate their tools and bots.

For additional documentation and help with Toolforge, see Portal:Toolforge.

Cloud VPS

Cloud Services

Cloud VPS (Virtual Private Server) is a cloud computing environment powered by OpenStack. It offers collaboratively owned collections of virtual private servers. You can use this infrastructure to create and maintain open source software projects that help the Wikimedia movement.

The environment includes access to a variety of data services. Cloud VPS allows developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the Toolforge environment.

Cloud VPS is for the advanced users to get involved in Wikimedia operations and software development. Cloud VPS contains many projects, each of which uses one or more instances.

Cloud VPS instances go through a request and approval processes. Instances are not permanent and are reviewed periodically for potential deletion/removal. Cloud VPS instances are resource intensive. Before requesting, explore whether Toolforge or another service will adequately meet your needs.

How is Cloud VPS organized?

Cloud VPS is divided into projects. Each project has separate members and administrators who can create and maintain virtual machines ("instances") for use by that project. Each project can have its own access policies, DNS records, etc.

What is a Cloud VPS project?

A project is a unit of privilege separation inside the Cloud VPS environment. Each project has separate management of membership, virtual machines, HTTPS proxies, firewall rules, etc. Examples of projects include Toolforge and the Beta Cluster.

How does Cloud VPS work?

Cloud VPS is a virtualization cluster and hosts various virtual machines (called instances) using OpenStack Compute. This is slightly different from your normal servers that you ssh to (i.e. Toolserver), as virtual machines do not exist physically, but reside inside a much bigger machine called the host machine. More details about the physical setup of Cloud VPS can be found under Portal:Cloud VPS/Infrastructure.

What is the difference between Cloud VPS and Toolforge?

Cloud VPS is an Infrastructure as a service (IaaS) solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.

Toolforge is a Platform as a service (PaaS) solution. It provides web servers, databases and other data storage, and a distributed job processing system as managed services that can be used by tools and their maintainers.

Data Services

Data Services are a collection of products which provide access to copies of Wikimedia's production wiki databases (with private information redacted) and access to Wikimedia data dumps. Use data services to create replicas of the production databases and other data for analysis and experimentation.

There are also services to interact with data in a web browser: Quarry and PAWS.

Quarry

Quarry

Quarry is a public querying interface for Wiki Replicas, a set of live replica SQL databases of public Wikimedia wikis. Quarry is designed to make running queries against Wiki Replicas easy. Quarry also allows researchers to share and review each other's queries.

Quarry queries are run by individual users. They can be saved and published and forked by other users.

To use Quarry you need only a Wikimedia login and a web browsers. A basic understanding of SQL is recommended. Learn about SQL queries.

PAWS

PAWS

PAWS is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running bots using Pywikibot, and more.

A single PAWS notebook is maintained by a single user, but they can be downloaded and forked by other users.

To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.

Which service is right for you?

Activity / Needs PAWS (DaaS) Quarry (DaaS) Toolforge (PaaS) Cloud VPS (IaaS)
Data as a service Data as a service Platform as a service Infrastructure as a service
Write documentation and create tutorials
Browser based
Write queries against replica databases
Run database dumps
Write and run bots
Run web services
Build tools to improve Wikimedia projects
Schedule or run continuous jobs
Terminal based
Need your own subdomain
Work with co-maintainers and co-admins
Administer your own virtual server
User knowledge curious—advanced curious—advanced intermediate—advanced advanced

Get started

Set up your Toolforge or Cloud VPS projects by following the instructions on Help:Getting Started.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about Wikimedia Cloud Services
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Technology stack

WMCS is a computing ecosystem built on OpenStack, GridEngine, and Kubernetes.

Learn more

Historical information

From 2011 until early 2017, Wikimedia Cloud Services was known as Wikimedia Labs. However, the term Labs was used for several different things.

Since 2017, the former Wikimedia Foundation Labs team and Tool Labs Support team merged into the Wikimedia Cloud Services team.