You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Portal:Toolforge/About Toolforge: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Diegodlh
(→‎The web cluster: Added link to Lighttpd documentation)
imported>Triciaburmeister
(→‎What tools are hosted on Toolforge?: Remove broken link and add Toolhub)
Line 28: Line 28:


* https://admin.toolforge.org/tools
* https://admin.toolforge.org/tools
* [https://toolhub.wikimedia.org/ Toolhub]
* [https://hay.toolforge.org/directory/ Hay's Tool Directory]
* [https://hay.toolforge.org/directory/ Hay's Tool Directory]
* [https://toolforge-gallery.toolforge.org/ Toolforge Gallery]


== What are the main features of Toolforge? ==
== What are the main features of Toolforge? ==

Revision as of 19:12, 25 July 2022

Overview

This page will help you understand what Toolforge is, some of its features, and how it is structured.

If you already know these basics, then you are ready to start developing tools! Visit the Toolforge Portal for more information.

Before you start

Before you can start using Toolforge to create and maintain tools, you'll need:

What is Toolforge?

<section begin=whatis /> Toolforge is a hosting environment, also known as Platform as a Service. Toolforge makes it easy for you to perform analytics, administer bots, run webservices, and create tools. Tools help project editors, technical contributors, and other volunteers who work on Wikimedia projects.

Toolforge is part of the Wikimedia Cloud Services (WMCS) suite of services. It is supported by Wikimedia Foundation staff and volunteers.<section end=whatis />

What is a tool?

Tools are software applications, web applications, gadgets, and bots that help people working on Wikimedia projects. Tools can do all kinds of tasks like helping editors discover quick tasks to do, making automatic edits, visualizing data, extracting metadata, and more.

What tools are hosted on Toolforge?

For a list of tools hosted on Toolforge, visit:

What are the main features of Toolforge?

  • A supported hosting environment for tools, web services, continuous bots, and scheduled tasks
  • Access to replicated production databases
  • Shared management of tool accounts, where tools and bots are stored
  • A grid engine for dispatching jobs
  • Support for mosh, SSH, SFTP
  • Version control via Gerrit and Git
  • Support for Redis
  • Support for Elasticsearch

How is Toolforge structured?

Toolforge is made up of four main parts: the bastion hosts, the grid, the web cluster, and the databases.

Bastion hosts

You can log in to Toolforge through a bastion host. There are two bastion hosts to choose from:

login.toolforge.org
user login to access tools interactively
dev.toolforge.org
functionally identical, please use this for heavy processing such as compiles

The grid

The grid, or Job Grid, implemented with Son of Grid Engine (an open-source fork of Sun Grid Engine) permits users to submit jobs from a log-in account on the bastion host or from a Web service.

Submitted jobs are added to a work queue, and the system finds a host to execute them. Jobs can be scheduled synchronously or asynchronously, continuously, or just once. If a continuous job fails, the grid will automatically restart the job, so that it keeps going.

The web cluster

The Toolforge web cluster is fronted by a web proxy, which supports SSL and is open to the Internet. The proxy distributes web requests among the web servers in this cluster. Any server in this web cluster can serve any of the hosted web tools, because Toolforge uses a shared storage system.

Each tool has its own lighttpd Web server, with full configuration options. FCGI scripts are supported with configuration options, and WSGI is supported using flup.server.fcgi.

The databases

Toolforge supports two sets of databases: the wiki replicas and user-created databases, which are used by individual tools. The wiki replicas follow the same setup as production wiki databases, and the information that can be accessed from them is the same as that which normal registered users (i.e.: not +sysop or other types of advanced permissions) can access on-wiki or via the API.

Note: some data has been removed from the replicas for privacy reasons. User-created databases can be created by either a user or a tool on the wiki replica servers or on a local ‘tools’ project database.

Rules of use

As part of Wikimedia Cloud Services, Toolforge is subject to the general Cloud Services Terms of use, and is governed by these additional rules.

Individual wiki policies

Each wiki has its own guidelines and procedures. When developing on Toolforge, please follow the bot policies of the wikis your bot will interact with.

For general information and guidelines, see Bot policy.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about Wikimedia Cloud Services
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

See Also