View source for User:4nn1l2/Help:Toolforge/Summary

{{outdated}}
{{Tool Labs nav}}

'''Tool Labs''' is a hosting environment for community developers working on tools and bots that help users maintain and use wikis. Tool Labs provides access to replicas of Wikimedia databases, allowing developers to easily re-use this information, for analytics, [[m:Bot|bot]] work, or by creating tools that help editors and other volunteers in their work. The infrastructure is supported by a dedicated group of Wikimedia Foundation staff and volunteers.
== Quick start ==
# On wikitech, visit [[Special:UserLogin/signup|Create an account]] and create your Labs wiki account.
#* make careful note of the ''wiki username'' and "Instance ''shell'' account name" you choose
# On wikitech, [https://toolsadmin.wikimedia.org/tools/membership/apply Fill out an access request for the Tools project].
# In a command-line terminal, generate an SSH-2 RSA key. See [[Help:Tool Labs/Access|Generating and uploading an SSH key]] if you don't know how.
# In a command-line terminal, enter: <kbd>$ cat ~/.ssh/id_rsa.pub</kbd> (or similar) to display your public SSH key that you created above, then copy it.
# On wikitech, log in with your labs ''wiki'' account, visit [[Special:Preferences#mw-prefsection-openstack|Preferences > OpenStack tab]] and paste in your public SSH key.
# Wait for your requests to be completed (you should receive messages on your [[Special:MyTalk | wikitech talk page]]).

=== Once this is all done you should be able to ===
* Use SSH to login to Tool Labs. In a command-line terminal, enter: <kbd>ssh nn1l2@login.tools.wmflabs.org</kbd>
* Use SSH-based utilities such as scp and sftp to transfer files between Tool Labs and your computer.
* Create tools (see {{section link||Creating a new Tool account}}).

=== Gotchas ===
* Your wikitech wiki username and your shell login username may be different. Visit [[Special:Preferences#mw-prefsection-personal|Preferences > User profile]] and check "Instance shell account name".
* The passwords you chose for your wikitech login and SSH key may be different.
* When you login with SSH you are in your personal folder. To quickly go to your tool account enter: <kbd>become ''tool_name''</kbd>
* You will also notice that web service for your tool is not started by default. To start it enter: <kbd>webservice start</kbd>

== What is Tool Labs ==

=== Rationale ===

Tool Labs was developed in response to the need to support external tools and their developers and maintainers. The system is designed to make it easy for maintainers to share responsibility for their tools and bots, which helps ensure that no useful tool gets ‘orphaned’ when one person needs a break. The system is designed to be reliable, scalable and simple to use, so that developers can hit the ground and start coding.

=== Features ===
==== Shared storage ====

[[Help:Shared storage]]

=== Architecture and terminology ===
===== Bastion hosts =====
; <tt>tools-login.wmflabs.org =</tt> <tt>login.tools.wmflabs.org</tt>

===== The grid =====
The Tool Labs grid, implemented with Open Grid Engine (the open-source fork of Sun Grid Engine) permits users to submit jobs from either a log-in account on the bastion host or from a web service. Submitted jobs are added to a work queue, and the system finds a host to execute them. Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going. For more information about the grid, please see {{section link||Submitting, managing and scheduling jobs on the grid}}.

== Getting access to Tool Labs ==
{{Help:Tool Labs/Access}}

== Using Tool Labs and managing your files ==
Tool Labs can be accessed in a variety of ways – from its public IP to a GUI client. Please see [[Help:Access]] for general information about accessing Labs.

=== The tools list ===
The [https://tools.wmflabs.org?list Tool labs tools list] page is publicly available and contains a list of all currently-hosted Tool accounts along with their maintainers. Tool accounts that have an associated web page appear as links. Users with access to the 'tools' project can create new tool accounts here, and add or remove maintainers to and from existing tool accounts.

=== SSH ===
Once set up, you <kbd>ssh</kbd> to Tool Labs via its bastion host login.tools.wmflabs.org, provided that a public SSH key has been uploaded to the Labs account.

<syntaxhighlight lang="bash">
ssh nn1l2@login.tools.wmflabs.org
</syntaxhighlight>

if you get disconnected frequently during ssh, consider setting the <code>ServerAliveInterval</code> option to a smaller number (~5-20 seconds) when connecting:

<syntaxhighlight lang="bash">
ssh -o ServerAliveInterval=5 yourshellaccountname@login.tools.wmflabs.org
</syntaxhighlight>

=== Updating files ===
After you can ssh successfully, you can transfer files via <kbd>sftp</kbd> and <kbd>scp</kbd>. Note that the transferred files will be owned by you. You will likely wish to transfer ownership to your tool account. To do this:

1. <code>become</code> your tool account:

<pre>
yourshellaccountname@tools-login:~$ become toolaccount
tools.toolaccount@tools-login:~$
</pre>

2. As your tool account, <code>take</code> ownership of the files:

<pre>
tools.toolaccount@tools-login:~$ take FILE
</pre>

The <code>take</code> command will change the ownership of the file(s) and directories recursively to the calling user (in this case, the tool account).

==== Handling permissions ====
if you're getting permission errors, note that you can also transfer files the other way around: copy the files as your tool account to <tt>/data/projects/<projectname></tt>.

Another, probably easier, way is to set the permission to group-writable for the tools directory. For example, if your shell account's name is <code>alice</code> and your tool name is <code>alicetools</code> you could do something like this after logged in as a shell user

<syntaxhighlight lang="bash">
become alicetools
chmod -R g+w /data/project/alicetools
logout
cp -rv /home/alice/* /data/project/alicetools/
</syntaxhighlight>
===== One-time steps per tool =====
First, you have to do some preparatory steps which you need only once per tool.
 become <YOURTOOL>
If you have not installed composer yet:
 mkdir ~/bin
 curl -sS https://getcomposer.org/installer | php -- --install-dir=$HOME/bin --filename=composer
If your local <code>bin</code> directory it not in your <code>$PATH</code> (use <code>echo $PATH</code> to find out), then create or alter the file <code>~/.profile</code> and add the lines:
 # set PATH so it includes user's private bin if it exists
 if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
 fi
Finish your session as &lt;YOURTOOL&gt; and start a new one, or:
 . ~/.profile
Now you are done with the one-time preparations.

===== For each instance of core =====
The following steps are needed for each new installation of MediaWiki. We assume that you want to access MediaWiki via the web in a directory named <code>MW</code> — you are free to use another name. If not already done:
 become <YOURTOOL>
Then:
 cd ~/public_html
If you plan to submit changes:
 git clone ssh://<YOURUSERNAME>@gerrit.wikimedia.org:29418/mediawiki/core.git MW
or else, if you only want to use MediaWiki without submitting changes:
 git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git MW
will do and spares resources. Next, recent versions of MediaWiki have external dependencies, so you need to install those:
 cd MW
 composer install
 git review -s
Now you should be able to access the initial pre-install screen of MediaWiki from your web browser as:
 https://tools.wmflabs.org/<YOURTOOL>/MW/
and proceed as usual. See [[/Database#Creating new databases|how to create new databases]] for your MediaWiki installations.

== Joining and creating a Tool account ==

=== What is a Tool account? ===
A Tool account is the "user" associated with a Tool on Tool labs.
Although each tool account has a user ID, they are not personal accounts (like a Labs account), rather services that consist of a user and group ID (i.e., a unix uid-gid pair) that are intended to run the actual tool or bot.
Anyone who has access to Tool Labs can create a Tool account.

* Unix user: tools.''toolname''
* Unix group: tools.''toolname''

Members of the Tool account's Unix group include:

* the tool account creator
* the tool account itself
* (optionally, but encouraged!) additional tool maintainers

Maintainers may have more than one tool account, and tool accounts may have more than one maintainer. Every member of the group has the authorization to sudo to the tool account. By default, only members of the group have access to tool account's code and data.

A simple way for maintainers to switch to the tool account is with <code>become</code>:
<syntaxhighlight lang="bash">
maintainer@tools-login:~$ become toolname
tools.toolname@tools-login:~$
</syntaxhighlight>

In addition to the user/group pair, each tool account includes:

* A home directory on shared storage: <code>/data/project/toolname</code>
* A <code>~/public_html/</code> directory, which is visible at <code><nowiki>http://tools.wmflabs.org/toolname/</nowiki></code>
* Database access credentials: <code>~/replica.my.cnf</code>, which provide access to the production database replicas as well as to project-local databases.
* Access to the continuous and task queues of the compute grid

=== Joining an existing Tool account ===
All tool accounts hosted in Tool Labs are listed on the [https://tools.wmflabs.org?list Tools list]. If you would like to be added to an existing account, you must contact the maintainer(s) directly.

If you would like to add (or remove) maintainers to a tool account that you manage, you may do so with the 'add' link found beneath the tool name on the Tools home page.

=== Creating a new Tool account ===
Members of the ‘tools’ project can create tool accounts from the Tools home page:

# Navigate to the [[toollabs:|Tools home page]].
# Select the "create new tool" link (found in the "Develop your own tool" section).
# Enter a "Service group name". The service group name will be used as the name of your tool account.

== Customizing a Tool account ==
Once you have created a tool account, there are a few things that you can customize to make the tool more easily understood and used by other users. These include:

* adding a tool account description (the description will appear on the Tools home page beside the tool name)
* creating a home page for your tool (if you create a home page for the tool, it will be linked from the Tools home page automatically)

=== Creating a tool web page ===
To create a web page for your tool account, simply place an index.html file in the tool account's <tt>~/public_html</tt> The page can be a simple description of the tool or bot with basic information on how to set it up or shut it down, or it contain an interface for the web service. To see examples of existing tool web pages, click any of the linked tool names on the [https://tools.wmflabs.org?list Tools list].

You will also need to start a webservice for your tool.

1. Log into your Labs account and become your tool account:

<syntaxhighlight lang="bash">nn1l2@tools-bastion-03:~$ become nn1l2bot</syntaxhighlight>

2. Start the web service:

<syntaxhighlight lang="bash">tools.nn1l2bot@tools-bastion-03:~$ webservice start</syntaxhighlight>

=== {{anchor|create-description}} Creating a tool description ===
To create a tool description:

1. Log into your Labs account and become your tool account:

<syntaxhighlight lang="bash">
nn1l2@tools-bastion-03:~$ become nn1l2bot
</syntaxhighlight>

2. Create a <code>.description</code> file in the tool account’s home directory. Note that this file must be HTML:

<syntaxhighlight lang="bash">
tools.nn1l2bot@tools-bastion-03:~$ vim .description
</syntaxhighlight>

3. Add a brief description (no more than 25 words or so) and save the file. You can use basic HTML markup in the file.

4. Navigate to the [https://tools.wmflabs.org?list Tools list]. Your tool account description should now appear beside your tool account name.

=== Configuring bots and tools ===
Tools and bot code should be stored in your tools account, where it can be managed by multiple users and accessed by all execution hosts. Specific information about configuring [[#Web server|web services]] and [[Help:Tool Labs/Developing|bots]], along with information about licensing, package installation, and shared code storage, is available at the {{section link||Developing on Tool Labs}} section.

== Submitting, managing and scheduling jobs on the grid ==

{{Help:Tool Labs/Grid}}

== Email ==
=== Mail to users ===

Mail sent to <code>''user''@tools.wmflabs.org</code> (where ''user'' is a shell account) will be forwarded to the email address that user has set in their Wikitech preferences, if it has been verified (the same as the 'Email this user' function on wikitech).

Any existing .forward in the user's home will be ignored.

=== Mail to tools ===

Mail can also be sent "to a tool" with:

<code>''toolname''.''anything''@tools.wmflabs.org</code>

Where "''anything''" is an arbitrary alphanumeric string. Mail will be forwarded to the first of:

* The email(s) listed in the tool's <code>~/.forward.''anything''</code>, if present;
* The email(s) listed in the tool's <code>~/.forward</code>, if present; or
* The wikitech email of the tool's individual maintainers.

Additionally, <code>tools.''toolname''@tools.wmflabs.org</code> is an alias pointing to <code>''toolname''.maintainers@tools.wmflabs.org</code> mostly useful for automated email generating from ''within'' Labs.

<code>~/.forward</code> and <code>~/.forward.''anything''</code> need to be readable by the user <code>Debian-exim</code>; to achieve that, you probably need to <code>chmod o+r ~/.forward*</code>

== Web server ==
{{Help:Tool Labs/Web}}

== Dumps ==
The 'tools' project, like all labs projects, has access to a directory storing the [http://dumps.wikimedia.org/ public Wikimedia datasets] (i.e. the dumps generated by Wikimedia). The most recent two dumps can be found in:

 /public/dumps/public

This directory is read-only, but you can copy files to your tool's home directory and manipulate them in whatever way you like.

If you need access to older dumps, you must manually download them from the [http://dumps.wikimedia.org/ Wikimedia downloads] server.

<code>/public/dumps/pagecounts-raw</code> contains some years of the [http://dumps.wikimedia.org/other/pagecounts-ez/ pagecount/projectcount data derived by Erik Zachte from Domas Mituzas' archives].

== CatGraph (aka Graphserv/Graphcore) ==
CatGraph is a custom graph database that provides tool developers fast access to the Wikipedia category structure. For more information, please see the [[Nova Resource:Catgraph|documentation]].

<nowiki>[[Category:Toolforge|summary]]</nowiki>