You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Production shell access

From Wikitech-static
Revision as of 01:03, 15 July 2016 by imported>Neil P. Quinn-WMF (Copyedit intro material.)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Shells!

This page deals with shell access to the production Wikimedia cluster. For shell access to Labs, see Help:Access.

The first and last rule to keep in mind here is that production access is extremely sensitive. Treat it that way by being security conscious, including by following these important rules:

  • Use your production SSH key for production access only. Do not reuse it anywhere else, including Labs.
  • Encrypt and maintain physical control over any machine which has your production SSH key on it.

If you have questions about security or accidentally break these rules, contact Tech Ops.

SSH configuration

Security

Ideally one restricts and does NOT forward a key into shared systems, so we don't allow agent forwarding. This helps to prevent SSH agent hijacking. Ensure you don't have any alias to automatically add -A to your ssh commands.

You should also add the following to your SSH config to ensure that you're not vulnerable to a serious SSH bug that was disclosed in January 2016:

Host *
    UseRoaming no

Sample SSH config

Host bast1001.wikimedia.org
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org *.wmnet !gerrit.wikimedia.org
    User your_username_here
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org
    IdentityFile ~/.ssh/your_production_ssh_key

In the example above you may replace bast1001.wikimedia.org (in the Eqiad cluster in Virginia, United States) with another bastion that's physically closer to you:

  • bast2001.wikimedia.org in the Codfw cluster in Texas, United States
  • bast3001.wikimedia.org in the Esams cluster in Amsterdam, The Netherlands
  • bast4001.wikimedia.org in the Ulsfo cluster in San Francisco, United States

Map of bastion hosts

Production ops

Ops go through the restricted bastion iron.wikimedia.org (eqiad) instead.

## Production & External Zones

Host iron.wikimedia.org bast1001.wikimedia.org bast2001.wikimedia.org bast3001.wikimedia.org bast4001.wikimedia.org
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org
    User your_username_here
    IdentityFile ~/.ssh/your_production_ssh_key
    ProxyCommand ssh -a -W %h:%p iron.wikimedia.org

## Internal Zones

Host *.wmnet
    User your_username_here
    IdentityFile ~/.ssh/your_production_ssh_key

Host *.eqiad.wmnet
    ProxyCommand ssh -a -W %h:%p iron.wikimedia.org

Host *.codfw.wmnet
    ProxyCommand ssh -a -W %h:%p bast2001.wikimedia.org

Host *.esams.wmnet
    ProxyCommand ssh -a -W %h:%p bast3001.wikimedia.org

Host *.ulsfo.wmnet
    ProxyCommand ssh -a -W %h:%p bast4001.wikimedia.org

## Networking Equipment

Host *-eqiad.mgmt.wikimedia.org
    ProxyCommand ssh -a -W %h:%p iron.wikimedia.org

Host *-codfw.mgmt.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast2001.wikimedia.org

Host *-esams.mgmt.wikimedia.org *-knams.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast3001.wikimedia.org

Host *-ulsfo.mgmt.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast4001.wikimedia.org

Fundraising infrastructure

Host tellurium.wikimedia.org
    User your_username_here
    IdentityFile ~/.ssh/your_frack_ssh_key
    ProxyCommand none

Host *.frack.* payments100* indium pay-lvs* silicon boron samarium db1025 db1008 barium thulium lutetium backup4001
    User your_username_here
    IdentityFile ~/.ssh/your_frack_ssh_key
    ProxyCommand ssh -a -W %h:%p tellurium.wikimedia.org

Other tips

Requesting access

Production shell users, their keys, and their permissions are managed in the modules/admin/data/data.yaml in the operations-puppet repository. To get yourself added to this file, follow the instructions below.

New users

Requesting shell access to the Wikimedia server cluster is granted strictly on an as needed basis, and is entirely under the purview of the Engineering Department and Operations Department managers. They can approve or deny access for any reason they deem, as security is of the highest priority.

To acquire shell access, you should be working on a project that requires this access on a regular and ongoing basis. Any one time requests should simply request what data is required, we will not grant access for one time requests.

  1. You will need a Phabricator account to complete this process.
  2. Read, comprehend, and sign the Acknowledgement of Wikimedia Server Access Responsibilities document
  3. If you are not a Wikimedia Foundation employee or contractor, read, comprehend, and sign the Volunteer NDA.
  4. Create a ticket for access.
    • Title should reflect the format: "Requesting access to $EXISTINGGROUP/RESOURCE for $USER[S]"
      • If it is a new user request, it should be one ticket per user!
    • Set the security dropdown to 'Access Request'. This will associate the Ops-Access-Requests project so the operations team will see your request. This will cause a second hidden subtask to be created. This is because your access request ticket may require a security review and have concerns that need to be addressed. This discussion involves the technical particulars of sensitive and restricted systems.
    • Include the following information:
      • Your full name
      • Your labs username/wikitech username (a link to your profile is welcome).
      • We base production UID from labs UID, so you have to sign up on labs/wikitech before you request access to the normal cluster.
      • Your preferred shell user name.
      • Your public RSA/DSA key. This must meet a few criteria:
        • Key must not be the same as the one used to access Labs, production access is considered more critical and the keys must be kept distinct.
        • Key must be uploaded via a non-email means, the following suggestions suffice:
          • Put a copy of your public key on your wiki user page.
          • Paste your public key into a phabricator task directly or onto a file/paste via web (but not via email!)
          • Upload your own patchset to gerrit which includes your public key.
      • The project being worked on with a full and detailed reason for access and what will be done with said access. Please include as much information as possible, as it will detail what servers you need access to, and we can ensure that we get all the proper permissions correct. There are varying levels of access via shell, and we will err on the side of 'less is more.' It is advised that you are as detailed as possible, or you may find yourself lacking the access you require. If there are existing users who have similar access it is useful to document the group from puppet.
      • Approval from your direct supervisor (this is nearly always a paid employee of the Foundation technical staff)
        • This approval should be via web reply to the phabricator task, which means they will be required to have a Phabricator account. (this prevents socially engineered or email spoofed approvals.)
      • Approval from project lead where your access will be granted.
        • This approval should also be via web reply to the phabricator task, which means they will be required to have a Phabricator account. (this prevents socially engineered or email spoofed approvals.)
  5. For most requests, a three business day waiting period must be observed after the request is filed in the Ops-Access-Requests project.
  6. If your access request includes any level of sudo privileges on a system, your request will have a 'mandatory' security review in the weekly operations meetings. Sudo access is granted on an extremely limited basis, and will typically apply to the smallest permissions possible (user/process restricted over all). Expect this process to take at least one business week.

If you feel an unreasonable amount of time has passed, you can comment on the ticket to request update and/or request an update directly from the Operations team member on Ops_Clinic_Duty that week.

Additional permissions for existing users

To escalate shell access, you should be working on a project that requires this access on a regular and ongoing basis. Any one time requests should simply request what data is required, access is not granted for one time requests.

  1. As the Phabricator document to sign/acknowledge is new (as of January 2015), any escalations of existing users should include said user reviewing and acknowledging the Acknowledgement of Wikimedia Server Access Responsibilities document.
  2. Create a ticket for access
    • Title should use the format: "Requesting access to $EXISTINGGROUP/RESOURCE for $USER[S]"
    • Group tickets are acceptable when a group is being escalated.
    • Set the security dropdown to 'Access Request'. This will associate the Ops-Access-Requests project so the operations team will see your request. This will cause a second hidden subtask to be created. This is because your access request ticket may require a security review and have concerns that need to be addressed. This discussion involves the technical particulars of sensitive and restricted systems.
  3. Please include your full name and the your username.
    • Your manager's approval is usually not required, as you've already been granted access to the cluster; the project lead of the cluster you request access to should sign off (if in doubt, ask the Ops_Clinic_Duty person for the week.)
  4. Include the project being worked on with a full and detailed reason for access and what will be done with said access. Please include as much information as possible, as it will detail what servers you need access to, and we can ensure that we get all the proper permissions correct. There are varying levels of access via shell, and we will err on the side of 'less is more.' It is advised that you are as detailed as possible, or you may find yourself lacking the access you require. If there are existing users who have similar access it is useful to document the group from puppet.
  5. A three business day waiting period must be observed after the request is filed in the Ops-Access-Requests project.
    • This may not be required when the change is correcting a previous request, but should be followed for escalations that include not previously approved permissions. It may not be required in some other circumstances.
  6. If your access request includes any level of sudo privileges on a system, your request will have a 'mandatory' security review in the weekly operations meetings. Sudo access is granted on an extremely limited basis, and will typically apply to the smallest permissions possible (user/process restricted over all). Expect this process to take at least one business week.