You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

SRE Offboarding: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Filippo Giunchedi
(Update icinga/victorops instructions)
imported>Platonides
No edit summary
Line 39: Line 39:


==== Completely remove user ====
==== Completely remove user ====
By default all non-priv LDAP/Phab groups are retained!. To completely remove a user we can add
By default all non-priv LDAP/Phab groups are retained!  To completely remove a user we can add
<pre>offboard-user --drop-all -l $user</pre>
<pre>offboard-user --drop-all -l $user</pre>


==== Applying the change ====
==== Applying the change ====
Theses commands will print create LDIF file to use for removal and the instructions command you need to run e.g.
These commands will print create LDIF file to use for removal and the instructions command you need to run e.g.


<pre>  
<pre>  
Line 70: Line 70:
[[phab:people/query/advanced/|Search for the username in Phabricator]], once you have the user's Phabricator username run the following command (again you can use --list-only to check first)
[[phab:people/query/advanced/|Search for the username in Phabricator]], once you have the user's Phabricator username run the following command (again you can use --list-only to check first)
<pre>offboard-user -p $user</pre>
<pre>offboard-user -p $user</pre>
if the user is a member of any privileged groups e.g. [[phab:tag/security/|Security]], [[phab:tag/wmf-nda/|WMF-NDA]] you will be promoted to remove them, do so.
if the user is a member of any privileged groups e.g. [[phab:tag/security/|Security]], [[phab:tag/wmf-nda/|WMF-NDA]] you will be prompted to remove them, do so.


If the Phab account is connected only to an WMF SUL/OAuth account on mediawiki.org (see the user profile on Phabricator), then the Phab user account shall get disabled, to make it clear to anyone that the person is not active anymore and to allow searching for tasks still assigned to the person not active anymore. The user cannot log in anyway anymore to Phab in that case, as their SUL/OAuth account on mediawiki.org is also disabled.
If the Phab account is connected only to an WMF SUL/OAuth account on mediawiki.org (see the user profile on Phabricator), then the Phab user account shall get disabled, to make it clear to anyone that the person is not active anymore and to allow searching for tasks still assigned to the person not active anymore. The user cannot log in anyway anymore to Phab in that case, as their SUL/OAuth account on mediawiki.org is also disabled.
Line 100: Line 100:


=== Validate shell access to the prod cluster ===
=== Validate shell access to the prod cluster ===
to remove a shell user we will need to prepare a patch against `modules/admin/data/data.yaml` to preform the following tasks
to remove a shell user we will need to prepare a patch against `modules/admin/data/data.yaml` to perform the following tasks
* add the user to the groups['absent'] at he very top of the file
* add the user to the groups['absent'] at the very top of the file
* remove the user from all other groups
* remove the user from all other groups
* update the users entry to:
* update the users entry to:
Line 110: Line 110:
Look at a [https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/484231/ real patch in phabricator] to see what this looks like.
Look at a [https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/484231/ real patch in phabricator] to see what this looks like.


Once this change has been accepted into puppet the user will be removed from all machines in ~30 minutes.  It is possible that a user may still have a process running e.g. screen, tmux.  in theses cases puppet will fail as userdel is unable to remove the user and puppet.  We will get an alert for this so monitor #wikimedia-operations for any alerts and intervene manually. if it is obvious the running process is not doing anything useful then just kill it otherwise double check with some one from the users team.
Once this change has been accepted into puppet the user will be removed from all machines in ~30 minutes.  It is possible that a user may still have a process running e.g. screen, tmux.  in these cases puppet will fail as userdel is unable to remove the user and puppet.  We will get an alert for this so monitor #wikimedia-operations for any alerts and intervene manually. If it is obvious the running process is not doing anything useful then just kill it otherwise double check with some one from the users team.


== SRE Staff ==
== People with higher privileges ==
For SRE staff there are additional steps we need to preform.  Some of theses changes require you to edit the private git repo.  When doing so ensure you have read the README file in /srv/private.  Specifically  
Some of these changes require you to edit the private git repo.  When doing so ensure you have read the README file in /srv/private.  Specifically  
* DO NOT EVER rewrite history in this repo. No git commit --amend, no git rebase
* DO NOT EVER rewrite history in this repo. No git commit --amend, no git rebase
* all changes and the git commit need to be preformed as root or with sudo
* all changes and the git commit need to be performed as root or with sudo


=== Review access to internal IRC channels ===
=== Review access to internal IRC channels ===
Line 132: Line 132:
</pre>
</pre>


more information can be found on the [https://wikitech.wikimedia.org/wiki/Mailman#Remove_an_individual_from_all_mailing_lists Mailman page]
more information can be found on the [[Mailman#Remove_an_individual_from_all_mailing_lists|Mailman page]]


=== Remove from Exim aliases from private.git ===
=== Remove from Exim aliases from private.git ===
Line 140: Line 140:


New users will probably only have one entry for root@wikimedia.org, however users who have been around for some time may appear in multiple locations
New users will probably only have one entry for root@wikimedia.org, however users who have been around for some time may appear in multiple locations
== SRE Staff ==
For SRE staff there are additional steps we need to perform.


=== Remove VictorOps user ===
=== Remove VictorOps user ===
Line 145: Line 148:


=== Remove Icinga user ===
=== Remove Icinga user ===
Create a patch to the puppet repository to remove all instance of the user [https://github.com/wikimedia/puppet/blob/production/modules/icinga/files/cgi.cfg modules/icinga/files/cgi.cfg].  Note the user name show hear is the  LDAP (Wikitech wiki) username which is not necessarily the same as the shell account.
Create a patch to the puppet repository to remove all instance of the user [https://github.com/wikimedia/puppet/blob/production/modules/icinga/files/cgi.cfg modules/icinga/files/cgi.cfg].  Note the user name shown here is the  LDAP (Wikitech wiki) username, which is not necessarily the same as the shell account.


=== Remove from pwstore ===
=== Remove from pwstore ===


For this you will need to contact Moritz or Daniel and ask them to preform this task.  Check the [https://office.wikimedia.org/wiki/Pwstore#User_database Pwstore] page for more information
For this you will need to contact Moritz or Daniel and ask them to perform this task.  Check the [https://office.wikimedia.org/wiki/Pwstore#User_database Pwstore] page for more information


=== Remove from Google group for private groups ===
=== Remove from Google group for private groups ===

Revision as of 15:47, 27 August 2021

Phabricator ticket

[] update LDAP permissions based on NDA status
[] update Phabricator permissions based on NDA status 
[] Check HBase/Haddoop permissions and inform the SRE analytics team
[] update user in [[https://github.com/wikimedia/puppet/blob/production/modules/admin/data/data.yaml | modules/admin/data/data.yaml]]
Additional task for SRE team members
[] Review access to internal IRC channels
[] Remove from ops mailing lists (ops and ops-private)
[] Remove from private Exim aliases
[] Remove VictorOps user
[] Remove Icinga user
[] Remove from pwstore
[] Remove from Technical Operations group (if part of Core SRE)
[] Review access to network devices (and potentially remove access)

All Users

The following sections needed to be completed for all users regardless of team

Volunteer status

Email the user and ask if they wish to remain on as a volunteer, is so ensure they have signed an NDA

Check Users LDAP access

When Off Boarding users in LDAP you have a number of options

  • remove the user from privileged groups leave them in all un-privileged groups
  • convert the user into a volunteer (Requires Signed NDA)
  • completely remove the user

The following commands can be run from mwmaint1002.eqiad.wmnet

Remove user from privileged groups

This is the default action if a user does not want to become a Volunteer

offboard-user -l $user

convert user to a Volunteer account

First ensure the user has signed the NDA, if so run

offboard-user --turn-volunteer -l $user

Completely remove user

By default all non-priv LDAP/Phab groups are retained! To completely remove a user we can add

offboard-user --drop-all -l $user

Applying the change

These commands will print create LDIF file to use for removal and the instructions command you need to run e.g.

 
 cat jbond.ldif
 dn: cn=ops,ou=groups,dc=wikimedia,dc=org 
 changetype: modify 
 delete: member 
 member: uid=jbond,ou=people,dc=wikimedia,dc=org 

 dn: cn=wmf,ou=groups,dc=wikimedia,dc=org 
 changetype: modify 
 delete: member 

 member: uid=jbond,ou=people,dc=wikimedia,dc=org
 ldapmodify -h ldap-labs.eqiad.wikimedia.org -p 389 -x -D "cn=scriptuser,ou=profile,dc=wikimedia,dc=org" -W -f jbond.ldif

if you just wish to check a users permission you can run the following command , we will use this for checking haddop permissions later

offboard-user --list-only -l $user

Check Users Phabricator access

If the user is remaining with Wikimedia as a volunteer and their Phabricator account is linked to an LDAP account or their Phabricator account is linked to a mediawiki.org account which does not end in "(WMF)", then they this step can be skipped. Check their profile page in Phabricator to see what their Phab account is linked to.

Search for the username in Phabricator, once you have the user's Phabricator username run the following command (again you can use --list-only to check first)

offboard-user -p $user

if the user is a member of any privileged groups e.g. Security, WMF-NDA you will be prompted to remove them, do so.

If the Phab account is connected only to an WMF SUL/OAuth account on mediawiki.org (see the user profile on Phabricator), then the Phab user account shall get disabled, to make it clear to anyone that the person is not active anymore and to allow searching for tasks still assigned to the person not active anymore. The user cannot log in anyway anymore to Phab in that case, as their SUL/OAuth account on mediawiki.org is also disabled.

Handle Hadoop leftovers

We first run the following script to check if the user has any haddop permissions

offboard-user --list-only -l $user

You need to make a list of all entries that mention hadoop in the following example we need to make a list of the groups statistics-privatedata-users and analytics-privatedata-users

User DN: uid=jbond,ou=people,dc=wikimedia,dc=org
Is member of the following unprivileged LDAP groups:
  cn=project-bastion,ou=groups,dc=wikimedia,dc=org (can be retained)
  cn=project-deployment-prep,ou=groups,dc=wikimedia,dc=org (can be retained)
  cn=project-tools,ou=groups,dc=wikimedia,dc=org (can be retained)
  cn=tools.perflogbot,ou=servicegroups,dc=wikimedia,dc=org (can be retained)
Is not a project admin in Nova
Is not a member in any privileged group
statistics-privatedata-users grants access to Hadoop/Hive, check PII leftovers
analytics-privatedata-users grants access to Hadoop/Hive, check PII leftovers

Once you have this information create a new fabricator task and tag it with Analytics and link it to your own Off boarding ticket. you can use the following for the ticket text

$Full_Name ($username) is currently being off-boarded and please check if they left stuff in their home dirs on stat*/notebook*/HDFS since they where part of:

statistics-privatedata-users
analytics-privatedata-users

Validate shell access to the prod cluster

to remove a shell user we will need to prepare a patch against `modules/admin/data/data.yaml` to perform the following tasks

  • add the user to the groups['absent'] at the very top of the file
  • remove the user from all other groups
  • update the users entry to:
 * remove the email entry
 * blank out the ssh keys i.e. set it to `ssh-keys: []`
 * mark them as absent

Look at a real patch in phabricator to see what this looks like.

Once this change has been accepted into puppet the user will be removed from all machines in ~30 minutes. It is possible that a user may still have a process running e.g. screen, tmux. in these cases puppet will fail as userdel is unable to remove the user and puppet. We will get an alert for this so monitor #wikimedia-operations for any alerts and intervene manually. If it is obvious the running process is not doing anything useful then just kill it otherwise double check with some one from the users team.

People with higher privileges

Some of these changes require you to edit the private git repo. When doing so ensure you have read the README file in /srv/private. Specifically

  • DO NOT EVER rewrite history in this repo. No git commit --amend, no git rebase
  • all changes and the git commit need to be performed as root or with sudo

Review access to internal IRC channels

We need to ensure Users are removed from private IRC chat channels. If you are not aware of the private channels or there associated Op please contact a member of the SRE security team

Remove from ops mailing lists (ops and ops-private)

First you can check which lists a user is on with the following command on the mailman server (lists1001.wikimedia.org at time of writing)

sudo /var/lib/mailman/bin/find_member user@example.org

To remove a user from a our mailing lists run the following commands

sudo /var/lib/mailman/bin/remove_members --fromall user@example.org

more information can be found on the Mailman page

Remove from Exim aliases from private.git

Remove the user from all alias files under the following dir

/srv/private/modules/privateexim/files/

New users will probably only have one entry for root@wikimedia.org, however users who have been around for some time may appear in multiple locations

SRE Staff

For SRE staff there are additional steps we need to perform.

Remove VictorOps user

Make sure the user has been deleted from https://portal.victorops.com/dash/wikimedia#/users

Remove Icinga user

Create a patch to the puppet repository to remove all instance of the user modules/icinga/files/cgi.cfg. Note the user name shown here is the LDAP (Wikitech wiki) username, which is not necessarily the same as the shell account.

Remove from pwstore

For this you will need to contact Moritz or Daniel and ask them to perform this task. Check the Pwstore page for more information

Remove from Google group for private groups

Email Mark, Faidon or Moritz and ask them to remove the user from the sre@wikimedia.org Google group

Review access to network devices (and potentially remove access)

Please review whether the user had access to network devices at https://github.com/wikimedia/operations-homer-public/blob/master/config/common.yaml

If so, open a Phabricator task asking for network access to be removed and tag it with "netops"