SRE/LDAP

LDAP in Production

There are 4 separate LDAP installations:

• ldap-corp1001/ldap-corp2001 provide a mirror of the LDAP server operated by OIT. It is used to perform LDAP lookups from the mail servers.
• seaborgium/serpens ("ldap-labs") are mostly used by services that need to write to LDAP (Wikitech, Striker, so on). Records in this LDAP directory are directly related to developer accounts used to provide access to Cloud VPS projects, Gerrit, Phabricator, and related systems.
• ldap-replica* ("ldap-ro") are read-only replicas of the ldap-labs cluster with LVS load balancing. They are used for authentication in WMCS and for some production services not using CAS-SSO.
• cloudservices2XXX-dev have a small cluster that is used like the ldap-labs and ldap-ro clusters on the OpenStack testing environment.

All installations are using OpenLDAP. The two servers of each cluster are running in mirror_mode, which provides bi-directional replication of write changes using the syncrepl protocol.

Installing/Configuring the server

Create indexes

After creating the indexes, on each server you need to stop the server, rebuild the indexes, and start the server:

Add a proxy agent

Since we are requiring authentication by default, the clients need a user to do authenticated lookups. We create a proxyagent user for this:

dn: cn=proxyagent,ou=profile,<basedn>
changetype: add
objectclass: top
objectclass: inetorgperson
objectclass: person
sn: agent
givenName: proxy
userpassword: <aPasswordGoesHere>
cn: proxyagent

Installing/Configuring the client manually

Install required packages

apt-get install ldap-utils sudo-ldap libpam-ldap libnss-ldap nss-updatedb libnss-db autofs5 autofs5-ldap nscd

Install the server certificate's CA

1. Install to /etc/ssl/certs/ldapca.crt
2. Run:

pushd /etc/ssl/certs
ln -s ldapca.crt $(openssl x509 -hash -noout -in ldapca.crt).0
popd

Configure openldap's ldap.conf

Add the following options to /etc/ldap/ldap.conf:

BASE            <basedn>
URI             ldap://<servername>:389
SSL             start_tls
TLS_CHECKPEER   yes
TLS_REQCERT     demand
TLS_CACERTDIR   /etc/ssl/certs
TLS_CACERTFILE  /etc/ssl/certs/ldapca.crt
TLS_CACERT      /etc/ssl/certs/ldapca.crt
SUDOERS_BASE    ou=sudoers,<basedn>

• Note: TLS_CACERTDIR is likely ignored, since gnutls doesn't support the directive, but for future compatibility, it should be defined.
• Note: Though we define the URI as ldap/389, we should always use encryption, so all clients should use StartTLS

Configure libnss's ldap.conf

uri             ldap://<server1>:389 ldap://<server2>:389
base            <basedn>
binddn          cn=proxyagent,ou=profile,<basedn>
bindpw          <proxyagentPasswordInTheClear>
pam_filter      objectclass=posixAccount
nss_base_passwd ou=people,<basedn>
nss_base_shadow ou=people,<basedn>
nss_base_group  ou=group,<basedn>
nss_base_hosts  ou=hosts,<basedn>
nss_base_netgroup     ou=netgroup,<basedn>
tls_checkpeer   yes
tls_cacertfile   /etc/ssl/certs/ldapca.crt
tls_cacertdir   /etc/ssl/certs
ssl             start_tls
pam_password    clear

Configure nss

passwd:         files ldap
group:          files ldap
shadow:         files ldap

hosts:          files dns ldap
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       ldap
automount:      files ldap
sudoers:        files ldap

The above works for clients with DNS access. For hosts without DNS access, the following line works better for hosts:

hosts:          files ldap dns

Configure pam

common-auth:

# here are the per-package modules (the "Primary" block)
auth    [success=2 default=ignore]      pam_unix.so nullok_secure
auth    [success=1 default=ignore]      pam_ldap.so use_first_pass
# here's the fallback if no module succeeds
auth    requisite                       pam_deny.so
# limit access to specific users
auth    required                        pam_access.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth    required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
# end of pam-auth-update config

common-account:

# here are the per-package modules (the "Primary" block)
account [success=2 new_authtok_reqd=done default=ignore]        pam_unix.so
account [success=1 default=ignore]      pam_ldap.so
# here's the fallback if no module succeeds
account requisite                       pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
account required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
# end of pam-auth-update config

common-password:

# here are the per-package modules (the "Primary" block)
password        [success=2 default=ignore]      pam_unix.so obscure sha512
password        [success=1 user_unknown=ignore default=die]     pam_ldap.so try_first_pass
# here's the fallback if no module succeeds
password        requisite                       pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
password        required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
# end of pam-auth-update config

common-session:

# here are the per-package modules (the "Primary" block)
session [default=1]                     pam_permit.so
# here's the fallback if no module succeeds
session requisite                       pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
session required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
session required        pam_unix.so
session optional                        pam_ldap.so
# end of pam-auth-update config

common-session-noninteractive:

# here are the per-package modules (the "Primary" block)
session [default=1]                     pam_permit.so
# here's the fallback if no module succeeds
session requisite                       pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
session required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
session required        pam_unix.so
session optional                        pam_ldap.so
# end of pam-auth-update config

Configure autofs

Configure /etc/autofs_ldap_auth.conf

Autofs's ldap configuration needs to be set to require tls, and authentication, and to use SASL PLAIN authentication with a DN and password. The following configuration provides that:

<?xml version="1.0" ?>
<autofs_ldap_sasl_conf
        usetls="yes"
        tlsrequired="yes"
        authrequired="yes"
        authtype="PLAIN"
        user="dn:cn=proxyagent,ou=profile,<basedn>"
        secret="<proxyagentPassword>"
/>

Be sure to fill in <basedn> and <proxyagentPassword> with real values.

Configuring overrides

nsswitch.conf is configured to use files, then ldap. It is possible, but not necessary to do this per map. It is required to do this at least for auto.master. If we want to pull all maps from ldap, the only line needed in /etc/auto.master is:

+auto.master

The above line tells automount to pull its maps from whichever service is configured in nss after files. If you wish to override LDAP, you can place entries above the + line. You can override individual entries for individual maps by doing something like the following:

• In auto.master:

/data    /etc/auto.data
+auto.master

• In auto.data:

overridenentry    <server>:/export
+auto.data

The above would still pull all auto.data entries, but would override the specific entry "overrideentry".

Configure sudo

sudo is configured via the nsswitch. As long as the sudo-ldap package is installed, and nsswitch.conf has ldap listed for sudoers, it will automatically pull from ldap.

Troubleshooting

Where to find logs

The main daemon logs can be found here:

user@seaborgium:~$ sudo journalctl -u slapd.service

Other interesting logs are:

• TODO: fill me

Corrupt nscd cache

The nscd cache is a bdb database, and can (rarely) become corrupt when a system shuts down uncleanly. You can forcibly clear the cache by doing the following:

/etc/init.d/nscd stop
rm -f /var/cache/nscd/*
/etc/init.d/nscd start

Invalid nscd cache

To purge an invalid cache, you can do the following:

nscd -i <database>

Where <database> is passwd, group, or services.

Debugging sudo-ldap

Sudo will print ldap debugging information, if you add the following to /etc/ldap/ldap.conf:

SUDOERS_DEBUG 2

Debugging autofs

Autofs will print debugging information to /var/log/syslog, if you add the following to /etc/default/autofs:

LOGGING="debug"

Common LDAP administrative actions

This shows you how to do user administration on LDAP. You need access to mwmaint1002 to perform write actions, and should be either a root or a member of the group ldap-admins. We shall use the wonderful ldapvi program, not invented by us but maintained by a real upstream, to do these tasks.

Display extended info about a user

[mwmaint1002:~] $ ldapsearch -x uid=$username

Replace $username with the actual user name (uid). You will see the "cn", "sn" and "uid". Note how these can be different. This also displays the email address and SSH key used. Useful to verify if a user has a WMF email address and doesn't have the same key as in production.

Show group members / check a user is already a member of a group

[mwmaint1002:~] $ ldapsearch -x cn=$grpname

Replace $grpname with the actual group name (e.g. wmf, ops, ...).

Find a user by email address

[mwmaint1002:~] $ ldapsearch -x mail="foobar@wikimedia.org"

[mwmaint1002:~] $ ldapsearch -x mail="ffoobar*"

This works with an exact email address or wildcards as well. Useful to find a user if the UID they picked is unknown but you know they used their (WMF) email address which can be derived from first and last name.

Add a user to a group

Method 1

[mwmaint1002:~] $ sudo modify-ldap-group wmf ('wmf' is the group name, other valid groups are 'ops', 'wmf', 'nda' or 'wmde')

An editor will open, copy/paste/modify a line, write and quit.

• When adding a user to the wmf LDAP group, please also add them to the Phabricator group wmf-nda (click 'members'->'add members'), see T290605 for background on this).

Method 2

ldapvi -b ou=groups cn=$groupname

This brings up an editor with all information about the group, including its list of members. You can add a member by adding the following line to the end of the editor:

member: uid=$shellname,ou=people,dc=wikimedia,dc=org

Then save and exit your editor (<ESC>:wq in vim), you'll be asked if you want to save these LDAP changes. Press y and enter to confirm, or q and enter to cancel

Remove a user from a group

Method 1

[mwmaint1002:~] $ sudo modify-ldap-group wmf ('wmf' is the group name, other valid groups are 'ops', 'wmf', 'nda' or 'wmde')

An editor will open, delete the relevant line, write and quit.

Method 2

ldapvi -b ou=groups cn=$groupname

This brings up an editor with all information about the group, including its list of members. You can find the member you want to remove (Press /$shellname to search in vim), remove that line entirely, and then exit and save.

Finding the shell name of a user

ldapsearch -x -b ou=people,dc=wikimedia,dc=org cn=$wikitechUserName

The shell username will be listed as 'uid', which is also in the DN.

Editing info about a user

If you want to change details about a user (for a rename, or a ssh key change or password reset), you can do

ldapvi -b ou=people,dc=wikimedia,dc=org uid=$shellusername

Then you can change all the info you want to and save to persist (y to commit changes v if you want to view them first).

Tool accounts have an extra organizational unit servicegroups so to change details about a tool account you can

ldapvi -b ou=people,ou=servicegroups,dc=wikimedia,dc=org cn=tools.$toolname

and proceed the same as per user accounts.

Changing the password of a user

To change the password of a user (as root, without knowing their current password):

sudo ldappasswd -H ldap://ldap-eqiad.wikimedia.org -x -D "cn=admin,dc=wikimedia,dc=org" -W -S "uid=<USER>,ou=people,dc=wikimedia,dc=org"

You will first be asked twice for a NEW password and then once for the admin password. You can find the admin password in pwstore.

See Also

• LDAP Groups - Rights granted by various LDAP groups
• Renaming users - Renaming a Wikitech/LDAP/Gerrit user
• digitalocean.com community tutorial - raw modifications to an openldap system
• Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta - LDAP account management for WMCS toolsbeta
• Portal:Cloud_VPS/Admin/Testing_deployment#LDAP - LDAP account management for WMCS codfw1dev openstack deployment