You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Mailman: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Nemo bis
(→‎Spam scores: https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html)
imported>Slyngshede
(39 intermediate revisions by 17 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:lists.wikimedia.org}}
{{TOCright}}
{{TOCright}}


Line 6: Line 5:
* [[meta:Mailing lists]]
* [[meta:Mailing lists]]
* [[meta:Mailing lists/Administration]]
* [[meta:Mailing lists/Administration]]
* For mailman2 operations: [[Mailman/Mailman2]]
== Components ==
Mailman3 has multiple components. Each doing a different thing. This is a simplified overview. For a more visual diagram, see the [https://docs.mailman3.org/en/latest/architecture.html upstream architecture overview].
* mailman3 is the service that handles emails and talks to MTA (in our case exim4) through LMTP/SMTP
* mailman3-web or postorius is a django-based web service that handles settings, subscriptions, etc. It doesn't do anything on its own (except user management) but it talks internally to the mailman3 service through REST API.
* hyperkitty (being used in the same webservice) is responsible for archives. It handles search, etc.
* Search backend: Is xapian and the search index is in /var/lib/mailman3/web/fulltext_xapian_index
* Databases: Two databases both in m5:
** mailman3 which mailman's core service talks to. It holds memberships, lists, etc.
** mailman3web which postorius and hyperkitty talk to. It holds archive of all emails and is rather big.
** MTA: It's exim4 (see [[Mailman/MTA]] for more information)


== How To ==
== How To ==


=== Create a mailing list ===
=== Create a mailing list ===
There are 2 ways to create a mailing list:
* Go to https://lists.wikimedia.org/postorius/lists/new/ (as an administrator)
# Via the web interface at [http://lists.wikimedia.org/mailman/create http://lists.wikimedia.org/mailman/create] - a ''list's creator password'' is needed. The ''site password'' works as well.
* or ssh to lists1001.wikimedia.org and run "sudo mailman-wrapper create --owner OWNER LISTNAME"
# Via shell on the mailing lists server ([[fermium]]). As root, run <tt>newlist</tt>.
 
In both cases, it's '''not necessary to add e-mail aliases''' anywhere! These are handled automatically by mailman with the mail transfer agent configured ([[exim]]). When creating a new mailing list, '''make sure it follows the [[m:Mailing_lists/Standardization#Naming_scheme|Naming scheme]]'''.
 
==== Step-by-step procedure ====
When you create a new mailing list for someone that could potentially not be experienced with Mailman administration, there's a few things that you need to do:
 
# Set yourself to be the initial list administrator, which means that you'll get an email with the list admin password.
## This will allow you to adjust settings for the list (if you don't have the site password) and also send the list password in one email to two or more list admins.
# Set "Auto-generate initial list password" to Yes.
# If relevant, also add any extra "Initial list of supported languages". For example, if you're creating a list for Wikimedia Israel, then add Hebrew as an extra supported language.
# Enter the list creator's or site password in the box at the bottom and hit "Create List".
# Now that you've created the list, login to the admin interface using the password sent to you in an email or the site admin password. Add the list admins' email addresses in the relevant field in the admin interface.
# Also add a "terse phrase" for the list (which will show up in the main list directory at [https://lists.wikimedia.org/mailman/listinfo https://lists.wikimedia.org/mailman/listinfo]).
# Add the list to [[m:Mailing lists/Overview]], or tell the requester to.
# If the list's archives are public, [http://gmane.org/subscribe.php add it to Gmane] (this needs to be done before the first message is sent to the list otherwise the mbox needs to submitted from the lists server); group name must be under [http://dir.gmane.org/index.php?prefix=gmane.org.wikimedia gmane.org.wikimedia], or [http://dir.gmane.org/index.php?prefix=gmane.science.linguistics.wikipedia gmane.science.linguistics.wikipedia] for Wikipedia language communities' lists.
# If the list was requested to be set as '''private''', you need to ensure the archives are private and that the subscription rules are changed to "Confirm and approve".
# Inform the lists' administrators of the [https://lists.wikimedia.org/mailman/listinfo/listadmins listadmins mailing list] where all notifications and changes will be announced if they are liable to impact mailman.
# Now you can send an email to the list owners letting them know that the list has been created. Here's a template email that you can use (adjusting as appropriate):
 
<div class="mw-code" style="white-space: pre-wrap; margin-top: 1em; margin-bottom: 1em;"><nowiki>Subject: Your new mailing list: <list name>
 
Body:
 
Hi <names of list owners>,
 
Your new mailing for the <purpose of list> has been created!  I've made a few tweaks to the mailing list settings, including adding a terse phrase and making the list publicly archived and listed on the main list directory (https://lists.wikimedia.org/mailman/listinfo).
 
Your mailing list admin password is:
 
    <paste password from list created email>
 
You need this password to configure your mailing list.
 
You can configure your mailing list at the following web page:
 
    https://lists.wikimedia.org/mailman/admin/<list name>
 
The web page for users of your mailing list is:
 
    https://lists.wikimedia.org/mailman/listinfo/<list name>


Please list and describe your mailing list at the appropriate section of:
Consider if you need to notify [https://office.wikimedia.org/wiki/Trust_and_Safety Trust and Safety] when creating new mailing lists.
 
    https://meta.wikimedia.org/wiki/Mailing_lists/Overview
 
If you ever would like to add more mailing list administrators, login to the admin interface, and add their e-mail addresses to the: "The list administrator email addresses" space.  Please also make sure you give the mailing list password to them.  Every admin logs in with the same password, and there's no way to send a "password reminder" or get the password back from the system... so don't forget it and make sure they all know what it is if it's changed.
 
Please also remember to subscribe yourself to the mailing list, because you're not automatically subscribed.
 
If you have any questions, please feel free to poke me and I'd be more than happy to help. :-)</nowiki></div>


=== Disable or re-enable a mailing list ===
=== Disable or re-enable a mailing list ===
Sometimes a mailing list may be requested to be removed but not deleted from the server because the archives may still be used such as old announcement lists. Disabling and re-enabling a list can be done through the mailman interface by list administrators but usually it is easier for someone who is experienced to handle it due to the complexity of mailman configuration.
* Remove owners and add disabled-lists@lists.wikimedia.org as owner
 
** This is important, otherwise we can't find it and change/fix its settings.
A shell script exists on the lists server under <tt>/usr/local/sbin/</tt> called <tt>disable_list</tt>. The script can be used to disable and enable lists easily and consistently.
* Set all emails to be discarded (both member and non-member)
 
* Ban <code>^.*@.*</code>
# Usage: ./disable_list [-e|--enable] <listname>
* Add to description that this mailing list is disabled like [Disabled] or [Archived]
 
==== Commands ====
The shell script simply does the following to disable a list:
 
# list=LISTNAME
# echo "advertised=0" | config_list -i /dev/stdin $list
# echo "emergency=1"  | config_list -i /dev/stdin $list
# echo "member_moderation_action=2" | config_list -i /dev/stdin $list
# echo "generic_nonmember_action=2" | config_list -i /dev/stdin $list
# echo "ban_list=[^.*@.*]" | config_list -i /dev/stdin $list
# rm /var/lib/mailman/data/heldmsg-$list-*.pck
 
And to re-enable a list, it does the opposite of the above.
# list=LISTNAME
# echo "advertised=1" | config_list -i /dev/stdin $list
# echo "emergency=0"  | config_list -i /dev/stdin $list
# echo "member_moderation_action=0" | config_list -i /dev/stdin $list
# echo "generic_nonmember_action=1" | config_list -i /dev/stdin $list
# echo "ban_list=[]" | config_list -i /dev/stdin $list
 
After using this script it's nice if you login one last time on the listinfo page and remove the former admins from the "list run by" field. This makes sure they never get any spam to the list owner alias. When you login you will see a message that "emergency moderation" is enabled. This tells you the script did the job and the list is disabled.


=== Remove a mailing list ===
=== Remove a mailing list ===
Removing a mailing list should only be done if the list will not be used at all and archives don't need to be public (otherwise see above). To remove a list but keep the archives stored, run (on the lists server):
Note: unless you know what you're doing, don't actually do this. It most likely does not do what you want. Just disable the list (see above) instead.  
# rmlist ''listname''
* Go to https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/delete (as an administrator)
To also remove all archives, use:
* or ssh to lists1001.wikimedia.org and run "sudo mailman-wrapper remove LISTNAME"
# rmlist -a ''listname''


=== Remove a message from the mailing list archives ===
=== Remove a message from the mailing list archives ===
Sometimes you are forced to [[remove a message from mailing list archive]] by court order or other 'command' (such as request from legal). The linked instructions tell why it is a procedure best avoided but also how to do it if legally obligated.
;Delete the whole thread
* Go to the thread in hyperkitty ([https://lists.wikimedia.org/hyperkitty/list/abstract-wikipedia@lists.wikimedia.org/thread/VPVALIXWEBAJDJYRI3EJX7HCGWXRFYFD/ example]).
* Click on "Delete this thread"


Wikimedia's mailing list archives, though public, are no longer indexed by search engines as they are excluded in <tt>robots.txt</tt>, other archives (such as Gmane) are indexed.
;Delete a single message
* Go to the message in hyperkitty (this can be tricky, go to the thread, click on the chain icon aka "Permanant link to this email"). [https://lists.wikimedia.org/hyperkitty/list/abstract-wikipedia@lists.wikimedia.org/message/VPVALIXWEBAJDJYRI3EJX7HCGWXRFYFD/ example]
* Click on "Delete this message"
=== Export a listing of all subscribers to a mailing list ===
* Go to members settings: https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/members/member/
* Click on "CSV export"


=== Export a listing of all subscribers to a mailing list ===
=== Check if an email address is subscribed to any lists ===
* Login to the mailing list server and run:
  <tt>/var/lib/mailman/bin/list_members -f -o <file to write to> <list name></tt>


=== Remove an individual from all mailing lists ===
=== Remove an individual from all mailing lists ===
Occasionally we need to remove an individual from every mailing list we have, such as when an email address no longer works but we don't want mailman to turn it off due to bounce detection. The remove_members command is the solution - this is a command line utility to remove one or more email addresses from a specific list or from all lists.
* ssh to lists1001.wikimedia.org
* remove an individual from a specific list:
* Run "sudo mailman-wrapper delmembers -m EMAIL_ADDRESS --fromall"
  /var/lib/mailman/bin/remove_members mylist user@example.com
* remove two addresses from all lists:
  /var/lib/mailman/bin/remove_members --nouserack --fromall user1@example.com user2@example.com
* remove from all private lists:
  /var/lib/mailman/bin/remove_from_private.sh <email>


=== Reset the admin password of a list ===
=== Reset a user's password ===
* ssh to lists1001.wikimedia.org.
* Run "sudo mailman-web changepassword USERNAME"


* If there is a request to reset a list password, first go to the admin page of the list, example if the list is called "wikimedia-l", the URL is: [https://lists.wikimedia.org/mailman/admin/wikimedia-l https://lists.wikimedia.org/mailman/admin/wikimedia-l]
=== Rename a mailing list ===
* Check who is listed in the "list run by" sentence in the footer. Note how when you hover over the admin names there is a mailto: link with a special list owner alias, example for wikimedia-l it's ''wikimedia-l-owner@lists.wikimedia.org''. Use that address to contact list admins, not random personal emails. It's the best way to ensure the request for a reset is valid and ALL admins get the new password, not just one of them, causing follow-up requests. Realize that there is just a single admin password all admins have to share and keep in sync.
* If it appears the request can be handled by one of the existing list admins, for example there are multiple admins and not all of them have stated they don't know the passwords, let them handle it and decentralize admin work where possible. Only if really needed, proceed to:
* Get the mailman master password from [[pwstore]]
* Login on the admin page using the master password and go to the "Passwords" section.
* Fill out the "Enter new administrator password:" form with a new random password and repeat it. There are also poster and moderator passwords but they are almost never used and if they are then let the admins handle giving out passwords to moderators. Click "Submit your changes"
* Send the new password to the list owner address described above. If a phabricator ticket exists for this (recommended!) and list admins are on phabricator, add them to ticket and resolve.


=== Rename a mailing list ===
Renaming a mailing list in the past was a fairly hands-on approach and required confidence with mailman at each stage. While this has not changed, a shell script has been made which makes the process more easier and more streamlined and reduces the risk of things going wrong within mailman itself. Although this is true, renames should be avoided without someone who is confident with mailman on-hand just in case any sort of issues pop up. Tailing the error and bounce logs as well before and after would be ideal.


The script can be ran on the lists server by running:
/usr/local/sbin/rename_list <old list> <new list>


The password and email address of the list can be set to anything (preferable not the list administrator's address) as it will be overwritten later on by the script.
=== Import from Google Groups ===
It is straightforward to migrate a list from Google Groups to Mailman. A list owner in Google should use Google Takeout ([https://support.google.com/groups/answer/9975859 see documentation]) to export the member list (members.csv) and archives (topics.mbox). File a task requesting a new list and include a link to the Takeout result (if the list contents are private, provide this information to the Mailman admins/SRE team privately).


==== Manual ====
Create the new list in Mailman like normal, then use the "Mass subscription" form at <https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/mass_subscribe/> to import the existing subscribers.
The manual process is documented below (and may be out of date as new optimisations are found) which can be done if the script does not work or the list is 'weird' in one way or another. It is best to avoid the manual method as it introduces more risks and human errors.


# Before you begin, verify that the requester is actually an administrator of the mailman list.
Copy the mbox to the mailing list server and run:
# Send an email to the -owner address of the list that you plan to rename a list.
# Create a [https://lists.wikimedia.org/mailman/create new list].
## Use the mailman master password as creator's (authentication) password.
## Follow the standardized [http://meta.wikimedia.org/wiki/Mailing_lists/Standardization#Naming_scheme naming scheme] where possible.
## Use your email as the initial list creator and receive auto-created password (will be overwritten by the move).
## Log into the administrative interface (https://lists.wikimedia.org/mailman/admin/<listname>) with the generated password or the site password.
# Before the next step be prepared to change the "real_name" value of the list in the interface, but don't send it yet. Have the mailman master password ready.
# Go to the lists server shell and copy the [http://www.mail-archive.com/mailman-users@python.org/msg43290.html config.pck] (this includes all settings, users, passwords!) and others (pending.pck, request.pck files) from old to the new list.
# Reload the administrative interface link and be logged out, because you have just overwritten the users and passwords as well, use the mailman master password and login again.
# Immediately change the real_name in the administrative interface.
# You should now see other users as list administrators and members, and now you can take your time and adjust other settings like the description field and "Prefix for subject line of list postings" and update the name there as well or let others do it.
# Copy the archive mbox from the old to new .mbox directory and rename it to reflect the new list name.
# Fix permissions (chown list:list *.mbox, chmod 664 *.mbox)
# Use [http://mail.python.org/pipermail/mailman-users/2003-August/031026.html arch] to recreate all html files from mbox to fix archive links.
# Decide if you want to keep old archives in place, you probably do. Don't break URLs.
# Add the old list email address to "acceptable aliases" on the new list administrative interface
# Merge a [https://gerrit.wikimedia.org/r/#/c/229978/3/files/exim/listserver_aliases mail alias] to redirect mail to the old list.
# Merge an [https://gerrit.wikimedia.org/r/#/c/229978/3/modules/mailman/templates/lists.wikimedia.org.erb url redirect] for the old listinfo page.
# Merge on palladium, run puppet on the mailing lists server. Apache and exim should automatically pick the change up.
# Test URL redirect.
# Test mail by announcing the change to the old list address.


=== Add a list to Gmane ===
<code>sudo mailman-web hyperkitty_import -l LISTNAME@lists.wikimedia.org /path/to/topics.mbox</code>
A new list can be added to Gmane very easily if the first message has not already been sent or history is not important (see [[#Step-by-step_procedure|creation steps above, #8]]). If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' ([https://phabricator.wikimedia.org/maniphest/task/create/?projects=Wikimedia-Mailing-lists%20operations easy link] stating which list needs to imported and to which Gmane group if it exists already.


=== Alter arbcom-l archive access list ===
This might take a while because of the "Synchronizing properties with Mailman" step, that's fine.
After a previous data leak, the English Wikipedia Arbitration Committee requested a high level of security for access to arbcom-l archives. We now have a second layer of password authentication in addition to Mailman's controls, implemented in Apache. Using HTTP authentication allows each user's page views to be tracked in the apache access logs.


To add a user:
Then update the search index:


* Have the user generate a GPG private key, and have them establish a link between their public key and their Wikipedia user account by posting the public key on Wikipedia. For Windows users, this can be done by following [https://en.wikipedia.org/wiki/User:Tim_Starling/Gpg4win_tutorial this tutorial].
<code>sudo mailman-web update_index_one_list LISTNAME@lists.wikimedia.org</code>
* Generate a password for them, for example using <tt>pwgen 24</tt>
* Choose a username.
* Update the htdigest file in the puppet private repo using <tt>htdigest ~/private/modules/secret/secrets/mailman/arbcom-l.htdigest 'arbcom-l archive' &lt;username></tt>
* Push the update out to the mailing lists server in the usual way.
* Send the new password to the user using encrypted email.


To do other things:
All done! Make sure to have the list owner in Google disable the group there, and remind subscribers to update the lists' email address to LISTNAME@'''lists.'''wikimedia.org.
 
=== Add a list to Gmane ===
A new list can be added to Gmane very easily if the first message has not already been sent or history is not important. If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' ([https://phabricator.wikimedia.org/maniphest/task/create/?projects=Wikimedia-Mailing-lists%20operations easy link] stating which list needs to imported and to which Gmane group if it exists already.


* Edit palladium:~/private/modules/secret/secrets/mailman/arbcom-l.htdigest, do a commit and run puppet on the mailing list servers (<tt>fermium.wikimedia.org</tt> atm)


=== Docs and links ===
=== Docs and links ===
Line 186: Line 92:
For a list of signed NDAs for staff/contractors there is [[phab:T83783|T83783]], [[phab:project/members/61|WMF-NDA]] and <tt>ldap/nda</tt> (or <tt>ldap/wmf</tt>).
For a list of signed NDAs for staff/contractors there is [[phab:T83783|T83783]], [[phab:project/members/61|WMF-NDA]] and <tt>ldap/nda</tt> (or <tt>ldap/wmf</tt>).


== Configuration details ==
=== Migrate servers ===
The new Mailman setup lives on [[fermium]], and uses the standard Debian package <tt>mailman</tt>. The mailing list state is under <tt>/var/lib/mailman/</tt>, the global configuration is in <tt>/etc/mailman/</tt>.
See [[Mailman/Migration]] for comments made about past migrations (lily to sodium, sodium to fermium, fermium to lists1001).


The mail server used is [[Exim]], the web server used is [[Apache]].
=== Upgrading Mailman3 ===
When upgrading Mailman3 packages for a version that includes schema changes, use the following process.


=== Mailman setup ===
# Downtime host in icinga
The local mailman configuration file is available to see on [https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/mailman/files/mm_cfg.py Diffusion]. Configuration is mostly default version config with few changes (L96 and below are non-default).
# Disable puppet
# Manually stop <code>mailman3</code> and <code>mailman3-web</code> systemd services
# Run apt to install the new packages
# Run <code>sudo mailman-wrapper help</code> to apply the mailman3 schema updates (yes, running just the "help" command will apply the schema changes)
# Run <code>sudo mailman-wrapper help</code> again to make sure the schema update didn't obviously break anything.
# Run <code>sudo mailman-web migrate</code> to apply the mailman3 schema updates
#Clear static CSS/JS caches: <code>sudo mailman-web collectstatic --clear --noinput && sudo mailman-web compress</code>
# Start mailman3 and mailman3-web systemd services again, keep an eye on errors in the logs
# Re-enable puppet and remove icinga downtime if it hasn't expired yet.
Post upgrade:


=== Mail server setup ===
* Verify that the messages the [[git:mediawiki/core/+/refs/heads/master/includes/installer/Installer.php|MediaWiki Installer screen-scrapes]] haven't changed.
Near the top of the <tt>exim4.conf</tt> file, there are several macros related to Mailman. These define system-specific settings/locations used by the router(s) and transport(s) in the rest of the configuration file. For a Debian/Ubuntu Mailman package, the following macro's are accurate:
# Mailman
MAILMAN_HOME = /usr/lib/mailman
MAILMAN_LISTS_HOME = /var/lib/mailman
MAILMAN_WRAP = MAILMAN_HOME/mail/mailman
MAILMAN_UID = list
MAILMAN_GID = list


There's a ''domain list'' that contains a list of all domains that can "contain" mailing lists, i.e. the domains for which the Mailman router(s) should run. This list is also used as part of the "local domains" list, the list for which this mail server accepts mail and handles it locally.
=== Block abusive traffic ===
domainlist mailman_domains = lists.wikimedia.org
* Use <code>profile::lists::web_deny_conditions</code> in private puppet to block at the Apache level (see [[gerrit:736211]])
domainlist local_domains = +system_domains : +mailman_domains
* Use the standard <code>abuse_networks</code> in private puppet which will block at the ferm/network level (see [[gerrit:736552]])


==== Main configuration ====
== Configuration details ==
Several tweaks have been made to the main configuration to make Mailman delivery go smooth.
The new Mailman3 setup lives on [[fermium|lists1001]]. Some state is under <tt>/var/lib/mailman3/</tt>, the global configuration is in <tt>/etc/mailman/</tt> via Puppet.


In case of high load / lots of incoming connections, mail from the local host (including Mailman) and other Wikimedia servers are given preference:
The mail server used is [[Exim]], the web server used is [[Apache]].
smtp_reserve_hosts = <; 127.0.0.1 ; ::1 ; +wikimedia_nets


For big mailing lists, Mailman needs to send a lot of recipients per mail / connection. Per default, Exim only queues mails that have > 10 recipients, to be delivered by a subsequent queue runner, which can cause significant delays. The default Mailman limit is 500 recipients per connection, so make Exim accept that:
=== Packages ===
smtp_accept_queue_per_connection = 500
We are currently using [https://apt-browser.toolforge.org/buster-wikimedia/component/mailman3/ custom backported packages] of the mailman3 suite. A few patches have been cherry-picked from upstream to fix issues we've run into. The packaging Git repositories can be found on apt1001 in legoktm's home directory.


Allow Exim to do 50 deliveries to remote hosts in parallel (this means 50 processes):
There is currently [[phab:T282348#7125322|one change]] that is hotpatched and not in the package yet.
remote_max_parallel = 50


==== Routers ====
=== Mailman setup ===
In Exim, the ''routers'' determine if a certain e-mail address is accepted for delivery or mail transport, and how it's going to be handled (routed). For Mailman, the following ''list'' router accepts a recipient that:
TODO: Add
* has a domain in the domain list ''mailman_domains''
* has a Mailman configuration file matching the local part (i.e. the mailing list exists)
Certain postfixes of the localpart, e.g. "-bounces" are accepted as well.


When the router accepts the recipient address, it's set up for delivery using the ''list'' transport (see below).
=== Mail server setup ===
 
See [[Mailman/MTA]]
<pre>
# Mailman list handling. Test the mailing list address without suffix
# first, as a mailing list like wikifi-admin is a valid list name.
 
list:
        driver = accept
        domains = +mailman_domains
        require_files = MAILMAN_LISTS_HOME/lists/$local_part/config.pck
        transport = list
 
list_suffix:
        driver = accept
        domains = +mailman_domains
        require_files = MAILMAN_LISTS_HOME/lists/$local_part/config.pck
        local_part_suffix = -bounces : -bounces+* : \
                                -confirm+* : -join : -leave : \
                                -owner : -request : -admin : \
                                -subscribe : -unsubscribe
        transport = list
</pre>
 
If the conditions for this router fail (i.e. the router is not run) then the ''no_more'' makes sure that no subsequent routers will be tried (in the current configuration there are none that might accept), and the recipient address is failed.
 
==== Transports ====
An Exim ''transport'' configures a way of transporting a message, e.g. over the network (SMTP), to a file (MBOX/Maildir/etc) or using a pipe to a process. The following transport sets up delivery to Mailman:
 
<pre>
# Mailman pipe transport
 
list:
        driver = pipe
        command = MAILMAN_WRAP \
                '${if def:local_part_suffix \
                        {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
                        {post}}' \
                $local_part
        current_directory = MAILMAN_LISTS_HOME
        home_directory = MAILMAN_LISTS_HOME
        user = MAILMAN_UID
        group = MAILMAN_GID
</pre>
 
For content scanning, temporary mbox files are written to <tt>/var/spool/exim4/scan</tt>, and deleted after scanning. Similarly, Exim keeps "hints" databases in <tt>/var/spool/exim4/db</tt>, which are non-essential caches. To improve performance somewhat, these directory is mounted as a tmpfs filesystem, using the following line in <tt>/etc/fstab</tt>:
tmpfs  /var/spool/exim4/scan  tmpfs  defaults        0      0
tmpfs  /var/spool/exim4/db    tmpfs  defaults        0      0
 
==== Mailing list privacy protection ====
It has happened in the past that by hitting on the ''Reply All'' button in one's e-mail client, private info from an internal list leaked to a public mailing list because it was listed in the CC list, and the imprudent sender did not notice. In order to try to catch these incidents, a little filter has been implemented.
 
# If the <tt>To:</tt> or <tt>CC:</tt> of a message body matches an item in a list of private mailing list addresses, and
# the list of recipients as known by the mailing list server contains a Wikimedia mailing list that's ''not'' a private mailing lists, then
# the message is bounced with the message ''Message rejected for privacy protection: The list of recipients contains both private and public lists''.
 
It's possible to circumvent this restriction by sending to the private list as a BCC, ''Blind Carbon Copy''.
 
This filter is implemented using an [http://www.exim.org/exim-html-4.63/doc/html/spec_html/ch42.html Exim system filter]:
<div class="mw-code" style="white-space: pre-wrap; margin-top: 1em; margin-bottom: 1em;"><nowiki># Exim filter
 
# Mailing list privacy protection
if foranyaddress $h_To:,$h_Cc: ( $thisaddress matches "\\N^(internal-l|private-l)@(lists\.|mail\.)?wiki[mp]edia\.org$\\N" ) then
    if foranyaddress $recipients ( $thisaddress matches "\\N@lists\.wikimedia\.org$\\N" and $thisaddress does not match "\\N^(internal-l|private-l)@\\N" ) then
        fail text "Message rejected for privacy protection: The list of recipients contains both private and public mailing lists"
    endif
endif</nowiki></div>
 
This filter is enabled in the configuration file using
system_filter = CONFDIR/system_filter
 
==== Address header rewriting ====
It turned out that, after the migration, many users kept sending mails to both the old and the new mailing list addresses, thereby causing duplicate messages. To reduce this, Exim has been configured to rewrite the old mailing list addresses to the new ones in the To: and CC: headers, using the following option on the <tt>list</tt> transport:
<div class="mw-code" style="white-space: pre-wrap; margin-top: 1em; margin-bottom: 1em;"><nowiki>list:
    ...
    # Rewrite body headers of old mailing list addresses to new ones
    headers_rewrite = \N^.*@(mail\.)?wiki[mp]edia\.org$\N "${if exists{MAILMAN_LISTS_HOME/lists/$local_part/config.pck}{$local_part@lists.wikimedia.org}fail}" ct</nowiki></div>


=== Web server setup ===
=== Web server setup ===
Line 306: Line 136:
Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.
Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.


See also http://www.gnu.org/software/mailman/mailman-install/node10.html
=== Backups ===
 
[[fermium|lists1001]] is backed up to [[backup1001]] using [[Bacula]]. All of /var/lib/mailman3 is backed up, which potentially includes on-disk templates and the search index. We do not back up /var/lib/mailman3/web which comes from the package and digests that haven't been sent yet.
=== SpamAssassin ===
SpamAssassin is installed using the default Ubuntu <tt>spamassassin</tt> package. A couple of configuration changes were made.
 
By default, spamd, if enabled, runs as root. To change this:
# adduser --system --home /var/lock/spamassassin --group --disabled-password --disabled-login spamd
 
The following settings were modified in <tt>/etc/default/spamassassin</tt>:
# Change to one to enable spamd
ENABLED=1
 
User preferences are disabled, spamd listens on the loopback interface only, and runs as user/group ''spamd'':
OPTIONS="--max-children 5 --nouser-config --listen-ip=127.0.0.1 -u spamd -g spamd"
 
Run spamd with nice level 10:
# Set nice level of spamd
NICE="--nicelevel 10"
 
=== Fighting spam in mailman ===
 
All list mail is sent through spamassassin which adds headers with a spam status and score.
 
This does not mean though that it automatically discards any messages. Only messages with a really high spam score (>= 12) are discarded immediately, below that messages are just tagged with a header. What actions to take based on this information is left to the list admins.
 
==== Spam from non-members ====
 
Most spam comes from generic spammers which harvested potential email address over the web and mailed them all: such spammers don't bother subscribing to the list. In contrast, it's rather rare for a public list to ever need receiving emails from non-members.
 
Multiple list admins report that they greatly reduced the amount of spam received simply by stopping acknowledging the messages, because in this way the spammer doesn't know the address is real.
 
This configuration is in Privacy options > Sender filter > generic_nonmember_action and you can set to "Discard". You can also reach this configuration at https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/sender/hold_these_nonmembers .


Do NOT discard emails from non-members if there is any chance that real people need to write to your list's subscribers without subscribing (typical examples: private committee lists, feedback lists). Just avoid "bounce" and hold for moderation; if it becomes impossible to handle moderated messages, try to stop acknowledging the receipt of messages held for moderation.
The database lives on m5, which is also regularly backed up into Bacula, too.
 
==== Spam scores ====
 
The mailman UI supports this via the configuration variable '''header_filter_rules''' aka. 'Spam Filter Regexp' (description: Filter rules to match against the headers of a message.).
See also https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html
 
This can be found in the administrative interface in '''Privacy options...-> [Spam filters] -> Spam Filter Regexp''' (or visit directly the URL, replacing YOURLIST with your list name: https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/spam/header_filter_rules ).
 
<!--
 
You can just rely on spam-status:
 
x-spam-status: yes 
 
or you can filter by score, example regex from [http://www.gnu.org/software/mailman/mailman-admin/node24.html mailman docs]:
 
X-Spam-Score: [*]{3,5}
This line will match from 3 to 5 stars in the value of this field.
 
-->
You can filter X-Spam-Score, which on lists.wikimedia.org as of 2015 is indicated with pluses (+) instead of stars (*):
 
X-Spam-Score: [+]{4,}
 
This line matches when 4 or more pluses are encountered on the same line, meaning a score of 4 or more. SpamAssassin considers messages to be spam over 4 points. See [[phabricator:T120446#1872322|some examples of real-life SpamAssassin scores (X-Spam-Report)]].
 
Additionally you have to pick an action for that which can be one of:
 
Defer  Hold  Reject  Discard  Accept
If you use "Hold" you will have to manually check the moderation queue every once in a while, but can avoid possible false positives, if you use "Discard" you don't have to do anything but _might_ discard some ham without noticing, though that is unlikely with the scores proposed above.
 
If spam is rampant, you could try and add other, more specific blacklists. For instance, if you find that a good portion of the spam is in BRBL or in MailSpike whitelist, you could add a rule after the score-based one, like
(RCVD_IN_BRBL_LASTEXT|RCVD_IN_MSPIKE_H2|RCVD_IN_MSPIKE_H1)
and set it to "Hold" messages. Do not discard messages based on hand-made filters like this, they may match good messages too. Instead, use "Hold" and go check the moderation queue in the next days, to ensure there are no false positives. Note however that "Hold" seems to override other "Discard" filters, so such a filter may prevent discarding of messages from non-members etc. if you have such discards configured.
 
=== Backups ===
[[fermium]] is backed up to [[helium]] using [[Bacula]]. The path on the source is /var/lib/mailman. The path on the target is whatever you choose in bconsole.


=== Tested failure modes ===
=== Tested failure modes ===
Line 391: Line 153:
If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.
If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.


== Who has the passwords? ==
== Spam fighting ==
See [[Mailman/Spam fighting]]
 
== Templates ==
Most emails sent out by Mailman3 can be customized using "templates". You can see the [[gerrit:plugins/gitiles/operations/software/mailman-templates/+/refs/heads/master/en/|default English templates]]. You can edit them by visiting [https://lists.wikimedia.org/postorius/lists/YOURLISTNAME.lists.wikimedia.org/templates https://lists.wikimedia.org/postorius/lists/<code>YOURLISTNAME</code>.lists.wikimedia.org/templates]. These are [[betawiki:Special:Translate/wikimedia-mailman-templates|translatable on translatewiki.net]].


The following people have the master (site) password that can be used to login to the admin interface of all lists.
It may also be useful to read the [https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/templates.html#line-wrapping template documentation] (the code parts can be ignored).


* all members of the operations team or users with root access on the ops bastion host
== Monitoring ==
* James Alexander
See [[Mailman/Monitoring]]
* (via James) Maggie Dennis


The following people have the list creator password that can be used to create new lists but not login to existing lists.
== Who has access? ==


* all members of the operations team or users with root access on the ops bastion host
=== Mailman3 ===
* James Alexander
The following people have "superuser" access via the web UI for admin access to all lists:
* (potentially other CA team members via James)


The following people have shell access on the list server. They can run mailman binaries which includes changing list configs and passwords.
* [[m:User:Legoktm|Legoktm]]
* [[m:User:Ladsgroup|Ladsgroup]]
* Nick Wilson (WMF)
* "Administrator" (all SRE members or users with root)
* "WMFOffice" (Trust and Safety team)


* all members of the operations team or users with root access on the ops bastion host
== Mailman2 data ==
Most private Mailman2 data is backed up in bacula in a long-term backup ([[phab:T282303#7114284|details]]). See [[Bacula#Restore_(aka_Panic_mode)|how to restore a file]].


[[Category:Mail]]
[[Category:Mail]]
[[Category:Services]]
[[Category:Services]]

Revision as of 12:53, 13 June 2022

See also

Components

Mailman3 has multiple components. Each doing a different thing. This is a simplified overview. For a more visual diagram, see the upstream architecture overview.

  • mailman3 is the service that handles emails and talks to MTA (in our case exim4) through LMTP/SMTP
  • mailman3-web or postorius is a django-based web service that handles settings, subscriptions, etc. It doesn't do anything on its own (except user management) but it talks internally to the mailman3 service through REST API.
  • hyperkitty (being used in the same webservice) is responsible for archives. It handles search, etc.
  • Search backend: Is xapian and the search index is in /var/lib/mailman3/web/fulltext_xapian_index
  • Databases: Two databases both in m5:
    • mailman3 which mailman's core service talks to. It holds memberships, lists, etc.
    • mailman3web which postorius and hyperkitty talk to. It holds archive of all emails and is rather big.
    • MTA: It's exim4 (see Mailman/MTA for more information)

How To

Create a mailing list

Consider if you need to notify Trust and Safety when creating new mailing lists.

Disable or re-enable a mailing list

  • Remove owners and add disabled-lists@lists.wikimedia.org as owner
    • This is important, otherwise we can't find it and change/fix its settings.
  • Set all emails to be discarded (both member and non-member)
  • Ban ^.*@.*
  • Add to description that this mailing list is disabled like [Disabled] or [Archived]

Remove a mailing list

Note: unless you know what you're doing, don't actually do this. It most likely does not do what you want. Just disable the list (see above) instead.

Remove a message from the mailing list archives

Delete the whole thread
  • Go to the thread in hyperkitty (example).
  • Click on "Delete this thread"
Delete a single message
  • Go to the message in hyperkitty (this can be tricky, go to the thread, click on the chain icon aka "Permanant link to this email"). example
  • Click on "Delete this message"

Export a listing of all subscribers to a mailing list

Check if an email address is subscribed to any lists

Remove an individual from all mailing lists

  • ssh to lists1001.wikimedia.org
  • Run "sudo mailman-wrapper delmembers -m EMAIL_ADDRESS --fromall"

Reset a user's password

  • ssh to lists1001.wikimedia.org.
  • Run "sudo mailman-web changepassword USERNAME"

Rename a mailing list

Import from Google Groups

It is straightforward to migrate a list from Google Groups to Mailman. A list owner in Google should use Google Takeout (see documentation) to export the member list (members.csv) and archives (topics.mbox). File a task requesting a new list and include a link to the Takeout result (if the list contents are private, provide this information to the Mailman admins/SRE team privately).

Create the new list in Mailman like normal, then use the "Mass subscription" form at <https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/mass_subscribe/> to import the existing subscribers.

Copy the mbox to the mailing list server and run:

sudo mailman-web hyperkitty_import -l LISTNAME@lists.wikimedia.org /path/to/topics.mbox

This might take a while because of the "Synchronizing properties with Mailman" step, that's fine.

Then update the search index:

sudo mailman-web update_index_one_list LISTNAME@lists.wikimedia.org

All done! Make sure to have the list owner in Google disable the group there, and remind subscribers to update the lists' email address to LISTNAME@lists.wikimedia.org.

Add a list to Gmane

A new list can be added to Gmane very easily if the first message has not already been sent or history is not important. If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' (easy link stating which list needs to imported and to which Gmane group if it exists already.


Docs and links

Authorized recipients for ops@lists.wikimedia.org

Anyone with a signed NDA can be on the ops@lists.wikimedia.org mailing list. For a list of signed NDAs for staff/contractors there is T83783, WMF-NDA and ldap/nda (or ldap/wmf).

Migrate servers

See Mailman/Migration for comments made about past migrations (lily to sodium, sodium to fermium, fermium to lists1001).

Upgrading Mailman3

When upgrading Mailman3 packages for a version that includes schema changes, use the following process.

  1. Downtime host in icinga
  2. Disable puppet
  3. Manually stop mailman3 and mailman3-web systemd services
  4. Run apt to install the new packages
  5. Run sudo mailman-wrapper help to apply the mailman3 schema updates (yes, running just the "help" command will apply the schema changes)
  6. Run sudo mailman-wrapper help again to make sure the schema update didn't obviously break anything.
  7. Run sudo mailman-web migrate to apply the mailman3 schema updates
  8. Clear static CSS/JS caches: sudo mailman-web collectstatic --clear --noinput && sudo mailman-web compress
  9. Start mailman3 and mailman3-web systemd services again, keep an eye on errors in the logs
  10. Re-enable puppet and remove icinga downtime if it hasn't expired yet.

Post upgrade:

Block abusive traffic

  • Use profile::lists::web_deny_conditions in private puppet to block at the Apache level (see gerrit:736211)
  • Use the standard abuse_networks in private puppet which will block at the ferm/network level (see gerrit:736552)

Configuration details

The new Mailman3 setup lives on lists1001. Some state is under /var/lib/mailman3/, the global configuration is in /etc/mailman/ via Puppet.

The mail server used is Exim, the web server used is Apache.

Packages

We are currently using custom backported packages of the mailman3 suite. A few patches have been cherry-picked from upstream to fix issues we've run into. The packaging Git repositories can be found on apt1001 in legoktm's home directory.

There is currently one change that is hotpatched and not in the package yet.

Mailman setup

TODO: Add

Mail server setup

See Mailman/MTA

Web server setup

To get Mailman running with Apache the puppet class role::lists is applied which uses module mailman. Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.

Backups

lists1001 is backed up to backup1001 using Bacula. All of /var/lib/mailman3 is backed up, which potentially includes on-disk templates and the search index. We do not back up /var/lib/mailman3/web which comes from the package and digests that haven't been sent yet.

The database lives on m5, which is also regularly backed up into Bacula, too.

Tested failure modes

Because mail delivery and transport should be reliable, I have tested what happens in certain failure modes, e.g. when SpamAssassin's spamd daemon is not running.

Spamd not running

Because of the /defer_ok modifiers in the Exim ACLs, Exim will act as if no spam filtering attempts are made when spamd is not running, and will accept the message. The following lines are logged:

spam acl condition: warning - spamd connection to 127.0.0.1, port 783 failed: Connection refused
spam acl condition: all spamd servers failed
H=xxx.xxxxxxx.xx [xx.xx.xx.xx]:xxxx I=[145.97.39.157]:25 U=exim Warning: ACL "warn" statement skipped: condition test deferred

Mailman not running

If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.

Spam fighting

See Mailman/Spam fighting

Templates

Most emails sent out by Mailman3 can be customized using "templates". You can see the default English templates. You can edit them by visiting https://lists.wikimedia.org/postorius/lists/YOURLISTNAME.lists.wikimedia.org/templates. These are translatable on translatewiki.net.

It may also be useful to read the template documentation (the code parts can be ignored).

Monitoring

See Mailman/Monitoring

Who has access?

Mailman3

The following people have "superuser" access via the web UI for admin access to all lists:

  • Legoktm
  • Ladsgroup
  • Nick Wilson (WMF)
  • "Administrator" (all SRE members or users with root)
  • "WMFOffice" (Trust and Safety team)

Mailman2 data

Most private Mailman2 data is backed up in bacula in a long-term backup (details). See how to restore a file.