As of Aug 2025 we run CiviCRM standalone (no CMS) and we currently have no patches/ hacks of our own. When we do patch CiviCRM we upstream the patch in parallel to try to get back to this unpatched state.
When upgrading CiviCRM the new process is to simply
1) checkout the right branch from upstream in the `core` directory
2) within the core directory run composer then deleted the vendor folder - this is because the vendor directory has been merged with our main vendor (using composer-merge-plugin) but there are still some packages like bower-components that could get updated this way
3) run composer update in the main directory to get any updates
4) check it all in - the gitignore should avoid any files from the core directory that we do not want being checked in
We are using the CiviCRM utility
cv
to execute CiviCRM actions from the command line. When calling cv we have a WMF wrapper wmf-cv. This means that it is not necessary to be in the webroot to call it and that the user is always set to 1 when calling it.
cv command
Where
What
wmf-cv updb
local dev and prod
Run any database updates that need to be run
wmf-cv dl <extension>
local dev
Download and install any security updates (these are then checked into git to deploy)
wmf-cv api4 Contact.get
local dev and prod
Run a civicrm api - the Contact.get action is probably not in itself useful but it does show how a api version 4 call would look
clean up invalid languages option value if no contact is using that language and option label equal name
Clearing template cache
You can run
/usr/local/bin/civicrm_cache_clear
on civi1002 to clear out the template cache dir if a change to a tpl isn't being reflected on the front end after a deploy. It will sudo and rm the dirs/files under
/srv/org.wikimedia.civicrm-files/private/cache/en_US
.
Troubleshooting setup
Make sure the settings directory (private/) is writeable by the user that runs setup.sh
Make sure you have the latest version of civicrm-buildkit in src/civicrm-buildkit.
If you see problems finding classes while enabling the civicrm extension, this may be due to stale metadata cached in redis. You can use the queues-redis-clear.sh script to clear that out (note you will also lose all messages in the donation and job queues).
Log Tables
We use CiviCRM logging to create log tables which we use for
- Threshold queries - query log_civicrm_contribution for contributon updates in the last 5 minutes (not includes any sort of update)
- Forensics - eg
https://phabricator.wikimedia.org/T311438
involved figuring out history of a contact around 18 months ago. Importantly determined they never had any contributions - ie didn't have them & then they were deleted, but never existing
Or best practices
CiviCRM uses the
drupal whitespace standards
(sadly adopted before the world co-alesced on something different) so all drupal modules & civicrm extensions (including the CiviCRM codebase) should use that. When using phpstorm installing the drupal plugin will add this standard. Within the vendor directory PSR2 is likely more appropriate
We agreed to name our civicrm extensions
with the prefix 'wmf-` where they are WMF specific and just the name if they are to be shared - ie `wmf-civicrm` and `deduper` respectively. This applies going forwards (we may or may not fix existing ones)
Running CiviCRM behind a firewall
In order to run CiviCRM behind a firewall we have disabled some SystemChecks on production (but not on our local developer environments) and use a ckeditor image uploader that embeds images. These are
documented in the main CiviCRM firewall documentation
. In our case the use of browser certificates for authentication means the server cannot reach it's own urls during crons
We supplement Core deduping behaviour with the
Deduper
extension - which we also maintain as an open source extension.
We have 4 ways of tackling duplicate contacts in CiviCRM
When piping contacts into CiviCRM we do a basic strict lookup based on first name, last name and email to try to match the contact
When manually importing we implement various rules - see
Importing Data
We have an automated script that runs through our most recently modified contacts all the time (based on civicrm_contact.modified_date) and attempts to run dedupe in safe mode. This includes various conflict resolution strategies described in the
Deduper extension documentation
and configured on our site at civicrm/admin/setting/deduper.
We have started to use the CiviCRM queue mechanism for some of our processes - with a view to using more broadly as we move away from Drupal7.
The main queue documentation is on
gitlab
but some updates that we have deployed on production have not (as of Nov 2023) been updated to gitlab. Notably this includes
new api interactions
which allow us to run the queue from drush for debugging.
Setting the queue paused Setting will not interfere with the current queue item being acted on, but once it is released no new items can be claimed.To see what is currently either claimed or stalled try the query
More or less do what coworker does but end when finished rather than keeping monitoring.
Queue log output
If there is an error processing a queue item it will output to syslog - you can grep for 'civicrm.queue' to find it (or civicrm.queue.ERROR to only find errors). Note the firtst part
(Task "omni-snooze#366920") gives both the queue_name and the queue_item id. The portion after the full stop is the error message from the failed action
civicrm.queue.ERROR: Task "omni-snooze#366920" failed and should be retried. Request failed: Recipient must be existing, and OPTED IN or SNOOZED, and (truncated)
Running coworker locally on the docker container, cd into the
/srv/civi-sites/wmff/drupal
and run the following commands to initialise CiviCRM and initiate the pipe for the process:
coworkerrun-vvv--pipe='drush ev "civicrm_initialize(); Civi::pipe();"'
Viewing the status of a queue
The status of the queue can be found on the
civicrm_queue
table on the
civicrm
database. Statuses include:
pending
,
aborted
, and
active
.
Automated emails from CiviCRM
We send out the following automated emails
Recurring failure notifications - these are send when a monthly recurring email is failing and encourages people to set up a new one. It is not sent if they have an active recurring email.
Thank you letters - these are send by an automated job for every donation in CiviCRM, unless the 'no_thankyou' field is populated or it is a recurring donations
End of year emails
- these are sent at the start of the year to cover all the recurring contributions in the previous year. These can also be sent ad hoc to individual donors (in which case they include all donations, not just the recurring ones)
Thank you , monthly convert, recurring failure notification and end of year email text & translations
The text for these emails is managed within CiviCRM in the
Message Templates interface
. using
Smarty
template language (version 2). This allows for the use of some variables and some conditional logic.
For example most templates have a block like the annotated one below. This code block uses the variables $first_name, $last_name, $contact_type and $email_greeting_display to construct a greeting
{if$first_nameand$last_name}// If we have both namesDear{$first_name},// then use the first name as a salutation {elseif$contact_type==='Organization'&&$email_greeting_display}// Otherwise, if they are an organization with a saved greeting {$email_greeting_display},// use the saved greeting{else}// otherwise Deardonor,// use a standard greeting (appropriate to the translation){/if}// always close the if!
Money formatting in emails
The {$amount} assigned to the template is formatted according to the relevant language - for example in Canadian French it renders to 4 000,99 $ US but in Vietnamese it displays as 4.000,99 US$
One gotcha with this formatting is the library assumes the currency is implicit when the currency matches the language. For example in the English United States template it does not specify the currency when the currency is USD. Fortunately (once
https://phabricator.wikimedia.org/T341101
goes out) we can assume that each language should have one currency it might drop the try currency for and we can add it back on a per-template basis.
The table below shows how we can add the iso code for USD only. It also shows that it only makes sense to do this when the language is English (United States) because in other languages it is already being added as it is not implicit, resulting in double ups. So the best option for English (Unitied States) templates is
{$amount} {if $currency ==='USD'}{$currency}(/if}
but for the portuguese template it is simply {$amount}.
However, Donor Relations makes use of the ISO codes down the bottom so duplication there appears to be OK
Currency
Language
Template Code
Displays
CAD
English (United States)
{$amount}
$CA 50.00
USD
English (United States)
{$amount}
$50.00
USD
Portuguese (Portugal)
{$amount}
4 000,99 US$
CAD
English (United States)
{$amount} {$currency}
$CA 50.00 CAD
USD
English (United States)
{$amount} {$currency}
$50.00 USD
USD
Portuguese (Portugal)
{$amount} {$currency}
4 000,99 US$ USD
CAD
English (United States)
{$amount} {if $currency ==='USD'}{$currency}(/if}
$CA 50.00
USD
English (United States)
{$amount} {if $currency ==='USD'}{$currency}(/if}
$50.00 USD
USD
Portuguese (Portugal)
{$amount} {if $currency ==='USD'}{$currency}(/if}
4 000,99 US$ USD
CiviMail records
When we send out thank you emails we track them in the CiviMailing system - even though they were not initiated by civimail. This attaches a verp address as explained in the
CiviCRM documentation
. In order to match any replies or bounces we get from this we need a record in the `civicrm_mailing_event_queue` with the queue_id and hash. This is of time-limited value - once we no longer care about the related mailing event records we don't need the queue record. The query below gives data about the mailing event records.
Currently we only have current data in 3 of the tables
- bounces (We are now creating activities for any received bounces or replies)
- replies (<500 in total)
- delivered events (these are created when we create the mailing records - they really aren't of much value to us.
CiviCRM supports sending out manual emails to small groups but expects you to use CiviMail to send to larger groups. We have configured out site to permit sending to 700 rather than the normal 50 under the 'small group' category via the `simple_mail_limit` setting. This is a hidden setting because there are implications to having a larger group in that
1) sending lots of emails without such things as unsubscribe options and organization details can negatively affect an mail servers reputation, affecting deliverability. In our case this is somewhat mitigated by the fact that this is a small portion of the email we send
2) bounce (and potentially unsubscribe requests) are not handled
Although the 'classic' way to use CiviMail is through the CiviMail interface it can be done from search results - which is the likely usage for WMF staff. In that case it still creates a group within CiviCRM for the receipients but the group is hidden from the user interface.
When CiviMail is used the 'from' address is changed to a 'verp' address - ie a hash followed by a dot and the email address of an internal email that we process via a scheduled job
We have added a number of fields that are calculated by triggers to CiviCRM. These calculate totals in various time periods, along with maximum donations, latest donations etc. The are calculated using MySQL triggers. Every maintenance week we need to check we have created sufficient future custom fields - eg. for a maintenance week in the latter half in 2025 we would check that we have already created the fields for 2026 and if not add them. The addition of these fields is likely to take around an hour due to the table size.
In addition to the 'normal' calculated fields we have 2 fields that are calculated according to fairly complex rules - the donor segment and donor status. The rules can be seen within CiviCRM
on this page
. It is important to treat this page as the source of truth as it is generated by the same code as the triggers, and hence is up to date with them. The status & segment fields need to be updated once a year when the financial year rolls over because
a) the triggers have the financial year hard-coded into them and
b) once the triggers are updated the field values need to be updated.
An easy way to set this off is via the upgrader - per this
patch
One gotcha with this process is that the Acoustic upload files are abnormally large after this runs as they include a more complete subset of the database. As of writing there is a
known issue with the phpsec library and
uploading large files - if we don't resolve this by July 1 we may be able to upload the file using normal sftp and then call Omnicontact.upload with the isAlreadyUploaded flag set to 1
Also note that for recurring donors the segment & status fields are updated more regularly by the segmentation_aging job - as these are more date-sensitive
Custom fields
Custom fields in CiviCRM can be created through the user interface. In order to allow flexibility to our users the arrangement we have with our super-users (Nora, Rosie) is that they can create custom fields through the UI but they should create a phab task so that fr-tech can follow up ( add field to advance search and so on ).
The follow up by frtech is in 2 parts - ensuring the fields are present in our dev environments and creating triggers.
Ensuring the fields are present in our dev environments
Keeping our dev environment fields in sync is a best-efforts endeavor rather than something we keep 100% in sync, but it does make it easier for us to develop locally. There is currently an 'old method' and a 'new method. The new method is to use the
CiviCRM managed entities functionality
as reflected in `
CustomGroupMatchingGiftInformation.mgd.php`
file. The previous barriers to this standardised approach, around the use of the more efficient BulkSave action and matching existing entities, are now resolved. However we DO set the action on update & cleanup to 'never' to avoid anything unexpected on production. The command to reconcile these is
wmf-cvapi4Managed.reconcile
The old method involves tracked/synced fields being declared in the CustomGroups.php file in the Managed directory in the `wmf_civicrm` extension. '. This file follows the conventions of the
CiviCRM managed entities functionality
and the fields declared in the file are added to our developer builds on install.
However, because the file is not a direct match to the Custom fields on prod, we have not registered this file with the civicrm_managed hook, and instead we have a custom wmf command which adds any declared fields in the CustomGroups.php, that are missing for the dev site.
The new command is
wmf-cvapi4WMFConfig.syncCustomFields
This command adds CustomGroups and CustomFields to dev sites if missing, but does not update them. It only creates option values if the field did not previously exist or it is being run in a development environment - ie we want to add but not update on live.
Note that
When declaring the fields in the CustomGroups array an easy way to get the data from live is to use the
API v4 explorer
- once the criteria are selected & execute has been hit the field data is listed in a json format and there is even an option next to it to switch to a php format. Fields that do not differ from the defaults (including is_active) should be removed from the resulting array, along with the id field. The explorer can be used in a similar way to get the CustomFields in the group and any option values (using the option_group_id from the custom field definition). Do not include option_group_id or custom_group_id in the checked in array
In CivICRM all field types can be extended with custom groups - however, CiviCRM must know that they can be extended. CiviCRM has a hard coded mapping of the common entities (Contribution, Contact etc) but also maintains an option group ` cg_extends` with other entities. When extending an entity type that is not extendable by default we need to ensure the option value exists.
In some cases the functionality of the custom fields are owned by extensions rather than WMF user driven. In these cases the fields are declared in the relevant extension (e.g the Omnimail extension installs 2 custom groups and the relationship block extension installs one). These are written into the Upgrade classes in the relevant extensions and extension upgrades are run using the following command.
drush@wmffcvapiExtension.upgrade
Update triggers on production
The process for updating the triggers we use for logging doing this is in
the subpage
You can run the following drush command on dev or staging to import (60) contributions and get timings.
drush@wmffqperf-d60"your comment"
Query tracking
If you wish to see what queries run you can add this line to your civicrm.settings.php file - the 'n' is just part of the file name, in case you want to do separate runs
define('CIVICRM_DEBUG_LOG_QUERY','n');
You can also capture
just the queries for one command
using env - eg. this will log the queries for the command in question but not other queries
envCIVICRM_DEBUG_LOG_QUERY=deleteddrush@wmff-vvcivicrm-cleanup-contacts--batch=500--date_cutoff="1 year ago"
The extension
org.wikimedia.systemtools
provides a script / api to help analyse this file - you may need to installl it - in the UI it is called "Home for WMF helpers".
Once enabled you can run the following
It will output a csv with a cleaned up version of the queries to the directory you are in - the output will give the filename (spoiler - it's gonna be query_log_parsed.csv).
Also note the extension has a Readme.md....
At this point I generally figure out where a single row starts & ends in the file & pick one from the middle of the file & discard the rest....
Redis monitoring
We use Redis for our caching service. Within CiviCRM values retrieved from Redis are 'mostly' cached within the php layer so Redis is hit once per process for most keys. This matters quite a lot for large arrays - as the serialization & unserialization of them has turned out to be slow. In general each process should not be doing `GET` requests for the same key multiple times (unless the cache has been flushed / the key deleted).
I wrote a rather long
blog
on debugging Redis related slow downs that goes over the next part in more depth
To see what Redis is hitting you can do the following
- Add to your ~/.profile
exportREDIS_PWD=*************
aliasredis-monitor='redis-cli -a $REDIS_PWD monitor'
Note the **** password is in civicrm.settings.php
At this point you can run
redis-monitor>redis.log
If you scp the file down locally there is a script to help make sense of it in the systemtools extension (which might need to be enabled) - e.g
- outputs '`' (backtick) separated file which can be imported into mysql (backtick seems to be otherwise not present so usable but we could make the separator a parameter)
Code cleanups
Preferred Language (completed code cleanup)
Historically our code just added together the language string 'en' and the country string 'DK' to get 'en_DK' - since that wasn't in the database it was just added. so we would love to clean up the language mess.
So we first view civicrm option values from
search kit
and find out that we have different types of the invalid language options, like typo 1 or a one as xx which represents nothing, or the one added historically.
First we write a drush commend which can filter out the languages that contain the same value and the name ( means they are added to db since wasn't there as en_DK ) then compare with the contacts, that if we do not have any contacts that use this option value, then it's ok to delete it from the civicrm_option_value table.
Initially want to replace them all in db with the civi install, as this
ticket
, then find we might encounter the issue of deadlock, so then decides to use the cron table.
In order to prevent the db deadlock, we use the cron table with batch size to run the contacts query, and then update the invalid language to the default one, as "update_language."
Used this process control cron job to update the invalid prefer languages for contact based on their current prefer languages as `sh -c "echo '{\"values\":{\"preferred_language\":\"en_US\"},\"where\":[[\"preferred_language\",\"NOT IN\",[\"en_US\", \"en_CA\",\"en_ZA\", \"en_AU\", \"en_GB\", \"en_NZ\", \"en_IN\"]], [\"preferred_language\",\"LIKE\",\"en_%\"]],\"limit\":5000, \"version\":4}' | drush @wmff cvapi Contact.update --in=json"`, refer to this
ticket
then we clean up the option values with this
patch
and run `drush @wmff cvapi WMFDataManagement.CleanInvalidLanguageOptions version=4` to actually delete from civicrm_otpion_value table;
Used this api explorer to update the in_active option value to active: Edit civicrm option value from
api explorer
, refer to this
ticket
eventually, we have the following language options:
MariaDB [civicrm]> select id, value, label, name, is_active, is_default from civicrm_option_value where option_group_id=86 and is_active = 1 order by value;
id
value
label
name
is_active
is_default
586
aa
Afar
aa_ET
1
0
585
ab
Abkhaz
ab_GE
1
0
596
ae
Avestan
ae_XX
1
0
587
af
Afrikaans (af)
af_ZA
1
0
588
ak
Akan
ak_GH
1
0
590
am
Amharic (am)
am_ET
1
0
592
an
Aragonese
an_ES
1
0
591
ar
Arabic (Egypt)
ar_EG
1
0
594
as
Assamese
as_IN
1
0
595
av
Avaric
av_RU
1
0
597
ay
Aymara
ay_BO
1
0
598
az
Azerbaijani (az)
az_AZ
1
0
600
ba
Bashkir
ba_RU
1
0
602
be
Belarusian (be)
be_BY
1
0
608
bg
Bulgarian
bg_BG
1
0
604
bh
Bihari
bh_IN
1
0
605
bi
Bislama
bi_VU
1
0
599
bm
Bambara
bm_ML
1
0
603
bn
Bengali (bn)
bn_BD
1
0
743
bo
Tibetan Standard, Tibetan, Central
bo_CN
1
0
607
br
Breton
br_FR
1
0
606
bs
Bosnian
bs_BA
1
0
610
ca
Catalan; Valencian
ca_ES
1
0
612
ce
Chechen
ce_RU
1
0
611
ch
Chamorro
ch_GU
1
0
617
co
Corsican
co_FR
1
0
618
cr
Cree
cr_CA
1
0
620
cs
Czech
cs_CZ
1
0
704
cu
Old Church Slavonic, Church Slavic, Church Slavonic, Old Bulgarian, Old Slavonic
