You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Fundraising"

From Wikitech-static
Jump to navigation Jump to search
imported>Pcoombe
(rv)
imported>Eileen
Line 1: Line 1:
{{notice|Fundraising tech documentation is migrating to [https://www.mediawiki.org/wiki/Fundraising_tech mediawiki.org]. This documentation may be outdated.}}
[[File:DonationPipeline 201302.png|500px|thumbnail|right|WMF Donation Pipeline overview]]
[[File:DonationPipeline 201302.png|500px|thumbnail|right|WMF Donation Pipeline overview]]
The fundraising infrastructure is made up of quite a few different components to facilitate online user contributions and tracking/storing donor information.  
This is the homepage for fundraising-tech documentation. If you can't find what you are looking from from here then take a look at our [[mw:Fundraising_tech/notes/Draft:Documentation_overhaul|documentation plan]] and add the appropriate header & links.


Get information on working with [[Fundraising/tech|Fundraising Software Development]].


==Fundraising Emergencies==
Note that much of the content on this page should be moved to linked pages.


===Fundraising Engineering Documentation===
* If you want to understand what fr-tech does, who we are and how to contact us head over to https://www.mediawiki.org/wiki/Fundraising_tech
[https://collab.wikimedia.org/wiki/Fundraising/Engineering Fundraising Engineering Documentation] has with system information and emergency response protocols. Or more specifically [https://collab.wikimedia.org/wiki/Fundraising/Engineering/Shutting_the_pipeline_down Shutting the pipeline down] details how/when to disable banner campaigns or other fundraising/payment services.  
*There is some information on this page -  [[Fundraising/tech|Fundraising Software Development]]. - that needs to be migrated onto this main page
 
== A Donor-facing endpoints ==
[[Fundraising/DonationForm|DonationForm]] - reusable frontend
 
== B Internal endpoints ==
 
=== Drupal ===
WMF fundraising uses [https://civicrm.org/ CiviCRM] to track donor data.  
 
CiviCRM requires a 'host CMS' and to that end we [https://www.drupal.org/drupal-7.0 use Drupal7]. Drupal 7 is [https://www.drupal.org/psa-2019-02-25 EOL in November 2022] and next year we plan to upgrade to Drupal 9 - or maybe even 10. Our goal, however, is that we do not use any CMS-specific code going forwards. While we currently expect to stick with Drupal in we should be equally able to move to Wordpress. To this end we are in the process of migrating our [[Fundraising/Internal Endpoints/Drupal modules|drupal modules]] to CiviCRM extensions.
 
==== drush ====
[https://www.drush.org/ Drush] is a really useful drupal command line utility. There is a lot of documentation about drush on the internet but a few things to know with regards to WMF.
 
* On production, staging and in our docker dev set up we have an alias 'wmff' which tells drush details about where the code is and to use user 1.
* Common usage:


===Fundraising On-Call documentation===
{| class="wikitable"
[https://office.wikimedia.org/wiki/Fundraising/Engineering/On_call Fundraising Engineering On-call documentation] is a quick-reference page for on-call duty.
|+
!Command
!Where
!What
|-
|`drush @wmff updb`
|local dev and prod
|Run any database updates that need to be run
|-
|`drush @wmff up --security-only`
|local dev
|Download and install any security updates (these are then checked into git to deploy)
|-
|`drush @wmff cvapi Contact.get version=4 checkPermissions=0`
|local dev and prod
|Run a civicrm api  - the Contact.get action is probably not in itself useful but it does show how a api version 4 call would look
|}


==Fundraising Engineering==
=== Our CiviCRM customisations ===
Someone in the Fundraising Tech department can usually be reached in IRC, on the channel #wikimedia-fundraising.
We have customised CiviCRM using both [[Fundraising/Internal Endpoints/CiviCRM extensions|CiviCRM Extensions]] and [[Fundraising/Internal Endpoints/Drupal modules|drupal modules]]


===For Fundraising Engineers===
== C External services ==
Not sure what to do next? See [https://phabricator.wikimedia.org/tag/fundraising/ Fundraising Tech's Phabricator Workboard].


On call, and something had gone amiss? See [https://office.wikimedia.org/wiki/Fundraising/Engineering/On_call Fundraising Tech's on-call documentation].
== D Cluster layout, deployment, codebases ==


===Feature / Bug Trackers===
== E Data and multi-component processes ==
There's loads of information about how fr-tech triages bugs here: https://wikitech.wikimedia.org/wiki/Fundraising/Bug_Triaging


==Payment processors==
=== Payment processors ===
We have the ability to use several payment processors for online donations.  Currently, we route most credit card donations to GlobalCollect.
We have the ability to use several payment processors for online donations.  Currently, we route most credit card donations to GlobalCollect.
* '''Ingenico''': http://www.ingenico.com/
* '''Ingenico''': http://www.ingenico.com/
Line 40: Line 69:
{| class="wikitable" cellpadding="5" |
{| class="wikitable" cellpadding="5" |
|-
|-
! !! Ingenico !! PayPal !! Amazon !! Adyen !! WorldPay
! !! Ingenico !! PayPal !! Amazon !! Adyen !! WorldPay
!AstroPay
!AstroPay
!Banks
!Banks
Line 65: Line 94:
|-
|-
| Countries
| Countries
|  
|
| [https://www.paypal.com/worldwide/ list]
| [https://www.paypal.com/worldwide/ list]
| USA<ref>[https://payments.amazon.com/sdui/sdui/about?nodeId=73479#feat_countries https://payments.amazon.com/sdui/sdui/about?nodeId=73479#feat_countries]</ref>
| USA<ref>https://payments.amazon.com/sdui/sdui/about?nodeId=73479#feat_countries</ref>
|
|
|
|
Line 76: Line 105:
| [https://www.x.com/developers/paypal/documentation-tools/api/currency-codes list]
| [https://www.x.com/developers/paypal/documentation-tools/api/currency-codes list]
| USD
| USD
| All<ref>[http://www.adyen.com/platform/all-countries-all-currencies/ http://www.adyen.com/platform/all-countries-all-currencies/]</ref>
| All<ref>http://www.adyen.com/platform/all-countries-all-currencies/</ref>
|
|
|
|
Line 98: Line 127:
| Mobile optimized
| Mobile optimized
| bgcolor="gray" | No
| bgcolor="gray" | No
| bgcolor="yellow" | n/i<ref>[https://www.paypal.com/uk/webapps/mpp/sell-mobile https://www.paypal.com/uk/webapps/mpp/sell-mobile]</ref>
| bgcolor="yellow" | n/i<ref>https://www.paypal.com/uk/webapps/mpp/sell-mobile</ref>
| bgcolor="yellow" | n/i<ref>[https://payments.amazon.com/sdui/sdui/business?sn=devfps/mps https://payments.amazon.com/sdui/sdui/business?sn=devfps/mps]</ref>
| bgcolor="yellow" | n/i<ref>https://payments.amazon.com/sdui/sdui/business?sn=devfps/mps</ref>
| bgcolor="yellow" | n/i
| bgcolor="yellow" | n/i
|
|
Line 138: Line 167:
| bgcolor="#66ff66" | Yes || Implemented
| bgcolor="#66ff66" | Yes || Implemented
|-
|-
| bgcolor=yellow | n/i || Not yet implemented
| bgcolor="yellow" | n/i || Not yet implemented
|-
|-
| bgcolor=gray | No || Unsupported by processor
| bgcolor="gray" | No || Unsupported by processor
|}
|}


Line 152: Line 181:
<references />
<references />


==Message queues==
=== Email integration - Acoustic ===
[[Fundraising/Data and Integrated Processes/Acoustic Integration]]
 
Following the data flow, the donor's email address gets onto our subscription list either by entering their address while donating, or at a "remind me" prompt.  This email is entered into our CRM, and then a nightly export job stores in a CSV file.  This file is uploaded to Silverpop and imported into their database, where segmentation and other queries can be run to cut multiple lists per experiment.  Emails are templated using this data, and contain links back to our servers, which include Contact ID and other tracking information about the mailing.  Donatewiki landing page views are instrumented, and we do analysis on the results and use the tracking information to correlate with donations, to inform future emails.
===Message queues===
This describes the WMF fundraising systems configuration.  See the MediaWiki.org page on payments [[mw:Fundraising tech/Message queues|message queues]] for a discussion of how message queues are used to buffer and decouple fundraising infrastructure, and to read about the format and content of [[Fundraising/Normalized donation messages|normalized messages]].
This describes the WMF fundraising systems configuration.  See the MediaWiki.org page on payments [[mw:Fundraising tech/Message queues|message queues]] for a discussion of how message queues are used to buffer and decouple fundraising infrastructure, and to read about the format and content of [[Fundraising/Normalized donation messages|normalized messages]].


Line 183: Line 216:
We should clean up any unused queues, and overly narrowly defined ones.
We should clean up any unused queues, and overly narrowly defined ones.


==Jenkins==
We are using [http://jenkins-ci.org/ Jenkins] as a cron-replacement for handling some of the automation of certain fundraising tasks.  For extremely outdated internal documentation, see [[Fundraising.wikimedia.org#Hudson]].
Our Jenkins instance cannot be reached without first establishing a tunnel. For more information on fundraising jenkins and cron jobs, see [https://collab.wikimedia.org/wiki/Fundraising/Engineering/Infrastructure#Automated_Jobs Fundraising Infrastructure: Automated Jobs on Collab].
'''TODO'''
Replace Jenkins with cron jobs.


==Contribution tracking==
===Contribution tracking===
{{see also|Fundraising/ContributionTracking}}
{{see also|Fundraising/ContributionTracking}}
When a potential donor visits the Wikimedia donation page, a tracking record is created in the drupal.contribution_tracking table. This record includes the user's language, referrer, donation comment, opt-out status, a timestamp, and various other data. The tracking is handled on the MediaWiki side by the [[#DonationInterface|DonationInterface]] and [[#ContributionTracking|ContributionTracking]] extensions. If the user makes a successful donation, a contribution record is passed to CiviCRM via ActiveMQ. The queue2civicrm module then inserts the contribution record into the CiviCRM database and updates the tracking record with the id given to the contribution by CiviCRM.
When a potential donor visits the Wikimedia donation page, a tracking record is created in the drupal.contribution_tracking table. This record includes the user's language, referrer, donation comment, opt-out status, a timestamp, and various other data. The tracking is handled on the MediaWiki side by the [[#DonationInterface|DonationInterface]] and [[#ContributionTracking|ContributionTracking]] extensions. If the user makes a successful donation, a contribution record is passed to CiviCRM via ActiveMQ. The queue2civicrm module then inserts the contribution record into the CiviCRM database and updates the tracking record with the id given to the contribution by CiviCRM.
Line 219: Line 244:
TODO
TODO


==Drupal modules==
These modules interface with CiviCRM.  They are found in the [https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/fundraising/crm wikimedia/fundraising/crm] repo, under [https://phabricator.wikimedia.org/diffusion/WFCG/browse/master/sites/all/modules;ced148ca810b174856eb78fc7c33a1964fff372c sites/all/modules].


===contribution_tracking===
===Mediawiki extensions===
(Redundant) applies the contribution_tracking table schema.  This should be done using the MediaWiki extension instead.
 
===exchange_rates===
This module handles converting foreign currencies into US Dollars. It consists of two major components. One function handles doing the conversions based on exchange rates stored in the drupal.exchange_rates table. This function is called from the queue2civicrm module when it is processing donations from the queue. Another function, which is activated by a cron job, looks up the exchange rates on the web and updates the exchange_rates table. The exchange rates are currently pulled from oanda.com, with http://www.ecb.int/stats/eurofxref/eurofxref-daily.xml used as a back-up source.
 
===ganglia_reporter===
(Theoretically) exposes per-gateway throughput counts.
 
===globalcollect_audit===
Compares automatically-downloaded nightly audit files against the CRM database, and logs discrepancies.
 
===large_donation===
Send an email to the Major Gifts team when any online donation exceeds a configured dollar amount threshold.
 
===log_audit===
(Theoretically) audits nightly PayPal IPN logs against CRM database.
 
===mod_install_refresher===
Some weird convenience for development schema alters.
 
===offline2civicrm===
Import check and chargeback spreadsheets into the CRM.
 
===paypal_audit===
(Deprecated) audits Payflow Pro logs.
 
===public_reporting===
Replicates incoming donations into an anonymized database suitable for a public analysis server.
 
===queue2civicrm===
This module pulls online contributions from the ActiveMQ queue 'donations' and writes them to the CiviCRM database via the CiviCRM API. It is activated by a periodic Jenkins job, which fires an included [[Fundraising.wikimedia.org#Drush|Drush]] command (found in queue2civicrm/queue_consume.drush.inc) to replace the former cron method.  Output from the execution is captured in the Jenkins console logs.
 
Queue2civicrm also contains a hook (queue2civicrm_import) which is invoked at the end of a singular transaction, passing transactional information on to other modules that will perform other actions on the transaction.  The following modules make use of this hook:
* [[Fundraising#thank_you|thank_you]]
 
It also contains a hook which is invoked at the end of processing a batch of messages, which allows for other modules to perform batch processing.  This hook is made use of in the following modules:
* [[Fundraising#Recurring|recurring]]
 
The settings for queue2civicrm can be configured at http://<web root for drupal>/admin/settings/queue2civicrm.  You can configure the connection information to ActiveMQ, what queue to consume from and the number of messages to consume at once.  Also, you can test the ActiveMQ connection and test pushing messages into the queue.
 
Formerly, this module interacted directly with the PayflowPro gateway to validate transactions and passed queued items ready for consumption to a custom API for inserting them into CiviCRM.  This module no longer performs any gateway-specific actions and uses the built-in CiviCRM API for inserting contributions.
 
====Execution====
There are a number of ways to execute queue consumption with queue2civicrm and queue2civicrm/recurring.  The easiest way is with [[Fundraising.wikimedia.org#Drush|Drush]].  From the Drupal root,
<pre>
$ drush --user=1 qc
</pre>
 
This executes the queue2civicrm Drush script, located in queue2civicrm/queue_consume.drush.inc
 
===recurring===
The recurring module is a sub-module to queue2civicrm and shares much of the functionality.  As is it is right now, it is built particularly to handle raw IPN messages sent from PayPal pertaining to recurring (or in PayPal's language, 'subscription') contributions.  Because CiviCRM does not have an API for recurring contributions and due to the immaturity of some aspects of the Civi APIs, this module performs a lot of custom queries to both the CiviCRM and Drupal databases.
 
This module can be configured from the [[Fundraising#queue2civicrm|queue2civicrm]] configuration.
 
It processes messages out of its own queue (as configured in the interface).  Queue consumption is invoked by making use of the hook at the end of batch processing in [[Fundraising#queue2civicrm|queue2civicrm]].
 
===recurring_globalcollect===
GlobalCollect does not actively recur subscriptions.  This module is run daily, and will query the database for GC recurring donations which should be charged that day.
 
===thank_you===
This module sends a thank you message to online donors. It is called via a drupal hook from the queue2civicrm module every time a contribution is successfully recorded in the database. The contents of the thank you message are set from Administer » Site configuration » Thank you settings in CiviCRM.
 
===wmf_civicrm===
Compatibility and utility layer on top of the CiviCRM API.
 
===wmf_common===
Shared convenience library.
 
===wmf_communication===
Generalized mailing job management and templating.  Wraps the sendmail library (currently PHPMailer).
 
===wmf_contribution_search===
A small workaround for [http://issues.civicrm.org/jira/browse/CRM-12326 this bug].
 
===wmf_eoy_receipt===
(Unused) Create an tax receipt listing all of a donor's contributions for that year.
 
===wmf_reports===
Provides custom CiviCRM reports and export methods.
 
===wmf_unsubscribe===
Perform donor mailing list opt-out.
 
===wmf_unsubscribe_qc===
Read from the queue and call opt-out routines from wmf_unsubscribe.
 
==Mediawiki extensions==
The following Mediawiki extensions related to fundraising are installed on the [https://payments.wikimedia.org payments wiki]:
The following Mediawiki extensions related to fundraising are installed on the [https://payments.wikimedia.org payments wiki]:


Line 344: Line 279:
=====Fraud filtering=====
=====Fraud filtering=====
There are a series of extra sub-extensions, or filters, for payflowpro_gateway that perform analysis on credit card transactions to determine the likelihood that a transaction is fraudulent.  Each of the filters helps determine the 'risk score' for a transaction.  Actions to take based on certain risk scores can be configured for payflowpro_gateway (reject, review, challenge, accept).  The filters currently available include:
There are a series of extra sub-extensions, or filters, for payflowpro_gateway that perform analysis on credit card transactions to determine the likelihood that a transaction is fraudulent.  Each of the filters helps determine the 'risk score' for a transaction.  Actions to take based on certain risk scores can be configured for payflowpro_gateway (reject, review, challenge, accept).  The filters currently available include:
* MaxMind/MinFraud - a third party solution that helps analyze the transaction.  They return their own 'risk score' for a transaction which heavily influences our own internal scoring.  
* MaxMind/MinFraud - a third party solution that helps analyze the transaction.  They return their own 'risk score' for a transaction which heavily influences our own internal scoring.
* Referrer - Regular expressions can be configured to be run on a transaction's 'referrer', and each regex can be configured to apply a different score in the event that the referrer is a match.
* Referrer - Regular expressions can be configured to be run on a transaction's 'referrer', and each regex can be configured to apply a different score in the event that the referrer is a match.
* utm_source - Same as referrer, but for the utm_source bit in the tracking fields.
* utm_source - Same as referrer, but for the utm_source bit in the tracking fields.
Line 374: Line 309:
This extension makes it possible to provide a user-facing link that will direct the user to ''n'' possible URLs ''x''% of the time.  For instance, this is useful for doing randomized A/B testing - you can configure the module to send 50% of clicks to one URL, and the other 50% to another URL.  The percentages are completely configurable as are the URLs.  This was used extensively to make the 'Donate' sidebar link on Wikipedia point to a number of different donation landing pages for testing purposes.
This extension makes it possible to provide a user-facing link that will direct the user to ''n'' possible URLs ''x''% of the time.  For instance, this is useful for doing randomized A/B testing - you can configure the module to send 50% of clicks to one URL, and the other 50% to another URL.  The percentages are completely configurable as are the URLs.  This was used extensively to make the 'Donate' sidebar link on Wikipedia point to a number of different donation landing pages for testing purposes.


==Public reporting==
=== High-level Overview of Donation Pipeline ===
The best public fundraiser stats are now found on [http://samarium.wikimedia.org/ samarium].
Click the images for further explanation.
<gallery widths="320px" heights="340px">
File:Landing page to activemq.jpg|From the landing page to ActiveMQ
File:Activemq to civicrm.jpg|Inserting to queues in ActiveMQ to CiviCRM
</gallery>
 
*
=== Miscellaneous Scripts ===
There are some miscellaneous scripts to help with things like Paypal Verification, queue handling, etc.  Details of which can be found on [[Fundraising.wikimedia.org]]. 
 
=== Translations ===
''See [[Fundraising/Translation]] for more info''
 
* Donatewiki translations go out regularly on the l10n cache
* TYs need to be manually deployed - make a task for this and put it in pending review in the current sprint
* [https://translatewiki.net/w/i.php?title=Special:Translations&message=MediaWiki%3ADonate+interface-email-subject%2Fen Subject line] needs to be manually deployed - make a task for this and put it in pending review in the current sprint
* Payments needs to be manually deployed - make a task for this and put it in pending review in the current sprint
 
 
===Public reporting - possibly entirely out of date===


Real-time donor comments: http://wikimediafoundation.org/wiki/Special:ContributionHistory<br/>
Contribution statistics: http://wikimediafoundation.org/wiki/Special:ContributionStatistics


The public reporting data is stored in the civicrm.public_reporting table on [[db9]]. This data gets replicated to the slave database, [[db10]], which is where the public reporting pages actually pull the data from.
The public reporting data is stored in the civicrm.public_reporting table on [[db9]]. This data gets replicated to the slave database, [[db10]], which is where the public reporting pages actually pull the data from.
Line 390: Line 342:
The reporting pages are created through the ContributionReporting mediawiki [http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/ContributionReporting/ContributionReporting.php exntension]
The reporting pages are created through the ContributionReporting mediawiki [http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/ContributionReporting/ContributionReporting.php exntension]


== Bulk email ==
== F How we work ==
Wikimedia Foundation sends a limited number of bulk emails to donors over time.  I believe that each donor receives at most three solicitations each year [Citation needed].


=== Data flow ===
===Fundraising Emergencies===
Following the data flow, the donor's email address gets onto our subscription list either by entering their address while donating, or at a "remind me" prompt. This email is entered into our CRM, and then a nightly export job stores in a CSV file. This file is uploaded to Silverpop and imported into their database, where segmentation and other queries can be run to cut multiple lists per experiment. Emails are templated using this data, and contain links back to our servers, which include Contact ID and other tracking information about the mailing. Donatewiki landing page views are instrumented, and we do analysis on the results and use the tracking information to correlate with donations, to inform future emails.
[https://collab.wikimedia.org/wiki/Fundraising/Engineering Fundraising Engineering Documentation] has with system information and emergency response protocols. Or more specifically [https://collab.wikimedia.org/wiki/Fundraising/Engineering/Shutting_the_pipeline_down Shutting the pipeline down] details how/when to disable banner campaigns or other fundraising/payment services.  


== Miscellaneous Scripts ==
===Fundraising On-Call documentation===
There are some miscellaneous scripts to help with things like Paypal Verification, queue handling, etc.  Details of which can be found on [[Fundraising.wikimedia.org]].
[https://office.wikimedia.org/wiki/Fundraising/Engineering/On_call Fundraising Engineering On-call documentation] is a quick-reference page for on-call duty.


== High-level Overview of Donation Pipeline ==
===Feature / Bug Trackers===
Click the images for further explanation.
There's loads of information about how fr-tech triages bugs here: https://wikitech.wikimedia.org/wiki/Fundraising/Bug_Triaging
<gallery widths="320px" heights="340px" >
File:Landing page to activemq.jpg|From the landing page to ActiveMQ
File:Activemq to civicrm.jpg|Inserting to queues in ActiveMQ to CiviCRM
</gallery>
 
== Deployment ==
 
Some tips for a successful Fundraising deployment.  FIXME: this is not as useful as it should be, due to hostname obfuscation.
 
=== CentralNotice ===
 
CentralNotice is the only extension we commonly deploy to the main MediaWiki production cluster.  Please put your schedule on the [[Deployments]] calendar, and follow the usual [[How_to_deploy_code|instructions]].
 
Log into mwlog1001 and watch /a/mw-log/hhvm.log for any fatal errors you might be causing with your deployment.
 
== Initiatives ==
 
* [[/Locale support/]] - [https://mingle.corp.wikimedia.org/projects/fundraiser_2012/cards/grid?color_by=estimate&filters%5b%5d=%5bType%5d%5bis%5d%5bTech+Task%5d&filters%5b%5d=%5bType%5d%5bis%5d%5bTech+Defect%5d&filters%5b%5d=%5bTech+Status%5d%5bis+not%5d%5bDeployed%5d&filters%5b%5d=%5bre-evaluate%5d%5bis%5d%5b%5d&filters%5b%5d=%5bSupertask%5d%5bis%5d%5b1009%5d&group_by%5brow%5d=priority cards]
* [[/SmashPig/]] - decouple payment processing from other infrastructure
* [[/DonationForm/]] - reusable frontend
 
== Translations ==
''See [[Fundraising/Translation]] for more info''
 
* Donatewiki translations go out regularly on the l10n cache
* TYs need to be manually deployed - make a task for this and put it in pending review in the current sprint
* [https://translatewiki.net/w/i.php?title=Special:Translations&message=MediaWiki%3ADonate+interface-email-subject%2Fen Subject line] needs to be manually deployed - make a task for this and put it in pending review in the current sprint
* Payments needs to be manually deployed - make a task for this and put it in pending review in the current sprint


Not sure what to do next? See [[phab:tag/fundraising/|Fundraising Tech's Phabricator Workboard]]
[[Category:Fundraising]]
[[Category:Fundraising]]
[[Category:Fundraising - Needs Updated]]
[[Category:Fundraising - Needs Updated]]

Revision as of 05:03, 21 October 2021

WMF Donation Pipeline overview

This is the homepage for fundraising-tech documentation. If you can't find what you are looking from from here then take a look at our documentation plan and add the appropriate header & links.


Note that much of the content on this page should be moved to linked pages.

A Donor-facing endpoints

DonationForm - reusable frontend

B Internal endpoints

Drupal

WMF fundraising uses CiviCRM to track donor data.

CiviCRM requires a 'host CMS' and to that end we use Drupal7. Drupal 7 is EOL in November 2022 and next year we plan to upgrade to Drupal 9 - or maybe even 10. Our goal, however, is that we do not use any CMS-specific code going forwards. While we currently expect to stick with Drupal in we should be equally able to move to Wordpress. To this end we are in the process of migrating our drupal modules to CiviCRM extensions.

drush

Drush is a really useful drupal command line utility. There is a lot of documentation about drush on the internet but a few things to know with regards to WMF.

  • On production, staging and in our docker dev set up we have an alias 'wmff' which tells drush details about where the code is and to use user 1.
  • Common usage:
Command Where What
`drush @wmff updb` local dev and prod Run any database updates that need to be run
`drush @wmff up --security-only` local dev Download and install any security updates (these are then checked into git to deploy)
`drush @wmff cvapi Contact.get version=4 checkPermissions=0` local dev and prod Run a civicrm api - the Contact.get action is probably not in itself useful but it does show how a api version 4 call would look

Our CiviCRM customisations

We have customised CiviCRM using both CiviCRM Extensions and drupal modules

C External services

D Cluster layout, deployment, codebases

E Data and multi-component processes

Payment processors

We have the ability to use several payment processors for online donations. Currently, we route most credit card donations to GlobalCollect.

  • Ingenico: http://www.ingenico.com/
    • Ingenico has the ability to handle payments from multiple international systems including: credit card, direct debit, real time bank transfer, eWallets and more.
  • PayPal:
  • Amazon: A widget on our page, integrated using Login and Pay with Amazon.
  • Adyen: https://www.adyen.com/, documentation
  • WorldPay An alternate credit card processor that we use primarily for France and RTL languages. We can do other methods through them, too, if we want.

See also, "Ways to Give" for our recommended donation methods according to country.

Payment processor capabilities:

Ingenico PayPal Amazon Adyen WorldPay AstroPay Banks Checks
Credit card Yes Yes Yes Yes Yes Yes
Bank transfer Yes Yes No Yes IBAN, Swift
Countries list USA[1]
Currencies [2] list USD All[3]
Direct debit [4] No No n/i
Recurring Yes Yes Yes Yes[5]
Mobile optimized No n/i[6] n/i[7] n/i
Languages [8] [9] ? ?
Account required No No Yes No No No
Refund by API n/i n/i n/i n/i
Fully automated auditing Yes Yes Yes Yes No Yes

Legend:

Yes Implemented
n/i Not yet implemented
No Unsupported by processor
Notification failure policies:

When we don't respond to an IPN message from a payment processor with a successful HTTP code, they usually resend it.

Adyen: back-off algorithm from 5 minutes to 8 hrs, then every 8 hrs for a week

Amazon: every hour for 14 days

Email integration - Acoustic

Fundraising/Data and Integrated Processes/Acoustic Integration

Following the data flow, the donor's email address gets onto our subscription list either by entering their address while donating, or at a "remind me" prompt. This email is entered into our CRM, and then a nightly export job stores in a CSV file. This file is uploaded to Silverpop and imported into their database, where segmentation and other queries can be run to cut multiple lists per experiment. Emails are templated using this data, and contain links back to our servers, which include Contact ID and other tracking information about the mailing. Donatewiki landing page views are instrumented, and we do analysis on the results and use the tracking information to correlate with donations, to inform future emails.

Message queues

This describes the WMF fundraising systems configuration. See the MediaWiki.org page on payments message queues for a discussion of how message queues are used to buffer and decouple fundraising infrastructure, and to read about the format and content of normalized messages.

WMF fundraising uses the ActiveMQ (http://activemq.apache.org/) message broker for most queues, and STOMP as its wire protocol. This queue server is outside of PCI scope, and communicates with CiviCRM.

The limbo queue *WILL SOON BE* stored in Redis on the payments cluster, which is important for PCI certification. We cannot afford to have our queue servers in scope.

Various queue wrangling techniques are available.

ActiveMQ

All queues feeding into services outside the fr-cluster live on a single ActiveMQ instance. This is a SPOF.

Frack Redis

The DonationInterface frontends set and delete messages on localhost's queue. Nice to have: if record to delete is not found, try the other masters.

Orphan slayer pops messages off of available masters in round-robin order. Set and delete are to the same queue the message came from.

Replication

Redis masters on payments1-3 replicate to three slaves on payments4.

codfw holds (three or) six replicas of the queues on eqiad payments1-4.

TODO

Port all STOMP queuing to high-availability Redis instead.

We should clean up any unused queues, and overly narrowly defined ones.


Contribution tracking

When a potential donor visits the Wikimedia donation page, a tracking record is created in the drupal.contribution_tracking table. This record includes the user's language, referrer, donation comment, opt-out status, a timestamp, and various other data. The tracking is handled on the MediaWiki side by the DonationInterface and ContributionTracking extensions. If the user makes a successful donation, a contribution record is passed to CiviCRM via ActiveMQ. The queue2civicrm module then inserts the contribution record into the CiviCRM database and updates the tracking record with the id given to the contribution by CiviCRM.

Stats

Banner impressions and landing page stats are collected from the production proxies. Fundraising_Analytics/Impression_Stats. The wmf:Thank_you page includes wmf:Template:Hide_banners which loads Special:HideBanners from multiple domains via image src. HideBanners sets cookies for donors which tell CentralNotice's bannerController.js not to pester them for a year or so.

Reports

Contribution tracking data is surfaced through a stats portal on foundation wiki. All entries have to be white listed under wgAllowedTemplates in CommonSettings.php so that we don't surface any doctored banners.

utm_source

This is a tracking variable which is supposed to collect information about the transaction. Currently, it is a period-separated concatenation of three components. One interpretation of the components is, 1) banner name, 2) landing page name, and 3) payment method. We are currently in the process of standardizing (see FR #965 and FR #673).

In theory, each component may be a tilde-concatenation of a sequence of landing pages, for example. That code is badly dysfunctional.

utm_medium

Donor was referred by this type of site: sitenotice, spontaneous, sidebar, socialmedia.

Seems unuseful at this broad granularity.

utm_campaign

The parent campaign for the banner where this donation was initiated.

utm_key

TODO


Mediawiki extensions

The following Mediawiki extensions related to fundraising are installed on the payments wiki:

DonationInterface

This module handles all of the payment gateway specific functionality for the fundraising system at the point the user makes a donation. Other handling (secondary verification, queue managing) is handled on Fundraising.wikimedia.org. DonationInterface It is made up of the following components:

activemq_stomp

Contains the Stomp library and provides a mechanism for correctly formatting a message for storage in ActiveMQ. This could probably be refactored a bit to allow the other extension pieces to more flexibly rely on this to communicate with ActiveMQ. As it is right now, the functionality exposed by activemq_stomp.php is very limited, however the included Stomp library exposes whatever you might need to communicate with the queue.

extras

This directory contains several different hooks that can be used as needed at various points in a gateway's donation workflow. Extras include conversion_log, custom_filters (used for fraud prevention), minfraud (which also exists as a custom filter), and recaptcha.

gateway_common

Generally, the gateway_common directory contains gateway adapter code written to be gateway-agnostic, but which all the gateway-specific classes are descended from.

gateway_forms

Forms and form classes that are not written to be gateway-specific should be kept here.

*_gateway

All gateway-specific code can be found here.

scripts

In the course of a normal globalcollect hosted credit card transaction, a small percentage of donors will complete a transaction on the globalcollect hosted credit card form, and not manage to come back to the globalcollect ResultSwitcher page. The function of the ResultSwitcher page is to both filter and finalize the pending transaction and record it in the case of a successful conversion, so the donor not returning to the ResultSwitcher would effectively strand the otherwise successful transaction in a sort of limbo state. The code contained in the globalcollect scripts directory was written to address the problem of these stranded transactions.

This maintenance script is currently running on a Jenkins job, and uses data pulled from the 'cc-limbo' queue in ActiveMQ.

modules

Generally speaking, all modules that could be said to be gateway-agnostic should live in this directory. These modules will be loaded as-needed with resource loader.

payflowpro_gateway

This exposes the credit card donation form (as opposed to the PayPal stuff) and handles the communication between the donation page and PayflowPro. It negotiates the entire process from the point the user submits the donation to insertion of a donation into ActiveMQ - if a donation is considered valid at the point of submission, it is placed in the 'donation' queue. Otherwise, it is either rejected or placed in the 'pending_pfp' queue for review. Further handling happens on pending messages by Fundraising.wikimedia.org#PayflowPro_Pending_Transaction_Verification.

Fraud filtering

There are a series of extra sub-extensions, or filters, for payflowpro_gateway that perform analysis on credit card transactions to determine the likelihood that a transaction is fraudulent. Each of the filters helps determine the 'risk score' for a transaction. Actions to take based on certain risk scores can be configured for payflowpro_gateway (reject, review, challenge, accept). The filters currently available include:

  • MaxMind/MinFraud - a third party solution that helps analyze the transaction. They return their own 'risk score' for a transaction which heavily influences our own internal scoring.
  • Referrer - Regular expressions can be configured to be run on a transaction's 'referrer', and each regex can be configured to apply a different score in the event that the referrer is a match.
  • utm_source - Same as referrer, but for the utm_source bit in the tracking fields.
Logging

Our configuration of payflowpro_gateway causes two local logfiles to be generated:

  • /var/log/mw/minfraud
  • /var/log/mw/payflow

The 'minfraud' log logs more than just 'minfraud' related information. It tracks a user's transaction through our fraud prevention filters, and if their transaction is considered safe enough to be sent off to PayflowPro, Payflow's response object also gets logged here. This log also gets used to help us determine patterns in fraudulent transactions, analyze more general patterns in our transactions and determine why or why not a certain transaction was accepted/rejected.

The 'payflow' log logs more than just payflow related information. It's more of a debug log, tracking a transactions progress through the different parts of the payflowpro_gateway experience. The only piece of transactional data that actually gets recorded in the log is our internal 'trxn_id', which is considered the 'invoice id' by PayPal/Payflow (this makes it possible to line up transactional information either later on our end [eg from CiviCRM] or from the PayPal manager with the information contained in this log). We primarily use this log to keep track of timing information - how long it takes to communicate with the various third party services (MaxMind and PayflowPro), timeouts, retries, etc.

It is important to note that for our purposes, these log files actually get aggregated and later archived - more details can be found on the documentation for the payments cluster.

paypal_gateway

Handles the redirect from the Wikimedia site to PayPal for donations.

We are implementing a stand-alone listener to talk to PayPal's IPN and push pending PayPal transactions into the proper queue (see Fundraising.wikimedia.org#PayPal_IPN_Listener).

ContributionTracking

See also: Tracking Architecture

The ContributionTracking extension provides an unlisted special page for logging online donations. When the donation process is initiated, the extension stores some basic information in the Drupal contribution_tracking table and the transaction is assigned a unique, internal ID. This ID is used to track the contribution through the process of being validated and ultimately consumed in CiviCRM.

This extension also exposes a function contributionTrackingConnection() which will allow you to connect to the database containing the contribution_tracking table.

VariablePage

This extension makes it possible to provide a user-facing link that will direct the user to n possible URLs x% of the time. For instance, this is useful for doing randomized A/B testing - you can configure the module to send 50% of clicks to one URL, and the other 50% to another URL. The percentages are completely configurable as are the URLs. This was used extensively to make the 'Donate' sidebar link on Wikipedia point to a number of different donation landing pages for testing purposes.

High-level Overview of Donation Pipeline

Click the images for further explanation.

Miscellaneous Scripts

There are some miscellaneous scripts to help with things like Paypal Verification, queue handling, etc. Details of which can be found on Fundraising.wikimedia.org.

Translations

See Fundraising/Translation for more info

  • Donatewiki translations go out regularly on the l10n cache
  • TYs need to be manually deployed - make a task for this and put it in pending review in the current sprint
  • Subject line needs to be manually deployed - make a task for this and put it in pending review in the current sprint
  • Payments needs to be manually deployed - make a task for this and put it in pending review in the current sprint


Public reporting - possibly entirely out of date

The public reporting data is stored in the civicrm.public_reporting table on db9. This data gets replicated to the slave database, db10, which is where the public reporting pages actually pull the data from.

The code that manages the public reporting data lives in /srv/org.wikimedia.civicrm/sites/all/bin/public_reporting on the grosley server (donate.wikimedia.org).

  • table.sql - Creates the public_reporting table
  • trigger.sql - Sets up the database triggers for updating the table
  • synchronize.sql - Can be used to manually sync the public reporting data with the existing contribution records

The reporting pages are created through the ContributionReporting mediawiki exntension

F How we work

Fundraising Emergencies

Fundraising Engineering Documentation has with system information and emergency response protocols. Or more specifically Shutting the pipeline down details how/when to disable banner campaigns or other fundraising/payment services.

Fundraising On-Call documentation

Fundraising Engineering On-call documentation is a quick-reference page for on-call duty.

Feature / Bug Trackers

There's loads of information about how fr-tech triages bugs here: https://wikitech.wikimedia.org/wiki/Fundraising/Bug_Triaging

Not sure what to do next? See Fundraising Tech's Phabricator Workboard