You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Fundraising: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ejegg
imported>Cstone
Line 81: Line 81:
* https://www.adyen.com/, [https://support.adyen.com/index.php?/Knowledgebase/List/Index/101/documentation documentation]
* https://www.adyen.com/, [https://support.adyen.com/index.php?/Knowledgebase/List/Index/101/documentation documentation]
* Backup credit card processor in most countries, primary in a few. As of December 2021 Adyen is the only gateway we use to process iDEAL and Apple Pay transactions.
* Backup credit card processor in most countries, primary in a few. As of December 2021 Adyen is the only gateway we use to process iDEAL and Apple Pay transactions.
==== [[Fundraising/Data_and_flow/PSP_integrations/D*Local|D*Local]] ====
==== [[Fundraising/Data_and_flow/PSP_integrations/dLocal|dLocal]] ====
* https://dlocal.com/
* https://dlocal.com/
* A payment processor specializing in the local payment methods of South America and India
* A payment processor specializing in the local payment methods of South America and India

Revision as of 20:19, 13 January 2022

WMF Donation Pipeline overview

This is the homepage for fundraising-tech documentation. If you can't find what you are looking from from here then take a look at our documentation plan and add the appropriate header & links.


Note that much of the content on this page should be moved to linked pages.

A Donor-facing endpoints

DonationForm - reusable frontend

B Internal endpoints

Drupal

WMF fundraising uses CiviCRM to track donor data.

CiviCRM requires a 'host CMS' and to that end we use Drupal7. Drupal 7 is EOL in November 2022 and next year we plan to upgrade to Drupal 9 - or maybe even 10. Our goal, however, is that we do not use any CMS-specific code going forwards. While we currently expect to stick with Drupal in we should be equally able to move to Wordpress. To this end we are in the process of migrating our drupal modules to CiviCRM extensions.

drush

Drush is a really useful drupal command line utility. There is a lot of documentation about drush on the internet but a few things to know with regards to WMF.

  • On production, staging and in our docker dev set up we have an alias 'wmff' which tells drush details about where the code is and to use user 1.
  • Common usage:
Command Where What
`drush @wmff updb` local dev and prod Run any database updates that need to be run
`drush @wmff up --security-only` local dev Download and install any security updates (these are then checked into git to deploy)
`drush @wmff cvapi Contact.get version=4 checkPermissions=0` local dev and prod Run a civicrm api - the Contact.get action is probably not in itself useful but it does show how a api version 4 call would look

Our CiviCRM customisations

We have customised CiviCRM using both CiviCRM Extensions and drupal modules

We also customise CiviCRM with custom fields and other data types. These are generally declared in the extension 'wmf_civicrm' in the 'managed' folder. Note that while we try to use the CiviCRM managed entities functionality the Custom fields use a pseudo version of this and to refresh custom fields managed entities use our command - `drush @wmff ucf`

Automated emails from CiviCRM

We send out the following automated emails

  • Recurring failure notifications - these are send when a monthly recurring email is failing and encourages people to set up a new one. It is not sent if they have an active recurring email.
  • Thank you letters - these are send by an automated job for every donation in CiviCRM, unless the 'no_thankyou' field is populated or it is a recurring donations
  • End of year emails - these are sent at the start of the year to cover all the recurring contributions in the previous year. These can also be sent ad hoc to individual donors (in which case they include all donations, not just the recurring ones)

C External services

D Cluster layout, deployment, codebases

Deployment

Current deployment documentation

Staging server

The staging server is part of fr-techs fundraising cluster. The database on it is manually updated from live every few months. The server is configured such that email does not go out from the box except to a small list of fr-tech emails per a config file at /etc/postfix/virtual. The quick and effective to see whether/where mail is delivered is /var/log/mail.log

E Data and multi-component processes

Payment processors

We have the ability to use several payment processors for online donations. Currently, we route most credit card donations to Ingenico (now called WorldLine).

Ingenico

  • http://www.ingenico.com/
  • Ingenico has the ability to handle payments from multiple international systems including: credit card, direct debit, real time bank transfer, eWallets and more. We're currently only using them for cards.

PayPal

Amazon

  • A widget on our page, integrated using Login and Pay with Amazon.

Adyen

  • https://www.adyen.com/, documentation
  • Backup credit card processor in most countries, primary in a few. As of December 2021 Adyen is the only gateway we use to process iDEAL and Apple Pay transactions.

dLocal

  • https://dlocal.com/
  • A payment processor specializing in the local payment methods of South America and India

See also, "Ways to Give" for our recommended donation methods according to country.

Payment processor capabilities:

Ingenico PayPal Amazon Adyen D*Local Banks Checks
Credit card Yes Yes Yes Yes Yes
Bank transfer Yes Yes No Yes Yes IBAN, Swift
Countries list USA[1] Latam+IN
Currencies [2] list USD All[3]
Direct debit [4] No No Yes
Recurring Yes Yes Yes Yes n/i
Mobile optimized No n/i[5] n/i[6] n/i
Languages [7] [8] ? ?
Donor needs account No Yes Yes No No
Refund by API n/i n/i n/i n/i n/i
Fully automated auditing Yes Yes Yes Yes Yes

Legend:

Yes Implemented
n/i Not yet implemented
No Unsupported by processor

Notification failure policies:

When we don't respond to an IPN message from a payment processor with a successful HTTP code, they usually resend it.

Adyen: back-off algorithm from 5 minutes to 8 hrs, then every 8 hrs for a week

Amazon: every hour for 14 days

  1. ↑ https://payments.amazon.com/sdui/sdui/about?nodeId=73479#feat_countries
  2. ↑ See GlobalCollectAdapter::getCurrencies
  3. ↑ http://www.adyen.com/platform/all-countries-all-currencies/
  4. ↑ DE, IT, NL (todo: AT, BE, CH, ES, FR, GB)
  5. ↑ https://www.paypal.com/uk/webapps/mpp/sell-mobile
  6. ↑ https://payments.amazon.com/sdui/sdui/business?sn=devfps/mps
  7. ↑ See GlobalCollectAdapter::getAvailableLanguages. Our code must find a fallback language if the donor's native tongue is unsupported.
  8. ↑ See PaypalAdapter::stage_locale. For unknown reasons, we have to specify language *by country*.

Email integration - Acoustic

Acoustic is the service we use to send out bulk emails. They are able to handle high volumes of emails and are responsible for managing server reputation to improve deliverability. Acoustic also provide tools for A-B testing to see which emails perform better. In order to be able to use our donor information from Acoustic we have a nightly upload job. We also re-import information from acoustic - for details go to Fundraising/Data and Integrated Processes/Acoustic Integration

Note that prior brandings of the Acoustic platform may still linger - ie Silverpop, WCM, Watson Campaign Manager, or sometimes just 'IBM'

Message queues

This describes the WMF fundraising systems configuration. See the MediaWiki.org page on payments message queues for a discussion of how message queues are used to buffer and decouple fundraising infrastructure, and to read about the format and content of normalized messages.

WMF fundraising uses the PHP-Queue library to abstract queue access. In production we use Redis lists as queue storage This redis server is outside of PCI scope, and communicates with CiviCRM.

Various queue wrangling techniques are available.

Redis

All queues feeding into services outside the fr-cluster live on a single Redis instance. This is a SPOF.

TODO

We should clean up any unused queues, and overly narrowly defined ones.

Contribution tracking

See also: Fundraising/ContributionTracking

When a potential donor visits the Wikimedia donation page, a tracking record is created in the drupal.contribution_tracking table. This record includes the user's language, referrer, donation comment, opt-out status, a timestamp, and various other data. The tracking is handled on the MediaWiki side by the DonationInterface extension, which retrieves a contribution_tracking_id from a sequence generator in Redis. If the user makes a successful donation, a contribution record is passed to CiviCRM via the donations queue. The queue2civicrm module then inserts the contribution record into the CiviCRM database and updates the contribution_tracking record with the id given to the contribution by CiviCRM.

Stats

Banner impressions and landing page stats are collected from the production proxies. Fundraising_Analytics/Impression_Stats. The wmf:Thank_you page includes wmf:Template:Hide_banners which loads Special:HideBanners from multiple domains via image src. HideBanners sets cookies for donors which tell CentralNotice's bannerController.js not to pester them for a year or so.

utm_source

This is a tracking variable which is supposed to collect information about the transaction. Currently, it is a period-separated concatenation of three components. One interpretation of the components is, 1) banner name, 2) landing page name, and 3) payment method. We are currently in the process of standardizing (see FR #965 and FR #673).

In theory, each component may be a tilde-concatenation of a sequence of landing pages, for example. That code is badly dysfunctional.

utm_medium

Donor was referred by this type of site: sitenotice, spontaneous, sidebar, socialmedia.

Seems unuseful at this broad granularity.

utm_campaign

The parent campaign for the banner where this donation was initiated.

utm_key

TODO

Mediawiki extensions

The following Mediawiki extensions related to fundraising are installed on the payments wiki:

DonationInterface

Renders donation forms and handles donor interaction, redirecting donors to payment processors when necessary and then either presenting an error or redirecting to a thank you page.

Extension documentation on mediawiki.org

Fraud filtering

There are a series of extra filters, that perform analysis on credit card transactions to determine the likelihood that a transaction is fraudulent. Each of the filters helps determine the 'risk score' for a transaction. Actions to take based on certain risk scores can be configured per gateway (reject, review, challenge, accept). The filters currently available include:

  • MaxMind/MinFraud - a third party solution that helps analyze the transaction. They return their own 'risk score' for a transaction which heavily influences our own internal scoring.
  • Referrer - Regular expressions can be configured to be run on a transaction's 'referrer', and each regex can be configured to apply a different score in the event that the referrer is a match.
  • utm_source - Same as referrer, but for the utm_source bit in the tracking fields.

FundraisingEmailUnsubscribe

Allows a donor to unsubscribe from fundraising-related emails.

Extension documentation on mediawiki.org

High-level Overview of Donation Pipeline

Click the images for further explanation.

  • From the landing page to ActiveMQ

    From the landing page to ActiveMQ

  • Inserting to queues in ActiveMQ to CiviCRM

    Inserting to queues in ActiveMQ to CiviCRM

Miscellaneous Scripts

There are some miscellaneous scripts to help with things like Paypal Verification, queue handling, etc. Details of which can be found on Fundraising.wikimedia.org.

Translations

See Fundraising/Translation for more info

  • Donatewiki translations go out regularly on the l10n cache
  • TYs need to be manually deployed - make a task for this and put it in pending review in the current sprint
  • Subject line needs to be manually deployed - make a task for this and put it in pending review in the current sprint
  • Payments needs to be manually deployed - make a task for this and put it in pending review in the current sprint


Public reporting

We export some extremely aggregated datasets at https://frdata.wikimedia.org/, generated every half hour by the public_data_export process-control job running the fundraiser_public_data_export and fundraiser_public_data_mover child jobs.

fundraiser_public_data_export runs the FundraiserStatisticsGen/fundstatgen.py script from the tools repo, then fundraiser_public_data_mover just rsyncs the data to the frdata server.

F How we work

Fundraising Emergencies

Fundraising Engineering Documentation has with system information and emergency response protocols. Or more specifically Shutting the pipeline down details how/when to disable banner campaigns or other fundraising/payment services.

Fundraising On-Call documentation

Fundraising Engineering On-call documentation is a quick-reference page for on-call duty.

Feature / Bug Trackers

There's loads of information about how fr-tech triages bugs here: https://wikitech.wikimedia.org/wiki/Fundraising/Bug_Triaging

Not sure what to do next? See Fundraising Tech's Phabricator Workboard


Subpages:

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Fundraising&oldid=585891"