You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Fundraising: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Eileen
imported>Cstone
 
(23 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[[File:DonationPipeline 201302.png|500px|thumbnail|right|WMF Donation Pipeline overview]]
[[File:Fundraising Tech Architecture Diagram.jpg|thumb|500x500px|WMF Donation Pipeline overview]]
This is the homepage for fundraising-tech documentation. If you can't find what you are looking from from here then take a look at our [[mw:Fundraising_tech/notes/Draft:Documentation_overhaul|documentation plan]] and add the appropriate header & links.
This is the homepage for fundraising-tech documentation. If you can't find what you are looking from from here then take a look at our [[mw:Fundraising_tech/notes/Draft:Documentation_overhaul|documentation plan]] and add the appropriate header & links.


Line 8: Line 7:
* There is some information on this page -  [[Fundraising/tech|Fundraising Software Development]]. - that needs to be migrated onto this main page
* There is some information on this page -  [[Fundraising/tech|Fundraising Software Development]]. - that needs to be migrated onto this main page


== A External-facing ==
== External-facing ==
==== Payments ====
=== [[Fundraising/External-facing/Payments|Payments]] ===
legacy [[Fundraising/DonationForm|DonationForm]] - reusable frontend
=== Mediawiki extensions===
==== [[Fundraising/External-facing/E-mail preference center|Email Preference Center]] ====
The following Mediawiki extensions related to fundraising are installed on the [https://payments.wikimedia.org payments wiki]:


== B Internal endpoints ==
====DonationInterface====
Renders donation forms and handles donor interaction, redirecting donors to payment processors when necessary and then either presenting an error or redirecting to a thank you page.


=== CiviCRM (Drupal) ===
Extension documentation on [[mw:Extension:DonationInterface|mediawiki.org]]
WMF fundraising uses [https://civicrm.org/ CiviCRM] to track donor data.


CiviCRM requires a 'host CMS' and to that end we [https://www.drupal.org/drupal-7.0 use Drupal7]. Drupal 7 is [https://www.drupal.org/psa-2019-02-25 EOL in November 2022] and next year we plan to upgrade to Drupal 9 - or maybe even 10. Our goal, however, is that we do not use any CMS-specific code going forwards. While we currently expect to stick with Drupal in we should be equally able to move to Wordpress. To this end we are in the process of migrating our [[Fundraising/Internal Endpoints/Drupal modules|drupal modules]] to CiviCRM extensions.
====FundraisingEmailUnsubscribe====
Allows a donor to unsubscribe from fundraising-related emails.


==== drush ====
Extension documentation on [[mw:Extension:FundraisingEmailUnsubscribe|mediawiki.org]]
[https://www.drush.org/ Drush] is a really useful drupal command line utility. There is a lot of documentation about drush on the internet but a few things to know with regards to WMF.


* On production, staging and in our docker dev set up we have an alias 'wmff' which tells drush details about where the code is and to use user 1.
===Public reporting===
* Common usage:
We export some extremely aggregated datasets at https://frdata.wikimedia.org/, generated every half hour by the public_data_export process-control job running the fundraiser_public_data_export and fundraiser_public_data_mover child jobs.


{| class="wikitable"
fundraiser_public_data_export runs the [[phab:diffusion/WFTO/browse/master/FundraiserStatisticsGen/fundstatgen.py|FundraiserStatisticsGen/fundstatgen.py script from the tools repo]], then fundraiser_public_data_mover just rsyncs the data to the frdata server.
|+
!Command
!Where
!What
|-
|`drush @wmff updb`
|local dev and prod
|Run any database updates that need to be run
|-
|`drush @wmff up --security-only`
|local dev
|Download and install any security updates (these are then checked into git to deploy)
|-
|`drush @wmff cvapi Contact.get version=4 checkPermissions=0`
|local dev and prod
|Run a civicrm api  - the Contact.get action is probably not in itself useful but it does show how a api version 4 call would look
|}


=== Our CiviCRM customisations ===
legacy [[Fundraising/DonationForm|DonationForm]] - reusable frontend
We have customised CiviCRM using both [[Fundraising/Internal Endpoints/CiviCRM extensions|CiviCRM Extensions]] and [[Fundraising/Internal Endpoints/Drupal modules|drupal modules]]
=== [[Fundraising/External-facing/E-mail preference center|Email Preference Center]] ===


====Custom fields====
=== [[Fundraising/External-facing/Donor Data Delivery Tool|Donor Data Delivery Tool]] ===
Custom fields in CiviCRM can be created through the user interface. In order to allow flexibility to our users the arrangement we have with our super-users (Nora, Rosie) is that they can create custom fields through the UI but they should create a phab task so that fr-tech can follow up ( add field to advance search and so on ).


The follow up by frtech is in 2 parts - ensuring the fields are present in our dev environments and creating triggers.
== Internal-facing ==


===== Ensuring the fields are present in our dev environments =====
=== [[Fundraising/Internal-facing/CiviCRM|CiviCRM]] ===
Keeping our dev environment fields in sync is a best-efforts endeavor rather than something we keep 100% in sync, but it does make it easier for us to develop locally. All tracked/synced fields are declared in the CustomGroups.php file in the Managed directory in the `wmf_civicrm` extension. This file follows the conventions of the [https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_managed/ CiviCRM managed entities functionality] and the fields declared in the file are added to our developer builds on install.


However, because the file is not a direct match to the Custom fields on prod, we have not registered this file with the civicrm_managed hook, and instead we have a custom wmf command which adds any declared fields in the CustomGroups.php, that are missing for the dev site.
==Libraries==
===[[Fundraising/SmashPig|SmashPig]]===


As of May 2022 this command is about to be changed via https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/801443 The new command will be<syntaxhighlight lang="bash">
==Service Providers==
drush @wmff cvapi WMFConfig.syncCustomFields version=4
{| class="wikitable"
</syntaxhighlight>
|+
 
!Service Provider Name
The old command was<syntaxhighlight lang="bash">
!Production URL
drush @wmff ucf
!SandBox URL
</syntaxhighlight>This command adds CustomGroups and CustomFields to dev sites if missing, but does not update them. It only creates option values if the field did not previously exist or it is being run in a development environment - ie we want to add but not update on live.
!Sandbox Accounts
 
!Test Cards
Note that
 
* [[File:Php-output.png|thumb]]When declaring the fields in the CustomGroups array an easy way to get the data from live is to use the [https://civicrm.wikimedia.org/civicrm/api4#/explorer/CustomGroup/get?where=%5B%5B%22is_active%22,%22%3D%22,%22true%22%5D%5D&limit=0 API v4 explorer]  - once the criteria are selected & execute has been hit the field data is listed in a json format and there is even an option next to it to switch to a php format. Fields that do not differ from the defaults (including is_active) should be removed from the resulting array, along with the id field. The explorer can be used in a similar way to get the CustomFields in the group and any option values (using the option_group_id from the custom field definition). Do not include option_group_id or custom_group_id in the checked in array
* In CivICRM all field types can be extended with custom groups - however, CiviCRM must know that they can be extended. CiviCRM has a hard coded mapping of the common entities (Contribution, Contact etc) but also maintains an option group ` cg_extends` with other entities. When extending an entity type that is not extendable by default we need to ensure the option value exists.<syntaxhighlight lang="php">
CRM_Core_BAO_OptionValue::ensureOptionValueExists([
  'option_group_id' => 'cg_extend_objects',
  'name' => 'civicrm_relationship',
  'label' => ts('Relationship'),
  'value' => 'Relationship',
]);
</syntaxhighlight>
 
* In some cases the functionality of the custom fields are owned by extensions rather than WMF user driven. In these cases the fields are declared in the relevant extension (e.g the Omnimail extension installs 2 custom groups and the relationship block extension installs one). These are written into the Upgrade classes in the relevant extensions and extension upgrades are run using the following command.
<syntaxhighlight lang="bash">
drush @wmff cvapi Extension.upgrade
</syntaxhighlight>
 
===== Update triggers on production =====
We use mysql triggers to log civicrm database updates to the log tables. These triggers are managed by CiviCRM. However, our production user does not have enough mysql permissions to create the triggers within mysql. To get around this we use a CiviCRM setting on production to log the sql to update the triggers to a file rather than live update them. We then check this file into our crm repo (sites/all/modules/wmf_civicrm/scripts/triggers.mysql currently) and fr-Ops run the file on live.
 
On development environments triggers are automatically updated in the database - which is generally easier - but to make your local output the triggers as live does the logging_no_trigger_permission setting can be enabled<syntaxhighlight lang="bash">
drush @wmff cvapi Setting.create logging_no_trigger_permission=1
</syntaxhighlight>Trigger generation needs to be done on production as the fields differ slightly on staging / dev environments. There are a few methods but turning logging off & back on generally generates consistent output. ie <syntaxhighlight lang="bash">
drush @wmff cvapi Setting.create logging=0
drush @wmff cvapi Setting.create logging=1
</syntaxhighlight>This will generate a file named something like CiviCRM.trigger62451ae5ab5a5mYm67702126718965e4a41105a08d6202e60.sql that will be in drupal/sites/default/files/civicrm/ConfigAndLog/ - copy this back to your home drive and scp it back to your local machine as sites/all/modules/wmf_civicrm/scripts/triggers.mysql. Changes to this files are committed, reviewed and deployed but they will not be 'live' until fr-tech-ops loads them so once deployed they need to be engages to run the latest triggers.mysql file
 
====Automated emails from CiviCRM====
We send out the following automated emails
 
*Recurring failure notifications - these are send when a monthly recurring email is failing and encourages people to set up a new one. It is not sent if they have an active recurring email.
*Thank you letters - these are send by an automated job for every donation in CiviCRM, unless the 'no_thankyou' field is populated or it is a recurring donations
*[[Fundraising/Internal Endpoints/End of year emails|End of year emails]] - these are sent at the start of the year to cover all the recurring contributions in the previous year. These can also be sent ad hoc to individual donors (in which case they include all donations, not just the recurring ones)
 
==C Libraries==
====SmashPig====
 
==D Service Providers==
 
==E Cluster layout, deployment, codebases ==
 
====[[Fundraising/Cluster/Deployments|Deployment]]====
 
====Staging server====
 
[[Fundraising/Cluster/Civicrm_staging_server|Civicrm Staging Server]]
 
==F Data and flow ==
 
===Payment processors===
We have the ability to use several payment processors for online donations.  Currently, we route most credit card donations to Ingenico (now called WorldLine).
====[[Fundraising/Data_and_flow/PSP_integrations/Ingenico|Ingenico]]====
*http://www.ingenico.com/
*Ingenico has the ability to handle payments from multiple international systems including: credit card, direct debit, real time bank transfer, eWallets and more. We're currently only using them for cards.
====[[Fundraising/Data_and_flow/PSP_integrations/PayPal_Express|PayPal]]====
*We get notified about payments via PayPal's IPN service (documentation: https://cms.paypal.com/cms_content/US/en_US/files/developer/IPNGuide.pdf)  The receiving endpoint is our [[Fundraising.wikimedia.org#PayPal_IPN_Listener|IPN listener]]
*Note: In general, note that the PayPal documentation tends to be incorrect, out of date, etc.
====[[Fundraising/Data_and_flow/PSP_integrations/Braintree|Braintree]]====
====[[Fundraising/Data_and_flow/PSP_integrations/Amazon|Amazon]]====
*A widget on our page, integrated using Login and Pay with Amazon.
====[[Fundraising/Data_and_flow/PSP_integrations/Adyen_Checkout|Adyen]] ====
*https://www.adyen.com/, [https://support.adyen.com/index.php?/Knowledgebase/List/Index/101/documentation documentation]
*Backup credit card processor in most countries, primary in a few. As of December 2021 Adyen is the only gateway we use to process iDEAL and Apple Pay transactions.
====[[Fundraising/Data_and_flow/PSP_integrations/dLocal|dLocal]]====
*https://dlocal.com/
*A payment processor specializing in the local payment methods of South America and India
 
See also, "[[foundation:Ways_to_Give|Ways to Give]]" for our recommended donation methods according to country.
 
Payment processor capabilities:
{| class="wikitable" cellpadding="5" |
|-
|-
! !!Ingenico!!PayPal!!Amazon!!Adyen
|Adyen
!D*Local
|https://ca-live.adyen.com/
!Banks
|https://ca-test.adyen.com/
!Checks
|Org email
|https://docs.adyen.com/development-resources/testing/test-card-numbers
|-
|-
|Credit card
|Google Pay
| bgcolor="#66ff66" |Yes
|https://pay.google.com/business/console/home
| bgcolor="#66ff66" |Yes
| bgcolor="#66ff66" |Yes
| bgcolor="#66ff66" |Yes
| bgcolor="#66ff66" |Yes
|
|
|Org email
|https://groups.google.com/g/googlepay-test-mode-stub-data
|-
|Apple Pay
|
|
|-
|Bank transfer
| bgcolor="#66ff66" |Yes
| bgcolor="#66ff66" |Yes
| bgcolor="gray" |No
| bgcolor="#66ff66" |Yes
| bgcolor="#66ff66" |Yes
|IBAN, Swift
|-
|Countries
|
|
|[https://www.paypal.com/worldwide/ list]
|FR-Tech SandBox [[Accounts]]
|USA<ref>https://payments.amazon.com/sdui/sdui/about?nodeId=73479#feat_countries</ref>
|
|
|Latam+IN
|-
|-
|Currencies
|Ingenico
|<ref>See [https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/globalcollect_gateway/globalcollect.adapter.php;b4b730ba30e90bf9685b8e5082cdfb8ecd1dd3dd$1642 GlobalCollectAdapter::getCurrencies]</ref>
|https://wpc.gcsip.com/
|[https://www.x.com/developers/paypal/documentation-tools/api/currency-codes list]
|USD
|All<ref>http://www.adyen.com/platform/all-countries-all-currencies/</ref>
|
|
|Org email
|[https://docs.google.com/spreadsheets/d/1JOhxAlKk4BdR-Eo75fuSfG0bUOuEgAHr0LI-XMxVjkY/edit#gid=1759048558 Ingenico Test Card (Google Sheets)]
|-
|-
| Direct debit
|DLocal
| bgcolor="#66ff66" |<ref>DE, IT, NL (todo: AT, BE, CH, ES, FR, GB)</ref>
|https://dashboard.dlocal.com/
| bgcolor="gray" |No
|https://dashboard.dlocal.com/
| bgcolor="gray" |No
|Org email
| bgcolor="#66ff66" |Yes
|Any card number sequence is accepted, error response is tested using description
|
|-
|PayPal
|https://www.paypal.com/signin
|https://www.sandbox.paypal.com/
|Config-Private
|Use FR tech account for testing
|-
|-
|Recurring
|Braintree
| bgcolor="#66ff66" |Yes
|https://www.braintreegateway.com/login
| bgcolor="#66ff66" | Yes
|https://sandbox.braintreegateway.com/login
| bgcolor="#66ff66" |Yes
|Config-Private
| bgcolor="#66ff66" |Yes
|Paypal - Use FR Tech Account for testing
| bgcolor="yellow" |n/i
|-
|-
| Mobile optimized
|Amazon
| bgcolor="gray" |No
|
| bgcolor="yellow" |n/i<ref>https://www.paypal.com/uk/webapps/mpp/sell-mobile</ref>
|
| bgcolor="yellow" | n/i<ref>https://payments.amazon.com/sdui/sdui/business?sn=devfps/mps</ref>
|
| bgcolor="yellow" |n/i
|
|
|-
|-
|Languages
|Acoustic
|<ref>See [https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/globalcollect_gateway/globalcollect.adapter.php;b4b730ba30e90bf9685b8e5082cdfb8ecd1dd3dd$1843 GlobalCollectAdapter::getAvailableLanguages].  Our code must find a fallback language if the donor's native tongue is unsupported.</ref>
|https://login-campaign-us-4.goacoustic.com/
| <ref>See [https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/paypal_gateway/paypal.adapter.php;b4b730ba30e90bf9685b8e5082cdfb8ecd1dd3dd$190 PaypalAdapter::stage_locale].  For unknown reasons, we have to specify language *by country*.</ref>
|
| ?
|
| ?
|
|
|-
|Donor needs account||No||Yes||Yes||No
|No
|-
|Refund by API
| bgcolor="yellow" |n/i
| bgcolor="yellow" | n/i
| bgcolor="yellow" |n/i
| bgcolor="yellow" |n/i
|n/i
|-
|Fully automated auditing
| bgcolor="#66ff66" |Yes
| bgcolor="gray" |Yes
| Yes
|Yes
| Yes
|}
|}


Legend:
==Cluster layout, deployment, codebases ==
{| class="wikitable"
|-
| bgcolor="#66ff66" |Yes||Implemented
|-
| bgcolor="yellow" |n/i||Not yet implemented
|-
| bgcolor="gray" |No ||Unsupported by processor
|}


====Notification failure policies:====
===[[Fundraising/Cluster/Deployments|Deployment]]===
When we don't respond to an IPN message from a payment processor with a successful HTTP code, they usually resend it.


Adyen: back-off algorithm from 5 minutes to 8 hrs, then every 8 hrs for a week
===Staging server===


Amazon: every hour for 14 days
[[Fundraising/Cluster/Civicrm_staging_server|Civicrm Staging Server]]


<references />
[[Fundraising/Cluster/Payments_staging_server|Payments Staging Server]]


===Email integration - Acoustic===
=== [[Fundraising/Cluster/Redis|Redis]] ===
Acoustic is the service we use to send out bulk emails. They are able to handle high volumes of emails and are responsible for managing server reputation to improve deliverability. Acoustic also provide tools for A-B testing to see which emails perform better. In order to be able to use our donor information from Acoustic we have a nightly upload job. We also re-import information from acoustic - for details go to  [[Fundraising/Data and Integrated Processes/Acoustic Integration]]


Note that prior brandings of the Acoustic platform may still linger - ie Silverpop, WCM, Watson Campaign Manager, or sometimes just 'IBM'
==[[Fundraising/Data and flow|Data and flow]] ==
===Message queues===
This describes the WMF fundraising systems configuration.  See the MediaWiki.org page on payments [[mw:Fundraising tech/Message queues|message queues]] for a discussion of how message queues are used to buffer and decouple fundraising infrastructure, and to read about the format and content of [[Fundraising/Normalized donation messages|normalized messages]].


WMF fundraising uses the [https://github.com/CoderKungfu/php-queue/ PHP-Queue] library to abstract queue access. In production we use Redis lists as queue storage This redis server is outside of PCI scope, and communicates with CiviCRM.
=== [[Fundraising/Data and flow/PSP integrations/Gateway Chooser|Gateway Chooser]] ===
* Routes donors to the correct payment processor based on the country, currency, and payment method.
* Incoming links from donatewiki and banners hit this special page on paymentswiki rather than going directly to individual gateways
** This allows us to disable a gateway on paymentswiki and have traffic flow immediately to the backup


Various [[Fundraising/Queue wrangling|queue wrangling techniques]] are available.
===[[Fundraising/Data_and_flow/PSP_integrations|Payment processors]]===
We have the ability to use several payment processors for online donations.  Currently, we route most credit card donations to Adyen.


''Redis''
*[[Fundraising/Data_and_flow/PSP_integrations/Adyen_Checkout|Adyen]] - Main credit card processor, also has iDEAL, ApplePay, and GooglePay


All queues feeding into services outside the fr-cluster live on a single Redis instance.  This is a SPOF.
*[[Fundraising/Data_and_flow/PSP_integrations/Amazon|Amazon]] - A widget on our page, integrated using Login and Pay with Amazon.


'''TODO'''
*[[Fundraising/Data_and_flow/PSP_integrations/Braintree|Braintree]]


We should clean up any unused queues, and overly narrowly defined ones.
*[[Fundraising/Data_and_flow/PSP_integrations/dLocal|dLocal]] - A payment processor specializing in the local payment methods of South America and India


===Contribution tracking===
* [[Fundraising/Data_and_flow/PSP_integrations/Ingenico|Ingenico]] - Ingenico has the ability to handle payments from multiple international systems including: credit card, direct debit, real time bank transfer, eWallets and more. We're currently only using them for cards.
{{see also|Fundraising/ContributionTracking}}
When a potential donor visits the Wikimedia donation page, a tracking record is created in the drupal.contribution_tracking table. This record includes the user's language, referrer, donation comment, opt-out status, a timestamp, and various other data. The tracking is handled on the MediaWiki side by the [[#DonationInterface|DonationInterface]] extension, which retrieves a contribution_tracking_id from a sequence generator in Redis. If the user makes a successful donation, a contribution record is passed to CiviCRM via the donations queue. The queue2civicrm module then inserts the contribution record into the CiviCRM database and updates the contribution_tracking record with the id given to the contribution by CiviCRM.


===Stats===
* [[Fundraising/Data_and_flow/PSP_integrations/PayPal_Express|PayPal]]
====Banner Impression/Landing Page Stats Collection====
Banner impressions and landing page stats are collected from the production proxies.  [[Fundraising_Analytics/Impression_Stats]].  The [[wmf:Thank_you]] page includes [[wmf:Template:Hide_banners]] which loads Special:HideBanners from multiple domains via image src.  HideBanners sets cookies for donors which tell CentralNotice's bannerController.js not to pester them for a year or so.


====utm_source====
See also, [https://donate.wikimedia.org/wiki/Ways_to_Give Ways to Give] for our recommended donation methods according to country.
This is a tracking variable which is supposed to collect information about the transaction.  Currently, it is a period-separated concatenation of three components.  One interpretation of the components is, 1) banner name, 2) landing page name, and 3) payment method.  We are currently in the process of standardizing (see [https://mingle.corp.wikimedia.org/projects/fundraiser_2012/cards/965 FR #965] and [https://mingle.corp.wikimedia.org/projects/fundraiser_2012/cards/673 FR #673]).


In theory, each component may be a tilde-concatenation of a sequence of landing pages, for example.  That code is badly dysfunctional.
===[[Fundraising/Data and flow/IPN listener|Instant Payment Notification (IPN) Listener]]===


====utm_medium====
===[[Fundraising/Data and Integrated Processes/Acoustic Integration|Email integration - Acoustic (Also known as Silverpop)]]===
Donor was referred by this type of site: sitenotice, spontaneous, sidebar, socialmedia.
Acoustic is the service we use to send out bulk emails. They are able to handle high volumes of emails and are responsible for managing server reputation to improve deliverability. Acoustic also provide tools for A-B testing to see which emails perform better. In order to be able to use our donor information from Acoustic we have a nightly upload job. We also re-import information from acoustic - for details go to  [[Fundraising/Data and Integrated Processes/Acoustic Integration]]


Seems unuseful at this broad granularity.
Note that prior brandings of the Acoustic platform may still linger - ie Silverpop, WCM, Watson Campaign Manager, or sometimes just 'IBM'


====utm_campaign====
===[[Fundraising/Data and flow/Queues|Message queues]]===
The parent campaign for the banner where this donation was initiated.
 
====utm_key ====
TODO
 
===Mediawiki extensions===
The following Mediawiki extensions related to fundraising are installed on the [https://payments.wikimedia.org payments wiki]:


====DonationInterface ====
===[[Fundraising/Data and flow/Stats pipeline|Stats Pipeline]]===
Renders donation forms and handles donor interaction, redirecting donors to payment processors when necessary and then either presenting an error or redirecting to a thank you page.


Extension documentation on [[mw:Extension:DonationInterface|mediawiki.org]]
===[[Fundraising/Data and flow/Recurring|Recurring]]===


=====Fraud filtering=====
===[[Fundraising/Data and flow/Monthly convert|Monthly Convert]]===
There are a series of extra filters, that perform analysis on credit card transactions to determine the likelihood that a transaction is fraudulent.  Each of the filters helps determine the 'risk score' for a transaction.  Actions to take based on certain risk scores can be configured per gateway (reject, review, challenge, accept).  The filters currently available include:
*MaxMind/MinFraud - a third party solution that helps analyze the transaction.  They return their own 'risk score' for a transaction which heavily influences our own internal scoring.
*Referrer - Regular expressions can be configured to be run on a transaction's 'referrer', and each regex can be configured to apply a different score in the event that the referrer is a match.
*utm_source - Same as referrer, but for the utm_source bit in the tracking fields.


====FundraisingEmailUnsubscribe====
===[[Fundraising/Data and flow/Audits|Audits]]===
Allows a donor to unsubscribe from fundraising-related emails.


Extension documentation on [[mw:Extension:FundraisingEmailUnsubscribe|mediawiki.org]]
=== [[Fundraising/Data and flow/Fraud filtering|Fraud filtering]] ===


===High-level Overview of Donation Pipeline===
==How we work (Team Processes)==
Click the images for further explanation.
Fundraising Tech is a [[:en:Scrum_(software_development)|Scrum]] team. We use [[:en:Phabricator|Phabricator]] to manage our backlog and we work in 2-week sprints.  
<gallery widths="320px" heights="340px">
File:Landing page to activemq.jpg|From the landing page to ActiveMQ
File:Activemq to civicrm.jpg|Inserting to queues in ActiveMQ to CiviCRM
</gallery>
 
*
===Miscellaneous Scripts===
There are some miscellaneous scripts to help with things like Paypal Verification, queue handling, etc.  Details of which can be found on [[Fundraising.wikimedia.org]].
 
===Translations===
''See [[Fundraising/Translation]] for more info''
 
*Donatewiki translations go out regularly on the l10n cache
*TYs need to be manually deployed - make a task for this and put it in pending review in the current sprint
*[https://translatewiki.net/w/i.php?title=Special:Translations&message=MediaWiki%3ADonate+interface-email-subject%2Fen Subject line] needs to be manually deployed - make a task for this and put it in pending review in the current sprint
*Payments needs to be manually deployed - make a task for this and put it in pending review in the current sprint
 
 
===Public reporting===
We export some extremely aggregated datasets at https://frdata.wikimedia.org/, generated every half hour by the public_data_export process-control job running the fundraiser_public_data_export and fundraiser_public_data_mover child jobs.
 
fundraiser_public_data_export runs the [[phab:diffusion/WFTO/browse/master/FundraiserStatisticsGen/fundstatgen.py|FundraiserStatisticsGen/fundstatgen.py script from the tools repo]], then fundraiser_public_data_mover just rsyncs the data to the frdata server.


==G How we work (Team Processes)==
* [[phab:dashboard/view/129/|Fundraising Tech Phabricator dashboard]]
* [[Fundraising/Team processes/Definition of Done|Fundraising Tech Definition of Done]]


===Fundraising Emergencies===
===Fundraising Emergencies===
Line 334: Line 170:
[https://office.wikimedia.org/wiki/Fundraising/Engineering/On_call Fundraising Engineering On-call documentation] is a quick-reference page for on-call duty.
[https://office.wikimedia.org/wiki/Fundraising/Engineering/On_call Fundraising Engineering On-call documentation] is a quick-reference page for on-call duty.


===Feature / Bug Trackers===
=== Feature / Bug Trackers===
There's loads of information about how fr-tech triages bugs at [[Fundraising/Bug Triaging]]
There's loads of information about how fr-tech triages bugs at [[Fundraising/Bug Triaging]]
Not sure what to do next? See [[phab:tag/fundraising/|Fundraising Tech's Phabricator Workboard]]
Not sure what to do next? See [[phab:tag/fundraising/|Fundraising Tech's Phabricator Workboard]]
Line 345: Line 181:
[[Category:Fundraising - Needs Updated]]
[[Category:Fundraising - Needs Updated]]


==H Development Tools==
==Development Tools==
=== Getting Started ===
===Getting Started===
Local setup for cluster SSH access
Local setup for cluster SSH access



Latest revision as of 21:15, 21 November 2022

WMF Donation Pipeline overview

This is the homepage for fundraising-tech documentation. If you can't find what you are looking from from here then take a look at our documentation plan and add the appropriate header & links.

Note that much of the content on this page should be moved to linked pages.

External-facing

Payments

Mediawiki extensions

The following Mediawiki extensions related to fundraising are installed on the payments wiki:

DonationInterface

Renders donation forms and handles donor interaction, redirecting donors to payment processors when necessary and then either presenting an error or redirecting to a thank you page.

Extension documentation on mediawiki.org

FundraisingEmailUnsubscribe

Allows a donor to unsubscribe from fundraising-related emails.

Extension documentation on mediawiki.org

Public reporting

We export some extremely aggregated datasets at https://frdata.wikimedia.org/, generated every half hour by the public_data_export process-control job running the fundraiser_public_data_export and fundraiser_public_data_mover child jobs.

fundraiser_public_data_export runs the FundraiserStatisticsGen/fundstatgen.py script from the tools repo, then fundraiser_public_data_mover just rsyncs the data to the frdata server.

legacy DonationForm - reusable frontend

Email Preference Center

Donor Data Delivery Tool

Internal-facing

CiviCRM

Libraries

SmashPig

Service Providers

Service Provider Name Production URL SandBox URL Sandbox Accounts Test Cards
Adyen https://ca-live.adyen.com/ https://ca-test.adyen.com/ Org email https://docs.adyen.com/development-resources/testing/test-card-numbers
Google Pay https://pay.google.com/business/console/home Org email https://groups.google.com/g/googlepay-test-mode-stub-data
Apple Pay FR-Tech SandBox Accounts
Ingenico https://wpc.gcsip.com/ Org email Ingenico Test Card (Google Sheets)
DLocal https://dashboard.dlocal.com/ https://dashboard.dlocal.com/ Org email Any card number sequence is accepted, error response is tested using description
PayPal https://www.paypal.com/signin https://www.sandbox.paypal.com/ Config-Private Use FR tech account for testing
Braintree https://www.braintreegateway.com/login https://sandbox.braintreegateway.com/login Config-Private Paypal - Use FR Tech Account for testing
Amazon
Acoustic https://login-campaign-us-4.goacoustic.com/

Cluster layout, deployment, codebases

Deployment

Staging server

Civicrm Staging Server

Payments Staging Server

Redis

Data and flow

Gateway Chooser

  • Routes donors to the correct payment processor based on the country, currency, and payment method.
  • Incoming links from donatewiki and banners hit this special page on paymentswiki rather than going directly to individual gateways
    • This allows us to disable a gateway on paymentswiki and have traffic flow immediately to the backup

Payment processors

We have the ability to use several payment processors for online donations. Currently, we route most credit card donations to Adyen.

  • Adyen - Main credit card processor, also has iDEAL, ApplePay, and GooglePay
  • Amazon - A widget on our page, integrated using Login and Pay with Amazon.
  • dLocal - A payment processor specializing in the local payment methods of South America and India
  • Ingenico - Ingenico has the ability to handle payments from multiple international systems including: credit card, direct debit, real time bank transfer, eWallets and more. We're currently only using them for cards.

See also, Ways to Give for our recommended donation methods according to country.

Instant Payment Notification (IPN) Listener

Email integration - Acoustic (Also known as Silverpop)

Acoustic is the service we use to send out bulk emails. They are able to handle high volumes of emails and are responsible for managing server reputation to improve deliverability. Acoustic also provide tools for A-B testing to see which emails perform better. In order to be able to use our donor information from Acoustic we have a nightly upload job. We also re-import information from acoustic - for details go to Fundraising/Data and Integrated Processes/Acoustic Integration

Note that prior brandings of the Acoustic platform may still linger - ie Silverpop, WCM, Watson Campaign Manager, or sometimes just 'IBM'

Message queues

Stats Pipeline

Recurring

Monthly Convert

Audits

Fraud filtering

How we work (Team Processes)

Fundraising Tech is a Scrum team. We use Phabricator to manage our backlog and we work in 2-week sprints.

Fundraising Emergencies

Fundraising Engineering Documentation has with system information and emergency response protocols. Or more specifically Shutting the pipeline down details how/when to disable banner campaigns or other fundraising/payment services.

Fundraising On-Call documentation

Fundraising Engineering On-call documentation is a quick-reference page for on-call duty.

Feature / Bug Trackers

There's loads of information about how fr-tech triages bugs at Fundraising/Bug Triaging Not sure what to do next? See Fundraising Tech's Phabricator Workboard

PCI Compliance

Payment Card Industry rules we have to follow to keep accepting credit cards.

Development Tools

Getting Started

Local setup for cluster SSH access

Docker

Testing locally

Gerrit

Gitlab

CI

Subpages: