You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Fundraising.wikimedia.org"

From Wikitech
Jump to navigation Jump to search
imported>Krinkle
imported>MarcoAurelio
(MarcoAurelio moved page Fundraising.wikimedia.org to Obsolete:Fundraising.wikimedia.org: per https://wikitech.wikimedia.org/?diff=1850682)
 
Line 1: Line 1:
Fundraising.wikimedia.org holds some miscellaneous fundraising tools and handles some of the payment processing for donations to the Foundation and runs on [[Grosley]].
+
#REDIRECT [[Obsolete:Fundraising.wikimedia.org]]
 
 
== PayPal IPN Listener ==
 
The PayPal IPN Listener is a stand alone script that allows us to validate PayPal transactions.  When a user makes a transaction for the Foundation through PayPal, PayPal POST's the transaction to our IPN Listener.  The listener pushes the message to a 'pending' queue on [[ActiveMQ]], then sends the message back to PayPal for verification of its authenticity. If the message is considered valid, it is then pushed to a 'verified' queue (likely the 'donations' queue) on [[ActiveMQ]].  If it is not, then the message is left on the pending queue for handling elsewhere.
 
 
 
=== Files ===
 
==== SVN ====
 
[http://svn.wikimedia.org/viewvc/wikimedia/trunk/fundraising-misc/queue_handling/paypal/IPN/ http://svn.wikimedia.org/viewvc/wikimedia/trunk/fundraising-misc/queue_handling/paypal/IPN/]
 
 
 
==== On Grosley ====
 
The PayPal IPN Listener lives in /opt/fundraising_misc/gateway_handling/paypal/IPN 
 
* IPNListener_Standalone.php: This file provides some configuration information and executes the business portion of PaypalIPNListener.php
 
* PaypalIPNListener.php: This contains a class with methods to communicate with ActiveMQ and Paypal, as well as to run through the logic of verifying payments from Paypal.
 
 
 
=== Logging ===
 
The listener logs output for the regular IPN listener to: <pre>/var/log/fundraising/paypal_ipn/out.log</pre>  The recurring IPN listener logs to: <pre>/var/log/fundraising/paypal_ipn/recurring.log</pre>
 
 
 
These logs are rotated daily and will stick around for 7 days.  When the log files are first rotated, a compressed copy is sent to storage3 for archival purposes.  They are stored on storage3 in: <pre>/archive/fundraising/logs/paypal_ipn/out.<rotate_date>.gz</pre>Wwhere <rotate date> is the date the logs were rotated in the format YYYYMMDD_HHiiss (in UTC).
 
 
 
=== Access ===
 
The listener is accessible to PayPal via HTTP at http://fundraising.wikimedia.org/IPNListener_Standalone.php
 
 
 
=== Testing ===
 
If you sign up for a PayPal sandbox account, you can send test IPN messages to the listener:
 
# Go to https://developer.paypal.com/
 
# Log in
 
# Under 'Test Tools' click on 'Instant Payment Notification (IPN) simulator'
 
# Enter the listner url (http://fundraising.wikimedia.org/IPNListener_Standalone.php)
 
# Select the transaction type (should test completed and declined) and fire it off
 
# See what happens in the queue and check the logs
 
 
 
=== Further documentation ===
 
* [https://cms.paypal.com/cms_content/US/en_US/files/developer/PP_WebsitePaymentsStandard_IntegrationGuide.pdf PayPal Website Payments Standard Integration]
 
* [https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/e_howto_admin_IPNIntro PayPal IPN documenation]
 
 
 
== CiviCRM Global Collect Recurring Payment Processor ==
 
 
 
This recurring payment processor relies on the [[Fundraising.wikimedia.org#StandaloneGlobalCollectAdapter|StandaloneGlobalCollectAdapter]] to initiate transactions with Global Collect.
 
 
 
=== See the code ===
 
 
 
This module is included with [http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-civicrm/ Drupal CiviCRM]:
 
 
 
http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-civicrm/sites/all/modules/civicrm_recurring_globalcollect
 
 
 
=== Configuration ===
 
 
 
== StandaloneGlobalCollectAdapter ==
 
 
 
You will need to copy the adapter to the library.
 
 
 
Go to the root of your Drupal CiviCRM installation:
 
 
 
<pre>
 
cd sites/all/modules/civicrm_recurring_globalcollect/library
 
 
 
mkdir StandaloneGlobalCollectAdapter
 
 
 
cd StandaloneGlobalCollectAdapter
 
 
 
svn co http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-misc/standalone/StandaloneGlobalCollectAdapter/ .
 
 
 
cd ..
 
 
 
ln -sn StandaloneGlobalCollectAdapter/StandaloneGlobalCollectAdapter.php
 
</pre>
 
 
 
== CiviCRM Recurring GlobalCollect Processor ==
 
 
 
=== Installation ===
 
 
 
Install the module in the Drupal admin interface.
 
 
 
==== Permissions ====
 
 
 
Home > Administer > User management > Permissions
 
 
 
Look for the section civicrm_recurring_globalcollect module
 
 
 
And enable for administrators and any other groups you deem necessary.
 
 
 
<strong style="color: red;">Remember, this is potentially dealing with donors personal financial accounts.</strong>
 
 
 
==== Configuration ====
 
 
 
Set the urls for using Global Collect or for testing.
 
 
 
Home > Administer > Site configuration > CiviCRM Recurring GlobalCollect Processor
 
 
 
If you leave the batch size at 0, the module will not process anything.
 
 
 
=== Drush ===
 
 
 
Drush has a [http://drupal.org/node/704848 bug] that needs to be patched:
 
 
 
http://drupal.org/files/issues/704848-drush-run-twice.patch
 
 
 
before you run this  command:
 
 
 
Get help from this Drupal module with Drush:
 
 
 
<pre>
 
drush help civicrm-recurring-globalcollect
 
</pre>
 
or
 
<pre>
 
drush help crg
 
</pre>
 
 
 
Your options:
 
 
 
<pre>
 
Process recurring payments to GlobalCollect
 
 
 
Examples:
 
drush civicrm-recurring-globalcollect    # Process recurring payments                                                 
 
drush crg --url true                      # TESTING: Always return a successful response from Global Collect           
 
drush crg --url false                    # TESTING: Always return a failed response from Global Collect (404)         
 
drush crg --url 404                      # TESTING: Always return a 404 response from Global Collect                   
 
drush crg --url 504                      # TESTING: Always return a 504 response from Global Collect                   
 
drush crg --url 404.php                  # TESTING: Send the request to a url relative to to tests/resources/responses/
 
drush crg --url "https://example.org/"    # TESTING: Send the request to a full url.                                   
 
 
 
Options:
 
--url                                    Allowed parameters: true | false | 404 | 504 | some-relative-url | https://example.org
 
 
 
Aliases: crg
 
</pre>
 
 
 
=== Payments ===
 
 
 
You can customize this processor:
 
 
 
Go to: Home > Administer > Site Configuration > CiviCRM Recurring GlobalCollect Processor
 
 
 
You have ability to set:
 
* Global Collect Url
 
* Default Testing Url
 
* StandaloneGlobalCollectAdapter Path
 
* Email Notification Address
 
* Global Collect Merchant ID
 
* Failure retry rate
 
* Catch up missed days
 
 
 
==== Successful Payments ====
 
 
 
Successful payments will be rescheduled for the following month.
 
 
 
==== Failed Payments ====
 
 
 
Failed payments will be cancelled on the third failure.
 
 
 
Each failure reschedules the payment for the following day, by default. This can be customized in CiviCRM: Failure retry rate
 
 
 
== StandaloneGlobalCollectAdapter ==
 
 
 
== Recurring Payments IPN Listener ==
 
The Recurring Payments IPN Listener is very similar to the PayPal IPN listener.  In fact, it exists as a child class of the PayPal listener.
 
 
 
At the moment, all it does differently from the regular PayPal listener is that it ensures that the transaction type is something relating to subscriptions (recurring payments), verifies the message against PayPal, and then queues the entire POST object from PayPal in a special recurring payments queue on [[ActiveMQ]].  We store the messages in the queue for now because we currently have no sane way to bring the transactional information into CiviCRM.
 
 
 
=== Files ===
 
==== SVN ====
 
[http://svn.wikimedia.org/viewvc/wikimedia/trunk/fundraising-misc/queue_handling/paypal/IPN/ http://svn.wikimedia.org/viewvc/wikimedia/trunk/fundraising-misc/queue_handling/paypal/IPN/]
 
 
 
==== On Grosley ====
 
The Recurring Payments IPN Listener lives in /opt/fundraising_misc/gateway_handling/paypal/IPN 
 
* IPNListener_Recurring.php: This file provides some configuration information and executes the business portion of PaypalRecurringIPNListener.php
 
* PaypalRecurringIPNListener.php: This contains a class with methods to communicate with ActiveMQ and Paypal, as well as to run through the logic of verifying payments from Paypal.
 
 
 
=== Logging ===
 
The recurring IPN listener logs to: <pre>/var/log/fundraising/paypal_ipn/recurring.log</pre>
 
 
 
These logs are rotated daily and will stick around for 7 days.  When the log files are first rotated, a compressed copy is sent to storage3 for archival purposes.  They are stored on storage3 in: <pre>/archive/fundraising/logs/paypal_ipn/recurring_out.<rotate_date>.gz</pre>
 
Where <rotate date> is the date the logs were rotated in the format YYYYMMDD_HHiiss (in UTC).
 
 
 
=== Access ===
 
The listener is accessible to PayPal via HTTP at http://fundraising.wikimedia.org/IPNListener_Recurring.php
 
 
 
== PayflowPro Pending Transaction Verification ==
 
The PayflowPro pending transaction verification script pulls transactions off of the ActiveMQ 'pending' queue for PayflowPro, and then attempts to verify those transactions with PayflowPro.  If a transaction is verified, it is then pushed onto the confirmed 'donations' queue.  If it fails with an error code of 126 ("Fraud Protection Services Filter — Flagged for review by filters") or 26 ("Invalid vendor account"), the message is left alone on the pending queue until the issue is resolved.  If it fails with any other error code, the message is discarded from the vendor queue.
 
 
 
For more information about the PayflowPro side of things, check out the PFP documentation: https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_gateway_payflowpro
 
In particular, I recommend the 'Developer Guide' PDF.
 
 
 
=== Files ===
 
The files that power this are located in /opt/fundraising_misc/gateway_handling/payflowpro
 
* stompPFPPendingProcessorSA.php - Contains the class StompPFPPendingProcessorSA which has the methods necessary to communicate with PayflowPro as well as ActiveMQ, and the logic to route transactions correctly.
 
* executeStompPFPPendingProcessorSA.php - configuration and execution for the processor.  Configuration should be fairly self-explanatory.  Of note, you can specify log location, batch size, queues to use, etc.
 
 
 
=== Logging ===
 
Output from the script is logged to: <pre>/var/log/fundraising/pfp_pending_processing/out.log</pre>
 
 
 
These logs are rotated daily and will stick around for 7 days.  When the log files are first rotated, a compressed copy is sent to storage3 for archival purposes.  They are stored on storage3 in: <pre>/archive/fundraising/logs/pfp_pending_processor/out.<rotate_date>.gz</pre>
 
Where <rotate date> is the date the logs were rotated in the format YYYYMMDD_HHiiss (in UTC).
 
 
 
=== Access ===
 
There is no web access to this script.  It currently gets periodically executed by a [[Grosley#Hudson|Hudson]] job.
 
 
 
=== Testing ===
 
There is a donation set up that mimics the credit card contribution page on payments.wikimedia.org, but passes everything through as tests (see [[#Credit_Card_donation_Test_Interface]] for more info).  In order to trigger a donation that should be sent to the 'pending' queue (rather than going directly to the 'donations' queue), set a donation value for over $9999.
 
 
 
# Go to http://fundraising.wikimedia.org/test/mediawiki/index.php/Donate (only works inside of the WMF office a la payments.wikimedia.org)
 
# Fill out the form, be sure to set payment method to 'Credit Card'.  Submit the form.
 
# Use some fake credit card information - be sure to get the appropriate credit card #s to use from the PayflowPro developer documentation (http://tinyurl.com/352axwu) - look on page 49.
 
# The transaction should then be logged in the 'test_pending_pfp' queue (see [[ActiveMQ]] to find out how to check).
 
# You can either manually execute the pending processor or wait for the Hudson job to run - to execute manually, just run php /path/to/executeStompPFPPendingProcessorSA.php
 
# You should then see the transaction pushed over to the 'test_donations' queue if it was a verifiable transaction.
 
 
 
== Jenkins ==
 
We are using [http://jenkins-ci.org/ Jenkins] as a cron-replacement for handling some of the automation of certain fundraising tasks. (Note that we were previously using Hudson, so some older documentation and code may refer to Hudson instead, although they are basically the same software.)
 
 
 
=== Why ===
 
Jenkins is like cron on 'roids - it offers many improvements over cron.  The folks at FourKitchens turned us onto it when we were preparing for the fundraising infrastructure improvements in July 2010.  They made a blog post extolling its virtues, so rather than repeat them, here's a link: http://fourkitchens.com/blog/2010/05/09/drop-cron-use-hudson-instead
 
 
 
The biggest advantages for our purposes are:
 
* Jenkins sends us an email when a 'build' (a singular job execution) fails, and then again when there's a successful build following a failure.
 
* Jenkins logs console output of the command it executes, and it can be reviewed through the Jenkins interface
 
* Jenkins keeps track of how long it takes for builds to be executed
 
 
 
=== What we use it for ===
 
Jenkins is currently being used to replace all the Drupal/CiviCRM functionality that we would normally use cron for.  The following is a list of 'jobs' that Jenkins manages:
 
* Civicrm.wikimedia.org cron run - This executes the Drupal/CiviCRM cron.php script, which performs some automated tasks and periodic site maintenance (clearing caches, etc)
 
* CiviMail Receive
 
* CiviMail Send
 
* Civicrm-dev.wikimedia.org cron run
 
* dev CiviMail receive
 
* dev CiviMail send
 
* Donations queue consume - executes the [[Fundraising#queue2civicrm|Queue2CiviCRM]] drush command which triggers queue consumption
 
* PayflowPro Queue Processing - executes the [[Fundraising.wikimedia.org#PayflowPro_Pending_Transaction_Verification|PayflowPro pending transaction verification script]] which verifies transactions in the pfp_pending queue.
 
 
 
=== Access ===
 
The Jenkins instance is available on port 8080 of fundraising.wikimedia.org.  For an account, contact Arthur.
 
 
 
=== Issues ===
 
See [[Grosley#Potential_issues]]
 
 
 
== Drush ==
 
For our Drupal/CiviCRM setup, we take advantage of [http://drupal.org/project/drush Drush] - a shell utility for Drupal.  It facilitates performing certain Drupal tasks at the command line, which you would otherwise need to do through the Drupal web interface.  It is possible to extend Drush by writing your own Drush script, which can give command-line functionality to your custom Drupal modules.
 
 
 
=== Files ===
 
We are currently using an antiquated version of Drush (v 3.x).
 
 
 
Drush is currently installed in /opt/drush, and the actual Drush utility is sym-linked to /usr/local/bin/drush.  In order to use it, you can either run:
 
<pre>
 
$ drush -r <path to Drupal installation> --help
 
</pre>
 
or
 
<pre>
 
$ cd /path/to/Drupal
 
$ drush --help
 
</pre>
 
The '--help' option will print out further usage instructions.
 
 
 
=== CiviCRM ===
 
We are using a development version of the CiviCRM Drush extension (details [hhttp://civicrm.org/blogs/deepak/civicrm-and-drush-integration here]).  This gives us some command line access to CiviCRM.  While not actually necessary for any of the automated tasks we use Drush for, it makes performing some CiviCRM maintenance (eg clearing caches) and database actions possible to do from the command line.
 
 
 
=== Queue2CiviCRM ===
 
The [[Fundraising#queue2civicrm|Queue2CiviCRM]] module takes advantage of Drush.  A Drush extension is packaged with the module (in queue2civicrm/queue_consume.drush.inc).  This extension add a Drush command which will execute queue2civicrm - causing queue2civicrm to consume donations from the queue in batches, as configured in the Drupal interface.  To execute queue2civicrm using Drush from the command line, run the following from within your Drupal directory:
 
<pre>
 
$ drush -v --user=1 queue-consume
 
</pre>
 
'queue-consume' can also be replaced with its command alias, 'qc'.  'The '-v' adds a level of verbosity to the output of the command (which is useful - we use this to capture output in Hudson during queue consumption to help us identify where something went wrong in the event that things explode).  '--user=1' causes Drupal to be bootstrapped as the admin user, which (based on the user permissions scheme in Drupal) is necessary for inserting contributions and contacts into CiviCRM.
 
 
 
{{lowercase}}
 
[[Category:Services]]
 
[[Category:Fundraising]]
 
[[Category:Fundraising - Needs Updated]]
 

Latest revision as of 19:31, 14 January 2020