You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Fundraising/Team processes/Onboarding

From Wikitech-static
< Fundraising‎ | Team processes
Revision as of 21:57, 10 March 2022 by imported>Wfan (→‎Day 6)
Jump to navigation Jump to search

Accounts & Privileges to get

Prep

  • Write your name inside the team contact list with phone number and email address, this will add you to Gerrit as project's git reviewer.
  • Get IRC join team #wikimedia-fundraising and #xxxx(ask your team member). Choose your own irc connection methods, limechat or irccloud or online works
  • Manage gmail filters
  • Add a team calendar for online meetings.
  • Know more about FR team & Fundraising, with lots links for payments and test links
  • Install your own IDE, JetBrains tool is free with wiki email if you use phpStorm.
  • Join Asana with gmail and join team “online fundraising” by searching coworker’s email.
  • Check Fr-tech priorities meeting notes for each spring's task overview

Set up local Env

  1. Refer to the Readme, get all the code we are working on in local env
  2. install git-review for peer review
  3. use Zuul to check code reveiw by Jenkins.
  • Database
  1. use any GUI you like, with the Docker up, get all the code and the database, Redis and so on based on containers
  2. if use the MysqlWorkBench, set useSSL=0 to advance others to enable no ssl for standard TCP/IP.
  • SSH
  1. Yubikey will be requested by your manger.
  2. With the Yubikey set up, able to go to the dev server with ssh access,
  3. To set up your Yubikey, you needs to be corporate with Dallas
  4. With ssh access, you can perform deploy(see below, ask peers to do it together at your first time.
  1. We have three projects (Smashpig, payments, civicrm) which smashipig is working for both payments and civicrm.
  2. Whenever we has some package version updates, need to update the correspondent package version for donor interface and smashpig
  3. the access could be granted by Elliott

Some account might not need at this moment but good to have later

Day 1

  • Participate the Onboarding meeting and set up some account
  • Get to know the team and set up some 1:1 with your onboard buddy
  • Get accesses per above list

Day 2

What is Gerrit and get to know code in general
  1. Register Gerrit and get your username, use it to download code from Gerrit and run setup.sh with Gerrit username as reviewer name to set docker up;
  2. Ask files under config-private, which will have two folder: payments contains local config and smashpig contains all the payment methods config
  3. All the config files can be found from server
    yourname@frdev1001:/srv/www/org/wikimedia/listeners/SmashPig/local-config$ ls -a
    
    set up your local env and start to get to know the code

Day 3

Get familiar with the payment process
  • After install docker successfully, go to https://localhost:9001 (if not able to open localhost in https with Chrome, enable this chrome://flags/#allow-insecure-localhost
  • Make test payment with adyen or ingenico test card, it will perform auth first, in local env, we need to manually check payment from Redis, then use queue testing doc to consume queue to db to actually capture payments.
./queues-redis-cli.sh  (this will log you in redis)
Keys * (this will show all the keys)
lrange donations 0 -1 (this will show all the jobs in donations queue)
cd <roor dir> (go back to root dir)
./civicrm-drush.show (this will logi you in civi)
drush --user-1 -v -r /srv/civi-sites/wmff/drupal qc (run queue consume to exec the queue to db)
docker@civicrm:/srv/civi-sites/wmff/drupal$ drush --user=1 -v afqc (this is consume fraud contributions)

After consume the queue, we will be able to see the transaction from

  1. CiviCrm
  2. CiviCrm database
  3. payment method's back office with payment details.
How to monitor production log
ssh frlog1001
cd /var/log/remote
tail -f payments-ingenico (Check out how many ingenico payments)
tail -f payments-listener-smashpig (Check out payments in general which smashpig handle auth)
grep <keyword> (to find the keyword from log)
What is Irc
  • Register your nickname
  • Set up cloak
Add google calendar and know agile meetings
  • Daily Autopsy (to check if there is any blocker)
  • Backlog Refinement (repeat every week to clear the backlog)
  • Spring Review, retro and planing ( Repeat every 2 weeks, to plan what should we do in next spring)
    • What is good this spring and what could be better with retro notes or miro dashboard
    • Review if last spring been cleared and are those tasks
    • Estimate story point for last spring's done issues
    • Vote for the name of next spring
    • Adding issues for next spring from triage
  • Talk Talk (Solve blocker together or sync up recent issues, one repeat every week and one repeat every two weeks)
  • Analytics sync up (repeat every 2 weeks to make sure we are sync in each Quarters)
  • Civi Fortnightly (Talk about Civi with whole advancement team, repeat every 2 weeks)
  • Alerts Retro (text based meeting to look at recent Failmail and alerts, repeat every 2 weeks)
  • some 1:1 meetings

Day 4

Get familiar with local test process:
  • A: queue’s testing
  • B: queue Message
  • c: Make payment with Redis queue monitor to observe all the transaction steps
    cd <rootdir>
    ./queues-rediscli.sh MONITOR
    
    If smashpig has too many pending transactions, which might due to fraud, then clear them with this cmd,
    docker-compose exec smashpig bash (go inside docker server first)
    cd /srv/smashpig/  (go to the right dir )
    php Maintenance/ConsumePendingQueue.php (this put all fraud into pending table first)
    php Maintenance/DeleteExpiredPendingMessages.php --gateway=ingenico --days=1
    
    To clear the pending transactions with a certain gateway and exist for more than 1 days
  • clear drush cache and system flush to clear any cache, then run the PendingTable.consume also in local source code open xdebug.ini inside config civicrm, update xdebug.remote_autostart=on then start debug.
    docker@civicrm:/srv/civi-sites/wmff/drupal$ drush cc
    docker@civicrm:/srv/civi-sites/wmff/drupal$ drush cvapi System.flush
    docker@civicrm:/srv/civi-sites/wmff/drupal$ drush cvapi PendingTable.consume version=4 gateway=ingenico
    

Day 5

Understand each dev test URLs
  • Payments URL: https://localhost:9001       payments forms
  • Payments http URL: http://localhost:9009      same as above, but over http (used to receive requests via the routable URL)
  • Payments test routable URL: https://paymentstest1.wmcloud.org (see README.md)   For Apple Pay testing
  • WMF CiviCRM install URL: https://wmff.localhost:32353/civicrm CiviCRM with all of our drupal modules and custom Civi modules
  • Generic CiviCRM install (based on upstream master) URL: https://dmaster.localhost:32353/civicrm  Core CiviCRM
  • Civicrm user/password: admin/admin
  • Mailcatcher - mails sent from CiviCRM core code: http://localhost:1080  For civi, email display like the EOY
  • CiviProxy URL: https://localhost:9005    do not need to worry about this now
  • SmashPig IPN listener routable URL: https://paymentsipntest1.wmcloud.org (see README.md) test adyen apple pay
  • E-mail Preference Center URL: https://localhost:9002/index.php/Special:EmailPreferences    not working on that
  • PrivateBin read-only URL: https://localhost:9007         fancy page for contributor to see their activities
  • PrivateBin read-write URL: https://localhost:9008        donor activities with write access
Git review
  • install git-review
  • submitting changes with git-review
    • Git pull from master
    • Git checkout -b <newbrach>
    • Make modification to the code, then use git status to check which code is been modified and will we add it or use ui to add certain change files
    • Git add <right paths of the code file>
    • Git commit
      WIP:<comment>
      
      <more desc for this change>
      
      Bug: <Gerrit Tag Name>
      if not satisfied, then
    • Git commit –amend (if your want to update exit with :wq)
    • Git review (this part submit the code for review and auto run by Jenkins with +2  if did not break anything, then all the commit will automatically linked with Gerrit and Phabricator).
    • Note: you can always reply "recheck" to your commit to let jenkins run the check again
  • Modify you commit
    • Beside the code you commit, you will find a button as "UPDATE CHANGE"
    • Click that one, then you can find the 4 steps instruction as
      1. Checkout this change locally and make your desired modifications to the files: `git fetch https://gerrit.wikimedia.org/r/xxxxxx && git checkout FETCH_HEAD
      2. Update the local commit with your modifications using the following command: `git add . && git commit --amend --no-edit` (Leave the "Change-Id:" line of the commit message as is)
      3. Push the updated commit to Gerrit: `git push origin HEAD:refs/for/master`
      4. Refresh this page to view the the update.
  • Review other’s commits
    • Go to the Gerrit dashboard and find the patches you want to review
    • git-review -d <tagName> to checkout that branch or from the Gerrit ui, click download, use anonymous http to checkout the new branch or just checkout to the patch branch.
    • If everything is ok, leave a comment below of the Phabricator task and then use Zuul to run reviewed code with Jenkins. If passed, then +2 this code is ready to move to production.
    • During debug, could use export PHP_IDE_CONFIG="serverName=civi"to enable xDebug for phpStorm (where depending on your IDE you may wish to set the environment variable PHP_IDE_CONFIG.For phpstorm it should hold a value like "serverName=wmff" - where wmff is configured as a deployment server in phpStorm. If you are using drush then the site alias will set this variable - e.g drush @wmff will run drush in the context of the wmff.localhost site, including the PHP_IDE_CONFIG.serverName key)

Day 6

Update package at Packagist
  • 1: Login to root dir where you have docker code, fundraising handle three main repos:

donation-interface

cd src/payments/extensions/DonationInterface
git remote -v   (if return https then convert to ssh)
git remote set-url origin ssh://<reviwer>@gerrit.wikimedia.org:29418/mediawiki/extensions/DonationInterface
git pull (run git remote -v again to check if remote set to ssh then pull the newest code, handle all the payment methods)

smash-pig

cd ~/<rootdir>/src/smashpig
git pull  (to get the most updated version of smaship which handle both civicrm -when donor is doing recurring payment and payments flow)

CiviCRM

cd ~/<rootdir>/src/civi-sites/wmff
git pull  (get the most updated version of civicrm which manage the customer backoffice)
git remote

Upgrade drupal 8 to 9 in year 2020, and in the future remove drupal just save civi

In wmff > drupal > sites>  all  (those are the old drupal module)

Git remote should be /crm

  • 2: update tag to Packagist if necessary: (after pull most updated code from master, we have two packages which are wikimedia/smash-pig and wikimedia/donation-interface)
    1. git tag (to check current tag)
    2. git tag v.x.x.x.x (to mark current repo to this tag version)
    3. git push --tags (to push new tag version)
    4. Go to Packagist, login, find the package, then hit update
  • 3: update packages then update it in code:
    1. docker-compose exec civicrm bash  (go to local civicrm server)
    2. cd src/civi-sites/wmff   (go to wmff repo)
    3. composer self-update (if necessary)
    4. composer update wikimedia/donation-interface wikimedia/smash-pig (must update them at the same time)
    5. Update the new tag version number in DonationInterface composer.json

Day 7

Understand redis queue
./queues-redis-cli.sh (login to redis)
Keys * (check all the keys in redis)

1: contribution-tracking  -see under current session, actual payment tried how many times with detail

2: payments-init -payment in pending before it finished

3: sequence_contribution-tracking  -hold the number of contribution tired

4: payments-antifraud  -contain all fraud transactions

5: pending -not completed

6: donations  - all complete, user redirect to thank you page (this is the donation we just made in json)

you can either use Redis desktop manager to view key value pairs or just use lrange <keyName> 0 -1 to view value with keyName.

Day 8

Lint for JS
  • Payment DonationInterface uses the mediawiki rules:
    • Under /src/payment/extension/DonationInterface Run `npm install -D stylelint-config-wikimedia`, then you will find your esLint enabled
  • CiviCrm is using civilint rule, also here is the doc civi has for phpStorm config.
Code style for PHP
XDEBUG for phpStorm
  • Go to Preferences -> PHP -> Servers, add three servers with
  1. payment-wiki:
    • path mapping: /src/payment <--> /var/www/html;
    • host: localhost;
    • port: 9001;
    • Debuger: Xdebug.
  2. civi:
    • path mapping: /src/citi-sites <--> /srv/citi-sites;
    • host: wmff.localhost;
    • port: 32353;
    • Debuger: Xdebug.
  3. smashpig:
    • path mapping: /src/smashpig <--> /srv/smashpig;
    • host: localhost;
    • port: 9006;
    • Debuger: Xdebug.
  • Edit configuration -> add PHP Remote Debug -> check Filter debug connection by IDE key
    • Payment-wiki:
      • Server: payment-wiki
      • IDE Key: PHPSTORM
    • Civi:
      • Server: civi
      • IDE Key: PHPSTORM
    • Smashpig:
      • Server: smashpit
      • IDE Key: PHPSTORM
  • In addition to the above, you would have to update the Xdebug web on payments and civicrm because there's a slight issue with docker on mac that affects Xdebug. Paste the following lines of code in <rootdir>/config/payments/xdebug-web.ini & <rootdir>/config/smashpig/xdebug-cli.ini xdebug.remote_autostart=on xdebug.remote_host=docker.for.mac.localhost
    xdebug.remote_autostart=on
    xdebug.remote_host=docker.for.mac.localhost
    
    In <rootdir>/config/civicrm/xdebug-cli.ini
    xdebug.remote_autostart=off
    xdebug.remote_host=docker.for.mac.localhost
    
    then restart docker’s smashpig and payments container to enable the change
  • Test payment-wiki
    1. add a breakpoint at /src/payments/extensions/DonationInterface/gateway_forms/Mustache.php setGateWay function
    2. Then select Payment-wiki as configuration
    3. click on the green bugs
    4. open localhost:9001 then click any payment links, you will see your xdebug is working
  • Test civi
    1. open phpstorm preferences -> php -> debug -> break at first line in php scripts
    2. add breakpoint at src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/org.wikimedia.smashpig/CRM/Core/Payment/SmashPigRecurringProcessor.php
    3. go to the civicrm docker container (./civicrm-drush.sh)
    4. run
      docker@civicrm:/srv/civi-sites/wmff/drupal/sites/default/civicrm/extensions$ drush --user=1 -v -r /srv/civi-sites/wmff/drupal qc
      
      You might get the error saying that you should have the php_ide_config set up
    5. then inside the civicrm docker, run
      export PHP_IDE_CONFIG="serverName=civi"
      
      the serverName will be the one you have set up at phpStorm server, the one have right path mapping
    6. skip the first break point, then it will work
  • Test smashpig
  1. set a breakpoint at src/civi-sites/wmff/vendor/wikimedia/smash-pig/Maintenance/ConsumePendingQueue.php execute function's new PendingQueueConsumer.
  2. inside the smashpig container, to go /srv/smashpig/Maintenance
    export PHP_IDE_CONFIG="serverName=smashpig"
    
    then
    docker@smashpig:/srv/smashpig/Maintenance$ php ConsumePendingQueue.php
    

Day 9

Failmail

Understand the error messages, permission will be added by Jeff,

e.g. Check forget me processing failed with code 1

  • After check Acoustic > data -> contact list > view's each db with the email you saw from civi forget me, if no matches then just delete it.
  • Use this to can delete where id is equal to the most recent id as above xxxx
  • Go this page, hit execute, we should see a large number id in result.

Day 10

Deployment
  • everything you need will be here
  • Otherwise just ask your buddy to do that with you for the first time, and it’s always good to have someone else's recent deployment from Gerrit to use as ref.
  • After Jenkins gives you all the good feedback,  you can use this link
  1. Log in to the active fundraising deploy server frpm1001
    ssh frpm1001
    
  2. Run fundraising_code_update to refresh the deploy server's staging area
    fundraising_code_update -p civicrm (if payment use payment-wiki)
    
  3. Confirm the reported commit changes match what you expect, you can inspect the contents of the project staging directory which is listed by
    fundraising_code_update -i
    
  4. Run rsync_blaster to propagate this staged code to the individual servers
    rsync_blaster ALL:civicrm (if payment use payment-wiki)
    
  5. Log action into #wikimedia-fundraising by typing the below (where xyz & abc are the commits) into IRC !log Updating civicrm from xyz to abc (if payment use payment-wiki)
  6. Turn your stuff back on and make sure it works.

Note:

  • We do civicrm upgrade if only the core got upgraded, and all you need to know is this link
  • For the customized civicrm upgrade, we need to have the submodule update as well
  • For payment-wiki you just need to follow the deployment link above and replace the civicrm to payment-wiki
  • For fraud filter deployment, since it's private files, not in Gerrit, so just do directly to frpm1001. Nano the localSettings, update it and deploy it, once you are done, test a real world donation and see from the log if everything is going well.
  • Deploy to staging site: Fundraising/Cluster/Civicrm staging server

Day 11

EOY Email

(Usually Elliot will run the cron job, rest of the team will watch the whole FailMail):

To update these numbers (safely because this CiviCrm is the slave db):

  • Run `ssh frdev1001`
  • Type `mysql`
  • Run `use civicrm`

then

SELECT year, status, COUNT(*) FROM wmf_eoy_receipt_donor GROUP BY year , status;

To see them live (using prod mysql access: it's in the .my.cnf file in your home directory)

  • Run `ssh civi1001`
  • Change directory to `/srv/org.wikimedia.civicrm/drupal`
  • Run `drush civicrm-sql-cli`

then

SELECT year, status, COUNT(*) FROM wmf_eoy_receipt_donor GROUP BY year , status;

To preview an EOY email locally:

admin@admins-MBP ~ % cd ~/wiki
admin@admins-MBP wiki % ./civicrm-drush.sh

Render email to preview

docker@civicrm:/srv/civi-sites/wmff/drupal$ drush @wmff cvapi EOYEmail.Render version=4 limit=1 year=2022 contactID=210

Then

docker@civicrm:/srv/civi-sites/wmff/drupal$ drush @wmff cvapi EOYEmail.Send version=4 limit=1 year=2022 contactID=210

Go then go to localhost:1080 (MailCatcher) we will see the test mail.

Day 12

FraudFilter

Refer to this Gerrit Ticket

Update the policy inside frpm1001:/localsettings/payment-wiki/LocalSettings.php, and do the deploy:

fundraising_code_update -p payment-wiki

then

rsync_blaster ALL:payment-wiki

then go to the frlog1001:/var/log/remote to check the corespondent payment method log, then always try to make a real donation and check the log to make sure nothing is been break due to the new policy we add.

Since this is private file, which we do not have code publish to Gerrit, so there is no way to do the git review process, just ask you peer to do that with you if you are not very sure about the whole process.

Day 13

Emergency Protocol / Incident Response

Day 14

Email Preference
  • local test site: https://localhost:9002/index.php/Special:EmailPreferences/emailPreferences?contact_hash=2261039855&contact_id=6
  • local test with json response example: https://localhost:9005/rest.php?entity=civiproxy&action=getpreferences&key=SITE_KEY&version=3&json=1&api_key=API_KEY&hash=2261039855&contact_id=6
  • Queue consume files:
  1. /src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/wmf-civicrm/CRM/Queue/PreferencesQueueConsumer.php
  2. /src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/wmf-civicrm/api/v3/Preferencesqueue/Consume.php
  3. /src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/wmf-civicrm/api/v3/Preferences/Create.php
  • Css file:
    • src/email-pref-ctr/extensions/DonationInterface/modules/css/emailPreferences.css
  • html file:
    • /src/email-pref-ctr/extensions/DonationInterface/email_forms/templates/emailPreferences.html.mustache
  • i18n file:
    • /src/email-pref-ctr/extensions/DonationInterface/email_forms/i18n/en.json
  • php file:
    • /src/email-pref-ctr/extensions/DonationInterface/special/EmailPreferences.php
  • Cmd to consume
    • drush cvapi Preferencesqueue.consume time_limit=1000 max_batch_size=1

Day 15

Monthly Convert

Currently we have Ingenico and Adyen those two methods to support the Monthly Convert, which we would love to use a Modal to ask our one time donor do they want to do the monthly donation after they one time payment.

Monthly convert Ingenico local test links:

  • https://localhost:9001/index.php?title=Special:IngenicoGateway&appeal=JimmyQuote&ffname=cc-vmad&payment_method=cc&recurring=0&uselang=en&language=en&currency=USD&amount=35&country=US&first_name=Jimmy&last_name=Wales&street_address=1+Montgomery+Street&city=San+Francisco&state_province=CA&postal_code=94104&email=jwales%40example.com&debug=true&variant=monthlyConvert
  • https://localhost:9001/index.php/Special:IngenicoGateway?debugMonthlyConvert=true

Set breakpoint @ src/payments/extensions/DonationInterface/gateway_common/RecurringConversion.api.php