You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Analytics/Cluster/Deploy/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Mforns
imported>Milimetric
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source].
#REDIRECT [[Analytics/Systems/Cluster/Deploy/Refinery-source]]
 
= How to deploy with Jenkins (and related steps) =
'''Before starting, please check:'''
* Check the latest Refinery source version. This can be done in various way, the easiest one is to check <code>git tag --list</code>
* The [https://github.com/wikimedia/analytics-refinery-source/commits/master Refinery source commit list] and make sure that the changelog.md has been updated with the latest version and possibly that [maven-release-plugin] has committed the related version bump changes (the last step is optional, it could be triggered manually following the instruction below).
* Make sure you are logged in to [https://integration.wikimedia.org/ci/ Jenkins] to see the pages linked in the steps below.
'''Have you read the instructions reported above? If so, please keep going, otherwise please read them!'''
 
Deploy procedure:
# Update the '''changelog.md''' file at the root of the repository with changes that are going to be deployed - commit and merge this change.
# Release a new version of refinery-source jars to Maven:
#* Visit https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ and <u>make sure that</u>:
#** Release Version should be the version number you wish to deploy
#** Deployment Version should be the next iteration of development version, usually Release Version + 1 with -SNAPSHOT at the end
#** Example: Release '''0.0.40''' - Development '''0.0.41-SNAPSHOT'''
#** If the versions are not the ones that you expect, you can refresh them with https://integration.wikimedia.org/ci/job/analytics-refinery-release/build?delay=0sec  This is known to refresh the values that you'll see in the next step, but usually they should be pre-filled.
#* Check the "'''Specify custom SCM tag'''" checkbox
#* Change '''refinery-x.y.z''' to '''vx.y.z''' in the "SCM tag" input text-box and update the number. Example: '''<s>refinery-0.0.40</s>''' is bad, '''v0.0.40''' is good
#* Hit '''Schedule Maven Release Build'''. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left <code>Build History</code> panel (In case the last job fails, it will have a red dot before its build number - see below)
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
# There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in [[Analytics/Cluster/Refinery|refinery]] oozie job(s). You then need to
#* bump up jar version in the related oozie jobs (see an example [https://gerrit.wikimedia.org/r/#/c/319084/ here])
#* [[Analytics/Cluster/Refinery|deploy refinery]] with your changes
#* [[labsconsole:Analytics/Cluster/Oozie/Administration#When_.properties_file_has_changed|restart the modified oozie jobs]].
 
{{Notice|Remember to log onto the analytics IRC channel upon successful deploy, for instance: <code>!log Deployed refinery-source using jenkins</code>.)}}
 
===== If the maven release job failed (step 2) =====
First find why looking at job logs, and then two solutions:
* Clean Maven commits for version upgrade in pom files and associated tag (<u>can break things, be careful !</u>)
** Clean maven commits:
*** <code>git reset --hard YOUR_LAST_COMMIT_HASH</code>
*** <code>git push --force origin master</code>
** Clean maven tag:
*** <code>git tag -d YOUR_VERSION_TAG</code>
*** <code>git push origin :refs/tags/YOUR_VERSION_TAG</code>
* Or, skip the version number you were currently deploying
** Update the changelog,md with the skipped version info
** Deploy again!
 
= How to deploy from the CLI =
 
{{Warning | You need archiva passwords in order to follow that procedure. Ask [[User:Ottomata|Andrew]] if needed. }}
 
* Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
*  Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
* Check everything looks Ok:
cat release.properties
* Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform
* Download the new version of the jars from archiva ({{Warning | git fat uses IDs to manage files, so it is important to use the correct jars. }}) For convenience, the commonly updated jars are:
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.hive/refinery-hive
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.core/refinery-core
** https://archiva.wikimedia.org/#artifact/org.wikimedia.analytics.refinery.camus/refinery-camus
* NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
* Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
* Update the symlinks in refinery/artifacts to the jars you just copied
* Make sure git fat is installed and configured according to the instructions in the Refinery README
* git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
* push for review
 
Please see the [[Analytics/Cluster/Refinery|refinery page]] to deploy the jars and oozie code.

Latest revision as of 13:44, 7 April 2017