You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.
- Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
- Make sure you are logged in to Jenkins to see the pages linked in the steps below.
- Releasing a new version of refinery-source jars to Maven
- If you need to trigger a build (not a release) you can do that hereː https://integration.wikimedia.org/ci/job/analytics-refinery-release/build?delay=0sec This is known to refresh the values that you'll see in the next step, but usually they should be pre-filled
- Visit https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/
- Update the version number in the Release Version and Development Version boxes if needed
- Release Version should be the version number you wish to deploy
- Deployment Version should be the next iteration of devlopement version, usually Release Version + 1 with -SNAPSHOT at the end
- Check the "Specify custom SCM tag" checkbox
- Change refinery-x.y.z to vx.y.z in the "SCM tag" input textbox and update the number
- Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left
Build Historypanel (In case the last job fails, it will have a red dot before its build number - see below)
- Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
- There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in refinery oozie job(s). You then need to
Remember to log onto the analytics IRC channel upon successful deploy, for instance:
If the maven release job failed (step 3)
First find why looking at job logs, and then two solutions:
- Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
- Clean maven commits:
git reset --hard YOUR_LAST_COMMIT_HASH
git push --force origin master
- Clean maven tag:
git tag -d YOUR_VERSION_TAG
git push origin :refs/tags/YOUR_VERSION_TAG
- Clean maven commits:
- Or, skip the version number you were currently deploying
- Update the changelog,md with the skipped version info
- Deploy again !
If the maven jar upgrade job failed (step 4)
How to deploy from the CLI
You need archiva passwords in order to follow that procedure. Ask Andrew if needed.
- Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
- Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
- Check everything looks Ok:
- Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform
- Download the new version of the jars from archiva (
) For convenience, the commonly updated jars are:git fat uses IDs to manage files, so it is important to use the correct jars.
- NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
- Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
- Update the symlinks in refinery/artifacts to the jars you just copied
- Make sure git fat is installed and configured according to the instructions in the Refinery README
- git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
- push for review
Please see the refinery page to deploy the jars and oozie code.