You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.
- Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
- Make sure you are logged in to Jenkins to see the pages linked in the steps below.
- Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/. Everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form. If pre-filled values are not correct, the simplest solution is to trigger a build (instead of a release) for the project: https://integration.wikimedia.org/ci/job/analytics-refinery-release/build?delay=0sec - It should update jenkins workspace with up-to-date values.
- Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left
Build Historypanel (In case the last job fails, it will have a red dot before its build number - see below).
- Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
- There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in refinery oozie job(s). You then need to
Remember to log onto the analytics IRC channel upon successful deploy, for instance:
If the maven release job failed (step 4)
First find why looking at job logs, and then two solutions:
- Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
- Clean maven commits:
git reset --hard YOUR_LAST_COMMIT_HASH
git push --force origin master
- Clean maven tag:
git tag -d YOUR_VERSION_TAG
git push origin :refs/tags/YOUR_VERSION_TAG
- Clean maven commits:
- Or, skip the version number you were currently deploying
- Update the changelog,md with the skipped version info
- Deploy again !
If the maven jar upgrade job failed (step 5)
How to deploy from the CLI
You need archiva passwords in order to follow that procedure. Ask Andrew if needed.
- Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
- Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
- Check everything looks Ok:
- Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform
- Download the new version of the jars from archiva (
) For convenience, the commonly updated jars are:git fat uses IDs to manage files, so it is important to use the correct jars.
- NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
- Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
- Update the symlinks in refinery/artifacts to the jars you just copied
- Make sure git fat is installed and configured according to the instructions in the Refinery README
- git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
- push for review
Please see the refinery page to deploy the jars and oozie code.