You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Analytics/Cluster/Refinery-source: Difference between revisions
Jump to navigation
Jump to search
imported>Paladox No edit summary |
imported>Madhuvishy (Deployment instructions with Jenkins) |
||
| Line 1: | Line 1: | ||
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source]. | Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source]. | ||
== How to deploy with Jenkins == | |||
# Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change. | |||
# Make sure you are logged in to Jenkins to see the pages linked in the steps below. | |||
# Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form. | |||
# Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. | |||
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success. | |||
= How to deploy = | = How to deploy = | ||
Revision as of 22:29, 7 July 2016
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.
How to deploy with Jenkins
- Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
- Make sure you are logged in to Jenkins to see the pages linked in the steps below.
- Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
- Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts.
- Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
How to deploy
 | You need archiva passwords in order to follow that procedure. Ask Andrew if needed. |
- Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
- Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
- Check everything looks Ok:
cat release.properties
- Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform
- Download the new version of the jars from archiva (
) For convenience, the commonly updated jars are:git fat uses IDs to manage files, so it is important to use the correct jars. - NOTE: to download, click on the new version you just created, then Artifacts, then copy the link to the jar and update 127.0.0.1:8080 with archiva.wikimedia.org and download that
- Copy the updated jars to the correct refinery path (somewhere like <refinery>/artifacts/org/wikimedia/analytics/refinery/)
- Update the symlinks in refinery/artifacts to the jars you just copied
- Make sure git fat is installed and configured according to the instructions in the Refinery README
- git add . && git commit (git fat will do some magic and replace the jar with a one-line id)
- push for review
Please see the refinery page to deploy the jars and oozie code.