You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Paladox
No edit summary
imported>Madhuvishy
(Deployment instructions with Jenkins)
Line 1: Line 1:
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source].
Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the [https://phabricator.wikimedia.org/diffusion/ANRS/ analytics/refinery/source].
== How to deploy with Jenkins ==
# Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
# Make sure you are logged in to Jenkins to see the pages linked in the steps below.
# Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
# Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts.
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.


= How to deploy =
= How to deploy =

Revision as of 22:29, 7 July 2016

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins

  1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
  2. Make sure you are logged in to Jenkins to see the pages linked in the steps below.
  3. Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
  4. Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts.
  5. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.

How to deploy

  • Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
  • Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
  • Check everything looks Ok:
cat release.properties
  • Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

Please see the refinery page to deploy the jars and oozie code.