You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Analytics/Cluster/Refinery-source: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Madhuvishy
(Deployment instructions with Jenkins)
imported>Joal
(→‎How to deploy with Jenkins: Add some links and recovery procedures.)
Line 3: Line 3:
== How to deploy with Jenkins ==
== How to deploy with Jenkins ==
# Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
# Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
# Make sure you are logged in to Jenkins to see the pages linked in the steps below.
# Make sure you are logged in to [https://integration.wikimedia.org/ci/ Jenkins] to see the pages linked in the steps below.
# Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
# Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
# Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts.
# Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left <code>Build History</code> panel (In case the last job fails, it will have a red dot before its build number - see below).
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
# Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
===== If the maven release job failed (step 4) =====
First find why looking at job logs, and then two solutions:
* Clean Maven commits for version upgrade in pom files and associated tag (<u>can break things, be careful !</u>)
** Clean maven commits:
*** <code>git reset --hard YOUR_LAST_COMMIT_HASH</code>
*** <code>git push --force origin master</code>
** Clean maven tag:
*** <code>git tag -d YOUR_VERSION_TAG</code>
*** <code>git push origin :refs/tags/YOUR_VERSION_TAG</code>
* Or, skip the version number you were currently deploying
** Update the changelog,md with the skipped version info
** Deploy again !
===== If the maven jar upgrade job failed (step 5) =====


= How to deploy =
= How to deploy =

Revision as of 18:47, 12 July 2016

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins

  1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
  2. Make sure you are logged in to Jenkins to see the pages linked in the steps below.
  3. Next step is to release a new version of the refinery-source jars to Maven. This can be done through our Jenkins setup, by visiting https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/, everything required on the form should be prefilled - the only thing to edit is to check the Specify custom SCM tag checkbox, and change the version to vx.y.z(e.g v0.0.29) format instead of refinery-x.y.z which is the default populated tag on the form.
  4. Hit Schedule Maven Release Build. This will build all the jars and deploy them to Archiva, and once done send a job success email to Analytics-alerts. You can follow the job at that address: https://integration.wikimedia.org/ci/job/analytics-refinery-release/, in the bottom-left Build History panel (In case the last job fails, it will have a red dot before its build number - see below).
  5. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/build, and supplying the version number of the latest jars released (e.g 0.0.29), and hitting Build. This will make a commit to refinery and send an email to Analytics-alerts on success.
If the maven release job failed (step 4)

First find why looking at job logs, and then two solutions:

  • Clean Maven commits for version upgrade in pom files and associated tag (can break things, be careful !)
    • Clean maven commits:
      • git reset --hard YOUR_LAST_COMMIT_HASH
      • git push --force origin master
    • Clean maven tag:
      • git tag -d YOUR_VERSION_TAG
      • git push origin :refs/tags/YOUR_VERSION_TAG
  • Or, skip the version number you were currently deploying
    • Update the changelog,md with the skipped version info
    • Deploy again !
If the maven jar upgrade job failed (step 5)

How to deploy

  • Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
  • Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
  • Check everything looks Ok:
cat release.properties
  • Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

Please see the refinery page to deploy the jars and oozie code.