You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Integration/Setup

From Wikitech-static
Jump to navigation Jump to search

Roles

integration-agent-{type}-XXXX

Updated September 2019 based on https://phabricator.wikimedia.org/T226233 Updated January 2021

The instances are created via https://horizon.wikimedia.org/project/instances/ , you will need a source image to pick and an instance flavor.



For the flavor the important parts are:

  • have enough disk space (docker role notably requests 24G for /var/lib/docker and you would need enough disk remaining for /srv).
  • have a 4xiops flavor which dramatically boost the underlying Disk IO rate limiting applied to all WMCS instances.
  • Create a new instance named integration-agent-{type}-XXXX where {type} is a role (example: docker) and XXXX increments starting from 1001.


Wait a few minutes (during which the instance is created, initial setup happens). Then connect to the instance over SSH and fix puppet:

  • sudo rm -fR /var/lib/puppet/ssl && sudo puppet agent -tv
  • If that complains:
    • get the instance fully qualified domain name (FQDN): hostname --fqdn
    • On integration-puppetmaster-02.integration.eqiad.wmflabs, clean the old and invalid certificate(s): sudo puppet cert clean <FQDN OF INSTANCE HERE

Apply the Puppet role:

The Docker agent will have a 24G /var/lib/docker partition, the remaining disk space is allocated to /srv.

Run Puppet on the instance (puppet agent -tv) and verify:

  • If a Docker agent, make sure there is a /var/lib/docker partition for Docker
  • Clean unused packages: apt-get autoremove --purge
  • Upgrade packages: apt-get -y dist-upgrade

Reboot the instance (Before adding to Jenkins). This cleans state, take in account the new Linux kernel if any, launches daemons. Once it is back, you can then add it to Jenkins

Add the instance to Jenkins

  1. Create "New Node" in Jenkins management
    • Name: (short hostname of instance)
    • Type: Permanent Agent
    • Executors: 1 (for Docker agents: 4, for Qemu agents: 1)
    • Remote root directory: /srv/jenkins/workspace
    • Labels:
      • For Docker agents: Docker
      • For Qemu agents: Qemu
    • Usage: EXCLUSIVE (Only build jobs with label restrictions matching this node)
    • Launch method: SSH
      • Host: (internal IP of instance)
      • Credentials: jenkins-deploy (key from role::ci::slave::labs::common)
    • Availability: Always (Keep this slave on-line as much as possible)

The Jenkins master will automatically trust the ssh key upon the first connection.

integration-dev

  1. Create instance:
    • m1.medium
    • Security group: Default
  2. Wait 10 minutes
  3. Reconfigure instance from wikitech: Enable role::ci::slave::labs.
  4. Via SSH, force a puppet run (applies role).

Utilities

puppet

Use sudo /usr/local/sbin/puppet-run &. Don't use sudo puppet agent -t, because that is not what cron uses and leads to inconsistencies with e.g. umask and other factors affecting default values used at runtime.