You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Wikimedia Cloud Services team/EnhancementProposals/Decision record T316866 Openstack Upgrade Cadence

From Wikitech-static
Jump to navigation Jump to search

Origin task: phab:T316866

Date of the decision: 5 October 2022

People in the decision meeting (alphabetical order):

Decision taken

Option 3

Rationale

Folks were interested in exploring both option 3 and option 4. Option 1 was removed during discussion. Ultimately option 4 was left as a future idea, and option 3 was selected.

Problem

Historically, WMCS has upgraded Openstack on a looser cadence, intending to following the stable -1 version. Without a tighter cadence, generally WMCS has lagged 1-2+ versions behind stable. At times this has caused issues with new feature adoption, for example with Trove and most recently Magnum requiring newer versions of Openstack before deployment.

Goals:

  • Better manage upgrades. Openstack releases in April and October. We should also plan consistent times of the year to do upgrades in response.
  • Run a generally newer Openstack version on average, while still seeking lag time for stability.
  • Make it easier to patch or run newer versions of Openstack as needed in response to a bug or desired feature

Constraints and Risks

  • A stable system is prioritized over features
  • Doing nothing will mean k8s clusters run by Magnum will almost always be EOL during operation.
    • This is due to the following. Kubernetes supports releases for 18 month. Openstack adopts a 9 month old release for stable. 6-9 months later, we upgrade to this version, thus 18 months have elapsed since the kubernetes upstream release, making it EOL.
    • It doesn't seem possible to upgrade Magnum k8s version without upgrading Openstack. This means, our Openstack and kubernetes versions will be tied together.
    • Note, currently our existing k8s version is EOL.
  • Today, WMCS is dependent on debian to package Openstack. In the past, this has led to delays due to this packaging work, as well as not all point releases being packaaged.
  • WMCS currently patches openstack, and will continue to do so

Proposals:

Option 1:

Do Nothing. Accept the status quo, including the adhoc upgrade cycle for openstack releases.

Pros

  • No change required
  • Maximum flexibility for planning work
  • Ability to defer upgrades and run EOL without missing expectations

Cons

  • No set expectations for ourselves or users
  • Potential for cumbersome scenarios requiring multiple unplanned upgrades to occur to fix an issue or add a feature
  • No goals met

Option 2:

Maintain n-1 target. Accept running EOL k8s. Schedule twice yearly upgrade months to set expectations.

Pros

  • Same as option 1, with only minor change and flexibility loss
  • Ensures predictability and maintenance of upgrades, rather than relying on adhoc efforts

Cons

  • Potential for cumbersome scenarios requiring multiple unplanned upgrades to occur to fix an issue or add a feature. Even if delay is accepted on adding a feature (according to the upgrade schedule), critical or security issues
  • Addresses only the first of the three stated goals

Option 3:

Create new n-0.5 target cadence. Upgrade to stable version 1-3 months after release.

Pros

  • Ensures openstack versions are up to date and supported during the entire time of operation
  • Ensures availability of a kubernetes version that is not EOL during the entire time of operation
  • Ensures predictability and maintenance of upgrades, rather than relying on adhoc efforts
  • No change to upgrade process required

Cons

  • Patching burden isn't improved
  • Maintains dependency on debian packaging

Option 4:

Create new n-0.5 target cadence. Upgrade to stable version 1-3 months after release. Utilize docker or similar for deployment.

Pros

  • Everything under option 3
  • Lower patching burden
  • Improved flexibility to upgrade or respond to issues
  • Meets all stated goals
  • Better target for automation

Cons

  • Requires changing how we deploy Openstack; this will require research and design