You are browsing a read-only backup copy of Wikitech. The live site can be found at

Wikimedia Cloud Services team/EnhancementProposals/ceph client refactor

From Wikitech-static
< Wikimedia Cloud Services team‎ | EnhancementProposals
Revision as of 17:10, 18 October 2021 by imported>Arturo Borrero Gonzalez (→‎Problem statement: typos)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

This page contains a puppet refactor proposal for the ceph client code.

Problem statement

The ceph rbd client code in puppet is a bit of a mess.

Sometimes it splits by host that uses it (ceph::client::rbd_cloudcontrol) sometimes by service (ceph::client::rbd_glance/rbd_libvirt) and for cinder it's in the main class instead (p:openstack::codfw1dev::cinder).

Additionally, there is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:

  • nova VM disks (often referred to as 'compute' in the puppet tree)
  • glance images
  • cinder volumes
  • radosgw

Moreover, setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.

In summary: we should refresh the code to support a matrix of combinations:

  • configuration per openstack deployment (there is already support in the puppet tree for having more than 1 deployment per DC,)
  • configuration per ceph cluster (currently there is only support for one ceph cluster per DC)
  • configuration per ceph cluster pool/user (no clear support for this as of this writing)


Each openstack role should be able to load ceph client credentials in an elegant way, for example:

* role::wmcs::openstack::eqiad1::control
** ( ..many other includes.. )
** include profile::ceph::XXXX::rbc_client::cloudcontrol
*** include profile::ceph::XXXX::rbd_client::nova
*** include profile::ceph::XXXX::rbd_client::glance
*** include profile::ceph::XXXX::rbd_client::cinder
*** include profile::ceph::XXXX::rbd_client::swift (or radosgw)
* role::wmcs::openstack::eqiad1::virt_ceph
** ( ..many other includes.. )
** include profile::ceph::XXXX::rbc_client::virt
*** include profile::ceph::XXXX::rbd_client::nova
* role::wmcs::openstack::eqiad1::cinder_backups <--- made up name, for illustration purposes; may happen for real soon
** ( ..many other includes.. )
** include profile::ceph::XXXX::rbc_client::backups
*** include profile::ceph::XXXX::rbd_client::cinder

Where XXXX is some kind of new identifier for a ceph cluster (see phab:T281250) TO BE DECIDED.

This includes the relevant layer of hiera overrides.

See also