You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Wikimedia Cloud Services team/EnhancementProposals/ceph client refactor"

From Wikitech-static
Jump to navigation Jump to search
imported>Arturo Borrero Gonzalez
 
imported>David Caro
Line 5: Line 5:
== Problem statement ==
== Problem statement ==


The ceph rbd client code in puppet is a bit of a mess.
The ceph rbd client code in puppet does not have a easy and single way to use it, some points:


Sometimes it splits by host that uses it (<code>ceph::client::rbd_cloudcontrol</code>) sometimes by service (<code>ceph::client::rbd_glance/rbd_libvirt</code>) and for cinder it's in the main class instead (<code>p:openstack::codfw1dev::cinder</code>).
* There's 3 ways of using it so far, sometimes it splits by host that uses it (<code>ceph::client::rbd_cloudcontrol</code>) sometimes by service (<code>ceph::client::rbd_glance/rbd_libvirt</code>) and for cinder it's in the main class instead (<code>p:openstack::codfw1dev::cinder</code>).


Additionally, there is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
* There is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
* nova VM disks (often referred to as 'compute' in the puppet tree)
** nova VM disks (often referred to as 'compute' in the puppet tree)
* glance images
** glance images
* cinder volumes
** cinder volumes
* radosgw
** radosgw


Moreover, setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.
* Setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.


'''In summary''': we should refresh the code to support a matrix of combinations:
== Proposed solution ==
* configuration per openstack deployment (there is already support in the puppet tree for having more than 1 deployment per DC,)
'''In summary''': we should refresh the code to support the mappings:
* configuration per ceph cluster (currently there is only support for one ceph cluster per DC)
* configuration per ceph cluster pool/user (no clear support for this as of this writing)


== Requirements ==
<openstack deployment>+<sevice> <-> <ceph deployment>+<access type>
 
Where:
* <code>openstack deployment</code> is one of eqiad1 or codfw1dev for now
* <code>service</code> is one of glance, cinder, nova, cinder-backups, ...
* <code>ceph deployment</code> is currently "ceph", though this should be extended in the future to allow multi-dc/multi-cluster deployments ([https://phabricator.wikimedia.org/T281250 task])
* <code>access type</code> this would be an abstraction over actual ceph auth permissions. Something like <code>glance_client</code>, <code>nova_client</code>, <code>radosgw_client</code>, <code>osd_server</code>, <code>mon_server</code>, ... this will then realize to the correct auth caps statements (ex. <code>[osd] profile rbd pool=equiad1-compute</code>).
 
== Proposal ==


Each openstack role should be able to load ceph client credentials in an elegant way, for example:
Each openstack role should be able to load ceph client credentials in an elegant way, for example:

Revision as of 10:37, 19 October 2021

This page contains a puppet refactor proposal for the ceph client code.

Problem statement

The ceph rbd client code in puppet does not have a easy and single way to use it, some points:

  • There's 3 ways of using it so far, sometimes it splits by host that uses it (ceph::client::rbd_cloudcontrol) sometimes by service (ceph::client::rbd_glance/rbd_libvirt) and for cinder it's in the main class instead (p:openstack::codfw1dev::cinder).
  • There is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
    • nova VM disks (often referred to as 'compute' in the puppet tree)
    • glance images
    • cinder volumes
    • radosgw
  • Setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.

Proposed solution

In summary: we should refresh the code to support the mappings:

<openstack deployment>+<sevice> <-> <ceph deployment>+<access type>

Where:

  • openstack deployment is one of eqiad1 or codfw1dev for now
  • service is one of glance, cinder, nova, cinder-backups, ...
  • ceph deployment is currently "ceph", though this should be extended in the future to allow multi-dc/multi-cluster deployments (task)
  • access type this would be an abstraction over actual ceph auth permissions. Something like glance_client, nova_client, radosgw_client, osd_server, mon_server, ... this will then realize to the correct auth caps statements (ex. [osd] profile rbd pool=equiad1-compute).

Proposal

Each openstack role should be able to load ceph client credentials in an elegant way, for example:

* role::wmcs::openstack::eqiad1::control
** ( ..many other includes.. )
** include profile::ceph::XXXX::rbc_client::cloudcontrol
*** include profile::ceph::XXXX::rbd_client::nova
*** include profile::ceph::XXXX::rbd_client::glance
*** include profile::ceph::XXXX::rbd_client::cinder
*** include profile::ceph::XXXX::rbd_client::swift (or radosgw)
* role::wmcs::openstack::eqiad1::virt_ceph
** ( ..many other includes.. )
** include profile::ceph::XXXX::rbc_client::virt
*** include profile::ceph::XXXX::rbd_client::nova
* role::wmcs::openstack::eqiad1::cinder_backups <--- made up name, for illustration purposes; may happen for real soon
** ( ..many other includes.. )
** include profile::ceph::XXXX::rbc_client::backups
*** include profile::ceph::XXXX::rbd_client::cinder

Where XXXX is some kind of new identifier for a ceph cluster (see phab:T281250) TO BE DECIDED.

This includes the relevant layer of hiera overrides.

See also