You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Wikimedia Cloud Services team/EnhancementProposals/ceph client refactor"

From Wikitech-static
Jump to navigation Jump to search
imported>Arturo Borrero Gonzalez
 
imported>Arturo Borrero Gonzalez
(drop migration section, now tracked in phab instead https://phabricator.wikimedia.org/T293752)
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{draft}}
This page contains a puppet refactor proposal for the ceph client code.
This page contains a puppet refactor proposal for the ceph client code.


== Problem statement ==
== Problem statement ==


The ceph rbd client code in puppet is a bit of a mess.
The ceph rbd client code in puppet does not have a easy and single way to use it, some points:
 
* There's 3 ways of using it so far, sometimes it splits by host that uses it (<code>ceph::client::rbd_cloudcontrol</code>) sometimes by service (<code>ceph::client::rbd_glance/rbd_libvirt</code>) and for cinder it's in the main class instead (<code>p:openstack::codfw1dev::cinder</code>).
 
* There is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
** nova VM disks (often referred to as 'compute' in the puppet tree)
** glance images
** cinder volumes
** radosgw


Sometimes it splits by host that uses it (<code>ceph::client::rbd_cloudcontrol</code>) sometimes by service (<code>ceph::client::rbd_glance/rbd_libvirt</code>) and for cinder it's in the main class instead (<code>p:openstack::codfw1dev::cinder</code>).
* Setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.


Additionally, there is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
== Proposed solution ==
* nova VM disks (often referred to as 'compute' in the puppet tree)
'''In summary''': we should refresh the code to support the mappings:
* glance images
* cinder volumes
* radosgw


Moreover, setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.
<openstack deployment>+<sevice> <-> <ceph deployment>+<access type>


'''In summary''': we should refresh the code to support a matrix of combinations:
Where:
* configuration per openstack deployment (there is already support in the puppet tree for having more than 1 deployment per DC,)
* <code>openstack deployment</code> is one of eqiad1 or codfw1dev for now
* configuration per ceph cluster (currently there is only support for one ceph cluster per DC)
* <code>service</code> is one of glance, cinder, nova, cinder-backups, ...
* configuration per ceph cluster pool/user (no clear support for this as of this writing)
* <code>ceph deployment</code> is currently "ceph", though this should be extended in the future to allow multi-dc/multi-cluster deployments ([https://phabricator.wikimedia.org/T281250 task])
* <code>access type</code> this would be an abstraction over actual ceph auth permissions. Something like <code>glance_client</code>, <code>nova_client</code>, <code>radosgw_client</code>, <code>osd_server</code>, <code>mon_server</code>, ... this will then realize to the correct auth caps statements (ex. <code>[osd] profile rbd pool=equiad1-compute</code>).


== Requirements ==
=== abstraction ===
 
Inspired by how acme-chief manages certificates:
 
* the source of truth is a private hiera hash with this layout, what it is called a '''DC-specific configuration hash''':
 
<pre>
profile::ceph::auth::load_all::configuration:
  cinder:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  glance:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  osd:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
</pre>


Each openstack role should be able to load ceph client credentials in an elegant way, for example:
* the hiera hash is per-DC (i.e, eqiad, codfw). Future iterations may evolve this to support multiple ceph farms per DC, but we don't want this at this point
* clients may declare which creds they want by using an additional hiera key, for example, on file '''hieradata/roles/codfw/wmcs/openstack/codfw1dev/control.yaml''':


<pre>
<pre>
* role::wmcs::openstack::eqiad1::control
profile::ceph::auth::deploy::selected_creds:
** ( ..many other includes.. )
  - cinder
** include profile::ceph::XXXX::rbc_client::cloudcontrol
  - glance
*** include profile::ceph::XXXX::rbd_client::nova
*** include profile::ceph::XXXX::rbd_client::glance
*** include profile::ceph::XXXX::rbd_client::cinder
*** include profile::ceph::XXXX::rbd_client::swift (or radosgw)
</pre>
</pre>
Once this hiera structures are in place, we have 2 important profiles/modules to handle that data:
* <code>profile::ceph::auth::load_all</code>: profile loaded on '''mons''', reads the hiera hash (has a parameter 'configuration'), and calls internally the inner module:
** <code>ceph::auth::load_all</code> receives a list of credentials and loads them to the ceph running system.
* <code>profile::ceph::auth::deploy</code>: profile loaded on ceph clients, like ''osd'', ''cloudcontrol'' and friends. Reads configuration hash (has a parameter 'auths'), and calls internally the inner module:
** <code>ceph::auth::deploy</code>: deploys a keyring file on the filesystem.
=== code organization proposal ===
With the abstraction described above, we could use the puppet code like this:


<pre>
<pre>
* role::wmcs::openstack::eqiad1::virt_ceph
* role::wmcs::ceph::mon
** ( ..many other includes.. )
** profile::ceph::auth::load_all <-- reads DC-specific configuration hiera hash (i.e, eqiad or codfw)
** include profile::ceph::XXXX::rbc_client::virt
 
*** include profile::ceph::XXXX::rbd_client::nova
* role::wmcs::ceph::osd
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files
 
* role::wmcs::openstack::codfw1dev::control
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files
</pre>
</pre>


A code example of this:
<syntaxhighlight lang="puppet">
class profile::ceph::auth::load_all (
  Hash $configuration = lookup('ceph::auth'),
) {
  class { 'ceph::auth::load_all':
    configuration => $configuration,
  }
}
class ceph::auth::load_all (
  Hash $configuration,
) {
    # for each entry in the configuration hash
    # load it using 'ceph auth get-or-create'
}
</syntaxhighlight>
Also:
<syntaxhighlight lang="puppet">
class profile::ceph::auth::deploy (
  Hash $configuration = lookup('ceph::auth'),
  Array[String] $selected_creds = lookup('profile::ceph::auth::deploy::selected_creds'),
) {
  class { 'ceph::auth::deploy':
      configuration = $configuration,
      selected_creds = $selected_creds,
  }
}
class ceph::auth::deploy(
  Hash $configuration,
  Array[String] $selected_creds, 
) {
  # filter configuration for the selected creds
  # if we find a match, then create a keyring file resource with the keydata
}
</syntaxhighlight>
Example hiera configuration:
File on '''private repository''' <code>hieradata/common/codfw.yaml</code> (which should be available to all systems on the DC --- TODO: we may want to have a more suitable place for this)
<pre>
<pre>
* role::wmcs::openstack::eqiad1::cinder_backups <--- made up name, for illustration purposes; may happen for real soon
ceph::auth:
** ( ..many other includes.. )
  cinder:
** include profile::ceph::XXXX::rbc_client::backups
    keydata: XXXXXX
*** include profile::ceph::XXXX::rbd_client::cinder
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
  glance:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
  osd:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
</pre>
</pre>


Where XXXX is some kind of new identifier for a ceph cluster (see [[phab:T281250]]) '''TO BE DECIDED'''.
File '''on the public repo''' <code>hieradata/role/codfw/wmcs/openstack/codfw1dev/control.yaml</code>, which should be available to all openstack codfw1dev cloudcontrol servers:
<pre>
profile::ceph::auth::deploy:
  - cinder
  - glance
</pre>


This includes the relevant layer of hiera overrides.
File '''on the public repo''' <code>hieradata/role/codfw/wmcs/openstack/codfw1dev/virt_ceph.yaml</code>, which should be available to all openstack codfw1dev virt servers:
<pre>
profile::ceph::auth::deploy:
  - compute
</pre>


== See also ==
== See also ==
Line 58: Line 181:
* [[phab:T292546 | T292546: cloud NFS: figure out backups for cinder volumes]]
* [[phab:T292546 | T292546: cloud NFS: figure out backups for cinder volumes]]
* [[phab:T281250 | T281250: ceph.operationalization: Refactor the current puppet code to allow per-cluster configurations (as opposed to per-DC as it does currently)]]
* [[phab:T281250 | T281250: ceph.operationalization: Refactor the current puppet code to allow per-cluster configurations (as opposed to per-DC as it does currently)]]
Potential inspiration:
* https://github.com/datacentred/puppet-ceph

Latest revision as of 12:34, 15 November 2021

This page contains a puppet refactor proposal for the ceph client code.

Problem statement

The ceph rbd client code in puppet does not have a easy and single way to use it, some points:

  • There's 3 ways of using it so far, sometimes it splits by host that uses it (ceph::client::rbd_cloudcontrol) sometimes by service (ceph::client::rbd_glance/rbd_libvirt) and for cinder it's in the main class instead (p:openstack::codfw1dev::cinder).
  • There is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
    • nova VM disks (often referred to as 'compute' in the puppet tree)
    • glance images
    • cinder volumes
    • radosgw
  • Setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.

Proposed solution

In summary: we should refresh the code to support the mappings:

<openstack deployment>+<sevice> <-> <ceph deployment>+<access type>

Where:

  • openstack deployment is one of eqiad1 or codfw1dev for now
  • service is one of glance, cinder, nova, cinder-backups, ...
  • ceph deployment is currently "ceph", though this should be extended in the future to allow multi-dc/multi-cluster deployments (task)
  • access type this would be an abstraction over actual ceph auth permissions. Something like glance_client, nova_client, radosgw_client, osd_server, mon_server, ... this will then realize to the correct auth caps statements (ex. [osd] profile rbd pool=equiad1-compute).

abstraction

Inspired by how acme-chief manages certificates:

  • the source of truth is a private hiera hash with this layout, what it is called a DC-specific configuration hash:
profile::ceph::auth::load_all::configuration:
  cinder:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  glance:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  osd:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  • the hiera hash is per-DC (i.e, eqiad, codfw). Future iterations may evolve this to support multiple ceph farms per DC, but we don't want this at this point
  • clients may declare which creds they want by using an additional hiera key, for example, on file hieradata/roles/codfw/wmcs/openstack/codfw1dev/control.yaml:
profile::ceph::auth::deploy::selected_creds:
  - cinder
  - glance

Once this hiera structures are in place, we have 2 important profiles/modules to handle that data:

  • profile::ceph::auth::load_all: profile loaded on mons, reads the hiera hash (has a parameter 'configuration'), and calls internally the inner module:
    • ceph::auth::load_all receives a list of credentials and loads them to the ceph running system.
  • profile::ceph::auth::deploy: profile loaded on ceph clients, like osd, cloudcontrol and friends. Reads configuration hash (has a parameter 'auths'), and calls internally the inner module:
    • ceph::auth::deploy: deploys a keyring file on the filesystem.

code organization proposal

With the abstraction described above, we could use the puppet code like this:

* role::wmcs::ceph::mon
** profile::ceph::auth::load_all <-- reads DC-specific configuration hiera hash (i.e, eqiad or codfw)

* role::wmcs::ceph::osd
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files

* role::wmcs::openstack::codfw1dev::control
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files

A code example of this:

class profile::ceph::auth::load_all (
  Hash $configuration = lookup('ceph::auth'),
) {

   class { 'ceph::auth::load_all':
     configuration => $configuration,
   }
}

class ceph::auth::load_all (
  Hash $configuration,
) {
    # for each entry in the configuration hash
    # load it using 'ceph auth get-or-create'
}

Also:

class profile::ceph::auth::deploy (
  Hash $configuration = lookup('ceph::auth'),
  Array[String] $selected_creds = lookup('profile::ceph::auth::deploy::selected_creds'),
) {
  class { 'ceph::auth::deploy':
      configuration = $configuration,
      selected_creds = $selected_creds,
  }
}

class ceph::auth::deploy(
  Hash $configuration,
  Array[String] $selected_creds,  
) {
   # filter configuration for the selected creds
   # if we find a match, then create a keyring file resource with the keydata
}

Example hiera configuration:

File on private repository hieradata/common/codfw.yaml (which should be available to all systems on the DC --- TODO: we may want to have a more suitable place for this)

ceph::auth:
  cinder:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
  glance:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
  osd:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"

File on the public repo hieradata/role/codfw/wmcs/openstack/codfw1dev/control.yaml, which should be available to all openstack codfw1dev cloudcontrol servers:

profile::ceph::auth::deploy:
  - cinder
  - glance

File on the public repo hieradata/role/codfw/wmcs/openstack/codfw1dev/virt_ceph.yaml, which should be available to all openstack codfw1dev virt servers:

profile::ceph::auth::deploy:
  - compute

See also

Potential inspiration: