You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Search/S3 Plugin Enable: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Bking
No edit summary
imported>Bking
No edit summary
Line 1: Line 1:
Brian's notes for deploying the S3 plugin. See [[phab:T309648|T309720]] for more details.
Brian's notes for deploying the S3 plugin. See [[phab:T309648|T309720]] for more details. and [[Search/S3 Plugin Enable/Beta Env|this page]] for more about the test environment.


=== Why Enable the S3 Plugin? ===
=== Why Enable the S3 Plugin? ===
Line 11: Line 11:


<code>export ES_PATH_CONF=/etc/elasticsearch/production-search-psi-codfw; /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key</code>
<code>export ES_PATH_CONF=/etc/elasticsearch/production-search-psi-codfw; /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key</code>
Permissions are also very important! The keystore file must have permissions ''root:elasticsearch'' and mode ''0640'' . If the elasticsearch service fails to start after a keystore change, check the paths and permissions. A brand-new <code>elasticsearch-keystore</code> file in <code>/etc/elasticsearch/</code> means the  <code>ES_PATH_CONF=</code>environment variable was not respected. If the<code>elasticsearch-keystore</code>  file is owned by <code>root:root</code> , the service will not start.
==== Path-style and bucket-style access ====
We use the thanos-swift cluster as our object store, via its S3-compatible API . "Real" S3 supports bucket-based access, which relies on DNS records. We don't have this, so we must use path-style access. Unfortunately, Elastic added, removed, then re-added support for this feature. As of this writing, we are on Elasticsearch 6.8, which does not support path-style access. As a result, we use our own custom version of the s3 plugin, installed via our elastic-plugins deb package.

Revision as of 18:08, 30 June 2022

Brian's notes for deploying the S3 plugin. See T309720 for more details. and this page for more about the test environment.

Why Enable the S3 Plugin?

Currently, there is no easy way to restore data from production to cloudelastic. The easiest way to do this is to use thanos-swift to move data around. Elasticsearch has better support for the S3 API as opposed to swift. This is (sadly) pretty common, as the Search platform team has seen with flink already.

Complicating factors

We have an unorthodox Elastic environment: specifically, we run 2 or 3 Elasticsearch instances on a single host. As a result, using the elasticsearch-keystore requires special care.

Getting the elastic keystore path right

By default, elasticsearch-keystore invokes java with the wrong es.path.conf. We can override by setting ES_PATH_CONF when invoking elasticsearch-keystore:

export ES_PATH_CONF=/etc/elasticsearch/production-search-psi-codfw; /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key

Permissions are also very important! The keystore file must have permissions root:elasticsearch and mode 0640 . If the elasticsearch service fails to start after a keystore change, check the paths and permissions. A brand-new elasticsearch-keystore file in /etc/elasticsearch/ means the ES_PATH_CONF=environment variable was not respected. If theelasticsearch-keystore file is owned by root:root , the service will not start.

Path-style and bucket-style access

We use the thanos-swift cluster as our object store, via its S3-compatible API . "Real" S3 supports bucket-based access, which relies on DNS records. We don't have this, so we must use path-style access. Unfortunately, Elastic added, removed, then re-added support for this feature. As of this writing, we are on Elasticsearch 6.8, which does not support path-style access. As a result, we use our own custom version of the s3 plugin, installed via our elastic-plugins deb package.