You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Search/S3 Plugin Enable: Difference between revisions
imported>Bking No edit summary |
imported>Bking No edit summary |
||
| Line 6: | Line 6: | ||
=== Complicating factors === | === Complicating factors === | ||
We have an unorthodox Elastic environment: specifically, we run 2 or 3 Elasticsearch instances on a single host. As a result, using [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/secure-settings.html#reloadable-secure-settings the elasticsearch-keystore] requires special care. | We have an unorthodox Elastic environment: specifically, we run 2 or 3 Elasticsearch instances on a single host. As a result, using [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/secure-settings.html#reloadable-secure-settings the elasticsearch-keystore] requires special care. | ||
==== Getting the elastic keystore path right ==== | |||
By default, ''elasticsearch-keystore'' invokes java with the wrong es.path.conf. We can override by setting ES_PATH_CONF when invoking elasticsearch-keystore: | |||
<code>export ES_PATH_CONF=/etc/elasticsearch/production-search-psi-codfw; /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key</code> | |||
Revision as of 16:00, 21 June 2022
Brian's notes for deploying the S3 plugin. See T309720 for more details.
Why Enable the S3 Plugin?
Currently, there is no easy way to restore data from production to cloudelastic. The easiest way to do this is to use thanos-swift to move data around. Elasticsearch has better support for the S3 API as opposed to swift. This is (sadly) pretty common, as the Search platform team has seen with flink already.
Complicating factors
We have an unorthodox Elastic environment: specifically, we run 2 or 3 Elasticsearch instances on a single host. As a result, using the elasticsearch-keystore requires special care.
Getting the elastic keystore path right
By default, elasticsearch-keystore invokes java with the wrong es.path.conf. We can override by setting ES_PATH_CONF when invoking elasticsearch-keystore:
export ES_PATH_CONF=/etc/elasticsearch/production-search-psi-codfw; /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key