You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

RESTBase: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Hnowlan
(→‎Adding a new RESTBase host: Move host provisioning documentation to Cassandra page)
imported>Hnowlan
(→‎Adding a new RESTBase host: Simplify conftool command)
Line 30: Line 30:
* [[gerrit:c/operations/puppet/+/632497|Add the hosts to conftool-data]]  
* [[gerrit:c/operations/puppet/+/632497|Add the hosts to conftool-data]]  
* If the hosts are healthy in Icinga at this point and if you feel it is safe as regards deployment timing and so on, pool the hosts:   
* If the hosts are healthy in Icinga at this point and if you feel it is safe as regards deployment timing and so on, pool the hosts:   
** <code>confctl select "dc=DC,cluster=restbase,service=restbase,name=restbaseNNNN.DC.wmnet"  set/pooled=yes</code>
** <code>sudo confctl select name=restbaseNNNN.DC.wmnet  set/pooled=yes</code>
** <code>confctl select "dc=DC,cluster=restbase,service=restbase-ssl,name=restbaseNNNN.DC.wmnet"  set/pooled=yes</code>
** <code>confctl select "dc=DC,cluster=restbase,service=restbase-backend,name=restbaseNNNN.DC.wmnet"  set/pooled=yes</code>  
* Verify that the hosts have been added and are healthy via [https://config-master.wikimedia.org/pybal/codfw/restbase-backend the pybal API]  
* Verify that the hosts have been added and are healthy via [https://config-master.wikimedia.org/pybal/codfw/restbase-backend the pybal API]  



Revision as of 12:19, 13 December 2021

RESTBase is an API proxy serving the REST API at /api/rest_v1/. It uses Cassandra as a storage backend.

It is currently running on hosts with the profile::restbase class.

Deployment and config changes

RESTBase is deployed by Scap.

What to check after a deploy

Deploys to do not always go according to plan, and regressions are not always obvious. Here is a list of things you should check after each deploy:

Other considerations

Be sure to log all actions ahead of time in #wikimedia-operations. Don't be shy about including details.

Administration

Adding a new RESTBase host

Before following these instructions, ensure you follow the provisioning documentation for a new Cassandra node.

Renewing expired certificates

Every now and again Cassandra certificates will come close to expiry (for example: SSL WARNING - Certificate restbase2016-a valid until 2020-11-29 09:26:14 +0000 (expires in 53 days)). Certificates need to be deleted and recreated in the Puppet secrets directory - See the Cassandra documentation for details.

Monitoring

instance-data

In production, the instance-data path is usually a RAID array. It is used for hints, commitlogs and caches - all vital to the stable operation of the Cassandra instances. Under unusual circumstances (a large rebalancing, an instance behaving erroneously etc) this mount can fill up quickly and space will sometimes be required to back out of this condition. For this reason, we set a lower threshold for disk free on this path than for other disks.

Debugging

To temporarily switch to local logging for debugging, you can change the config.yaml log stanza like this:

logging:
  name: restbase
  streams:
    # level can be trace, debug, info, warn, error
    - level: info 
      path: /tmp/debug.log

Alternatively, you can log to stdout by commenting out the streams sub-object. This is useful for debugging startup failures like this:

cd /srv/deployment/restbase/deploy/
sudo -u restbase node restbase/server.js -c /etc/restbase/config.yaml -n 0

The -n 0 parameter avoids forking off any workers, which reduces log noise. Instead, a single worker is started up right in the master process.

Analytics and metrics

Hive query for action API & rest API traffic:

use wmf;

SELECT
  SUM(IF (uri_path LIKE '/api/rest_v1/%', 1, 0)) as count_rest,
  SUM(IF (uri_path LIKE '/w/api.php%', 1, 0)) as count_action
FROM wmf.webrequest
WHERE webrequest_source = 'text'
  AND year = 2017
  AND month = 9
  AND (uri_path LIKE '/api/rest_v1/%' OR uri_path LIKE '/w/api.php%');