You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Graphite for synthetic testing
We have our own instance of Graphite running outside of our environment to make it easy to add as many metrics as needed. You can see that metrics/data in our Grafana instance.
The instance run on AWS on an m4.xlarge instance with an extra volume. We use AWS since our agents that collects the data uses AWS and then we can use security groups to make sure only our instances can post data to the instance.
The the size of the instance was chosen because a big company that also runs Graphite use the same setup for their synthetic testing. We can change that in the future.
When you setup a new instance, you need to make sure it stores the data on a disk that don't belong to that instance. We have 200 gb extra running on an instance, setup using the official AWS documentation. Mount it and make sure it is automatically mounted after a reboot. The instance lives under /data/.
We run the official dockerized version of Graphite using a docker-compose file. To setup Graphite the way we want it, we need to setup five volumes/mappings.
- whisper is where we store all the metrics
- graphite.db is the database where Graphites annotations is stored
- storage-schemas.conf configures how long time we want to store the metrics
- storage-aggregation.conf configures how we want to aggregate metrics
- carbon.conf is carbon/whisper setup, we have our own version because the default one has a very moderate number of new metrics created per minute.
version: "3" services: graphite: image: graphiteapp/graphite-statsd:1.1.5-12 ports: - "2003:2003" - "8080:80" restart: always volumes: - /data/whisper:/opt/graphite/storage/whisper - /data/graphite.db:/opt/graphite/storage/graphite.db - /home/ubuntu/graphite/storage-schemas.conf:/opt/graphite/conf/storage-schemas.conf - /home/ubuntu/graphite/storage-aggregation.conf:/opt/graphite/conf/storage-aggregation.conf - /home/ubuntu/graphite/carbon.conf:/opt/graphite/conf/carbon.conf
# Aggregation methods for whisper files. Entries are scanned in order, # and first match wins. This file is scanned for changes every 60 seconds # # [name] # pattern = <regex> # xFilesFactor = <float between 0 and 1> # aggregationMethod = <average|sum|last|max|min> # # name: Arbitrary unique name for the rule # pattern: Regex pattern to match against the metric name # xFilesFactor: Ratio of valid data points required for aggregation to the next retention to occur # aggregationMethod: function to apply to data points for aggregation # [min] pattern = \.min$ xFilesFactor = 0.1 aggregationMethod = min [max] pattern = \.max$ xFilesFactor = 0.1 aggregationMethod = max [sum] pattern = \.count$ xFilesFactor = 0 aggregationMethod = sum [default_average] pattern = .* xFilesFactor = 0.0 aggregationMethod = average
# Schema definitions for Whisper files. Entries are scanned in order, # and first match wins. This file is scanned for changes every 60 seconds. # # [name] # pattern = regex # retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ... # Carbon's internal metrics. This entry should match what is specified in # CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings [carbon] pattern = ^carbon\. retentions = 60:1d [sitespeed] pattern = ^sitespeed_io\. retentions = 30m:60d [cath_them_all] pattern = .* retentions = 30m:60d
The instance has it own security group that make sure we only get data from our blessed instances. The Inbound group looks like this:
Custom TCP 8080 - Access from the proxy (install1002) that is used by grafana.wikimedia.org
Custom TCP 8080 - Access from the agents security group that runs sitespeed.io/browsertime/webpagetest (for annotations)
Custom TCP 2003 - Access from the agents security group that runs sitespeed.io/browsertime/webpagetest (for metrics)
SSH TCP 22 - Access only from the .pem file.