You are browsing a read-only backup copy of Wikitech. The primary site can be found at

Performance/Graphite/Synthetic Instance

From Wikitech-static
< Performance
Revision as of 18:05, 21 October 2019 by imported>Phedenskog (Added security group)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Graphite for synthetic testing

We have our own instance of Graphite running outside of our environment to make it easy to add as many metrics as needed. You can see that metrics/data in our Grafana instance.


The instance run on AWS on an m4.xlarge instance with an extra volume. We use AWS since our agents that collects the data uses AWS and then we can use security groups to make sure only our instances can post data to the instance.

The the size of the instance was chosen because a big company that also runs Graphite use the same setup for their synthetic testing. We can change that in the future.

When you setup a new instance, you need to make sure it stores the data on a disk that don't belong to that instance. We have 200 gb extra running on an instance, setup using the official AWS documentation. Mount it and make sure it is automatically mounted after a reboot. The instance lives under /data/.

We run the official dockerized version of Graphite using a docker-compose file. To setup Graphite the way we want it, we need to setup five volumes/mappings.

  • whisper is where we store all the metrics
  • graphite.db is the database where Graphites annotations is stored
  • storage-schemas.conf configures how long time we want to store the metrics
  • storage-aggregation.conf configures how we want to aggregate metrics
  • carbon.conf is carbon/whisper setup, we have our own version because the default one has a very moderate number of new metrics created per minute.


version: "3"
        image: graphiteapp/graphite-statsd:1.1.5-12
            - "2003:2003"
            - "8080:80"
        restart: always
            - /data/whisper:/opt/graphite/storage/whisper
            - /data/graphite.db:/opt/graphite/storage/graphite.db
            - /home/ubuntu/graphite/storage-schemas.conf:/opt/graphite/conf/storage-schemas.conf
            - /home/ubuntu/graphite/storage-aggregation.conf:/opt/graphite/conf/storage-aggregation.conf
            - /home/ubuntu/graphite/carbon.conf:/opt/graphite/conf/carbon.conf


# Aggregation methods for whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds
#  [name]
#  pattern = <regex>
#  xFilesFactor = <float between 0 and 1>
#  aggregationMethod = <average|sum|last|max|min>
#  name: Arbitrary unique name for the rule
#  pattern: Regex pattern to match against the metric name
#  xFilesFactor: Ratio of valid data points required for aggregation to the next retention to occur
#  aggregationMethod: function to apply to data points for aggregation
pattern = \.min$
xFilesFactor = 0.1
aggregationMethod = min

pattern = \.max$
xFilesFactor = 0.1
aggregationMethod = max

pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

pattern = .*
xFilesFactor = 0.0
aggregationMethod = average


# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds.
#  [name]
#  pattern = regex
#  retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...

# Carbon's internal metrics. This entry should match what is specified in
pattern = ^carbon\.
retentions = 60:1d

pattern = ^sitespeed_io\.
retentions = 30m:60d

pattern = .*
retentions = 30m:60d

Security group

The instance has it own security group that make sure we only get data from our blessed instances. The Inbound group looks like this:

Custom TCP 8080 - Access from

Custom TCP 8080 - Access from the agents security group that runs (for annotations)

Custom TCP 2003 - Access from the agents security group that runs (for metrics)

SSH TCP 22 - Access only from the .pem file.