You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Netflow

From Wikitech-static
Jump to navigation Jump to search

High level description on https://en.wikipedia.org/wiki/NetFlow

Goal

Gather network level (Layer 4) traffic flows metadata to assist with traffic engineering and DoS mitigation.

How does it work?

Netflow diagram.
Netflow architecture

On the routers:

  • 1 out of 1000 flows crossing the routers' external interfaces (both inbound and outbound) gets its metadata sent to a configured collector once the flow timeout is reached (here 10s)
    • Example metadata are: source/dest IP/port/AS#, IP protocol, TCP flag...
  • The routers share their full BGP view with the collector

On the collectors:

  • Samplicator duplicates the IPFIX packets to Fastnetmon and Pmacct, while spoofing the source IP (so they still seem to come from the routers)
  • Pmacct (nfacct) extrapolates the flow size and packets based on the sampling rate (eg. do *1000)
  • Pmacct uses a prefix list (exported from Puppet) to enrich the collected flows with traffic direction
  • Pmacct uses the BGP data provided by the routers to enrich the collected flows metadata (adds peer src/dst AS#, AS path, src/dst AS#)
  • Pmacct uses an IP to location database to enrich the collected flows metadata (adds source and destination country)
  • Pmacct exports the enriched flow data to Druid via Kafka
  • Fastnetmon monitors inbound traffic for both known attack patterns and traffic level threshold and if any condition is met:
    • sends a notification email including a traffic signature if able
    • Triggers our monitoring system

How to deploy?

  1. Apply role::netinsights to a server
  2. Configure sampling on the router
  3. Add a BGP session from router to collector

Troubleshooting

Check if pmacct is sending data to kafka

$ kafkacat -b kafka-jumbo1001.eqiad.wmnet -t netflow -C

Real time Fastnetmon dashboard

$ fastnetmon_client

Check the logs

Both Pmacct and Fastnetmon log to syslog, grep for nfacctd or fastnetmon

Detected attack details are logged in /var/log/fastnetmon_attacks/

Visualization

Limitations

  • Fastnetmon misreports attack type and protocol - T241374

Resources

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/services-ipfix-flow-template-flow-aggregation-configuring.html

https://github.com/pavel-odintsov/fastnetmon/

https://github.com/pmacct/

roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - T248394