You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

User:Jbond/debuging

From Wikitech-static
< User:Jbond
Revision as of 10:51, 3 February 2021 by imported>Jbond (Created page with "= Sampled-1000.json on centrallog1001 = === Grep-able oputput === <syntaxhighlight lang=console> $ jq -r "[.uri_path,.hostname,.user_agent,.ip] | @csv" /srv/log/webrequest/...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Sampled-1000.json on centrallog1001

Grep-able oputput

$ jq  -r "[.uri_path,.hostname,.user_agent,.ip] | @csv" /srv/log/webrequest/sampled-1000.json

Select all requests with a specific user_agent and .referer

$ jq -r 'if .user_agent == "-" and .referer == "-" then [.uri_path,.hostname,.user_agent,.ip] else empty end | @csv' /srv/log/webrequest/sampled-1000.json

List of the top 10 IPs by response size

$ head -n 2560000 /srv/log/webrequest/sampled-1000.json | jq -r '.ip + " " + (.response_size | tostring)' | awk '{ sum[$1] += $2 } END { for (ip in sum) print sum[ip],ip }' | sort -nr | head -10

mw server

list all ips which have made more the 100 large requests

$ awk '$2>60000 {print $11}' /var/log/apache2/other_vhosts_access.log | sort | uniq -c | awk '$1>100 {print}'

LVS Server

Sample 100k pkts and list top talkers

$ sudo tcpdump -i enp4s0f0 -pn -c 100000 | sed -r 's/.* IP6? //;s/\.[^\.]+ .*//' | sort | uniq -c | sort -nr | head -20