You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Switch Datacenter/Parsercache warmup

From Wikitech-static
Jump to navigation Jump to search
  • Download the list of titles from the dumps, for example, enwiki-20160305-all-titles-in-ns0
  • Then execute from a local mediawiki node:
while true; do line=$(( ( `od -A n -t d -N 4 /dev/urandom` + 2147483648 ) % 12307009)); title=`tail -n+$line enwiki-20160305-all-titles-in-ns0 | head -n1`; echo -e "$title"; /usr/bin/time -f %E curl -s -H 'Host: en.wikipedia.org' "http://localhost/wiki/$title" | wc -c; done
  • For api usage, execute:
while true; do line=$(( ( `od -A n -t d -N 4 /dev/urandom` + 2147483648 ) % 12307009)); title=`tail -n+$line enwiki-20160305-all-titles-in-ns0 | head -n1`; echo -e "$title"; /usr/bin/time -f %E curl -s -H 'Host: en.wikipedia.org' "http://localhost/w/api.php?action=parse&prop=text&page=$title&format=json" | wc -c; done

Of course, it can be parallelized for faster warmup. The difference between parsecache/memcached-stored GETs and full parsing can be as high as 10 seconds vs. 100 miliseconds.