You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
I didn't see a period in October with 2.8T/month growth in thumb use (which would be 90+GB/day, right?) (cause I couldn't actually divide properly.)
Here's my notes.
I've been collecting some numbers so we can make some guesses about ms5 running out of space.
Rates of thumb space fill vary from 8 to 9 GB a day in August and September to 30 GB / day in early October and November, to 60 GB / day in mid November and 90 GB / day in mid October.
I've been running a job which cleans up thumbs of images not in use on our projects. They can of course be in use elsewhere, and some of them are, because I checked and about 1/4 of them are recreated. Over the past three days I've see us reclaim about 78 GB / day from that job. At the rate I run it, trying not to overload either ms5 or the scalers, it will take about another 14 days to complete, so we'll be done on Dec 20 or so.
At this moment (Dec 6 4:30 pm EET) we have 8347 of 11012 GB in use, for about 2260 GB left (we can't really use every last byte but let's say we could). On the 20th, if the numbers hold up, we'll have another 78*14 = 1090 GB free; adding in about 400GB that we still have free in the LVM, that gives us about 3750 GB free.
So when do we run out? Best case: 3750/9 = 416 days, worst case: 3750/90 = 41 days, middle of the road: 3750/40 = 93 days after the end of the run, i.e. from Dec 20. I don't see us returning to the August/September rates of use, I'd guess 80 or 85 days left from Dec 20, or around mid March. But that's using *all* available space.
That puts us right at mid-March as the Serious Drop Dead Date. Out Of Room, no ifs ands or bytes.
I can watch these trends and warn if we see big shifts. I'll try to get better data about the spike in thumbs generation space use for Oct/Nov in the next couple of days, it doesn't seem to correlate directly with uploads.
this was my method of getting to a 2.8T/mo burn rate during October:
(actually I just held a ruler up to the screen, but you get the idea)
- So what I did was generate custom graphs with ganglia3 for the parts of the graph I wanted, then looked at min-max values. Example: Oct 21 8 am through Oct 31 8 am, 2.17k to 1.27k = 2170 gb to 1270 gb, difference of ...dammmit. So I can't actually multiply or divide. Bah. I had 9 instead of 90. And may I point out that I was a math major? I'll fix it above. -- ArielGlenn 18:24, 6 December 2011 (UTC)
More info: a random 1/256 of the thumb space, i.e. 0/00, had about 470k files last I looked; that makes for around 120 million thumbs total, and we'll have more by the time we need to do the move. How long is the copy to Swift going to take for those? -- ArielGlenn 18:10, 8 December 2011 (UTC)