Data Platform/Systems/Hive/Compression

Uncompressed vs. Snappy compressed Sequence Files

I just ran some rough comparisons of data sizes and Hive queries of webrequest data stored in HDFS uncompressed vs. as Snappy compressed Sequence Files.

Uncompressed

Size

In the first 8 hours of January 7th, 2014, uncompressed JSON formatted mobile webrequest logs imported into HDFS via Kafka totaled 91.1 GB. Each hourly import was between 8 and 12 GB each.

hdfs dfs -du -s -h /wmf/data/external/webrequest_mobile/hourly/2014/01/07/{00..08}
10.6 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/00
10.9 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/01
11.3 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/02
11.7 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/03
10.6 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/04
9.9 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/05
9.1 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/06
8.6 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/07
8.4 G  /wmf/data/external/webrequest_mobile/hourly/2014/01/07/08

91.1 G

Query Time

Running a select count(*) query on a single hour took 44.276 seconds, and launched 42 mappers. Running the same query on 8 hours of data took 158.627 seconds and launched 343 mappers.

-- 
select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour=00;
...
MapReduce Total cumulative CPU time: 16 minutes 11 seconds 670 msec
Ended Job = job_1387838787660_0365
MapReduce Jobs Launched:
Job 0: Map: 42  Reduce: 1   Cumulative CPU: 971.67 sec   HDFS Read: 11422909543 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 11 seconds 670 msec
OK
_c0
16641115
Time taken: 44.276 seconds

select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour between 00 and 08;
...
MapReduce Total cumulative CPU time: 0 days 4 hours 16 minutes 12 seconds 420 msec
Ended Job = job_1387838787660_0363
MapReduce Jobs Launched:
Job 0: Map: 343  Reduce: 1   Cumulative CPU: 15372.42 sec   HDFS Read: 98055786272 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 0 days 4 hours 16 minutes 12 seconds 420 msec
OK
_c0
143199253
Time taken: 158.627 seconds

Snappy compressed Sequence Files

I recently got SequenceFileRecordWriterProvider.java merged upstream in LinkedIn's Camus. Using this rather than StringRecordWriterProvider.java writes out the same data as Snappy compressed Hadoop Sequence Files .

Size

JSON data imported for the same 8 hour period and Snappy compressed was 21.9 GB, 24% of the original size.

hdfs dfs -du -s -h /user/otto/data/compressed/webrequest_mobile/hourly/2014/01/07/{00..08}

2.5 G  data/compressed/webrequest_mobile/hourly/2014/01/07/00
2.6 G  data/compressed/webrequest_mobile/hourly/2014/01/07/01
2.7 G  data/compressed/webrequest_mobile/hourly/2014/01/07/02
2.8 G  data/compressed/webrequest_mobile/hourly/2014/01/07/03
2.5 G  data/compressed/webrequest_mobile/hourly/2014/01/07/04
2.4 G  data/compressed/webrequest_mobile/hourly/2014/01/07/05
2.2 G  data/compressed/webrequest_mobile/hourly/2014/01/07/06
2.1 G  data/compressed/webrequest_mobile/hourly/2014/01/07/07
2.1 G  data/compressed/webrequest_mobile/hourly/2014/01/07/08

21.9 G

Query Time

The same select count(*) query on a single hour of compressed data took 86.232 seconds, about twice as long as on uncompressed data. Running the query on 8 hours worth of compressed data took 158.627 seconds, which is only 8% longer than when run on 8 hours of uncompressed data. The number of mappers launched was the same as in the uncompressed case.

select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour=00;
...
MapReduce Total cumulative CPU time: 16 minutes 11 seconds 670 msec
Ended Job = job_1387838787660_0365
MapReduce Jobs Launched:
Job 0: Map: 42  Reduce: 1   Cumulative CPU: 971.67 sec   HDFS Read: 11422909543 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 11 seconds 670 msec
OK
_c0
16641115
Time taken: 44.276 seconds

select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour between 00 and 08;
...
MapReduce Total cumulative CPU time: 0 days 4 hours 16 minutes 12 seconds 420 msec
Ended Job = job_1387838787660_0363
MapReduce Jobs Launched:
Job 0: Map: 343  Reduce: 1   Cumulative CPU: 15372.42 sec   HDFS Read: 98055786272 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 0 days 4 hours 16 minutes 12 seconds 420 msec
OK
_c0
143199253
Time taken: 158.627 seconds

Summary

Using Snappy to compress the JSON webrequest logs results in significant space savings, and only a slight reduction in performance for large queries. Query performance is affected for smaller data sets. I will run another test once I have more data to compare (a month), but if results are approximately the same I will not update this page.

Recommendation: use snappy compression for all webrequest imports.