Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> decompressing bzip2 data with a custom InputFormat

Copy link to this message
decompressing bzip2 data with a custom InputFormat

I'm setting up a map-only job that reads large bzip2-compressed data files, parses the XML and writes out the same data in plain text format. My XML InputFormat extends TextInputFormat and has a RecordReader based upon the one you can see at http://xmlandhadoop.blogspot.com/ (my version of it works great for uncompressed XML input data). For compressed data, I've added io.compression.codecs to my core-site.xml and set it to o.a.h.io.compress.BZip2Codec. I'm using Hadoop 0.20.2.

Have I forgotten something basic when running a Hadoop job to read compressed data? Or, given that I've written my own InputFormat, should I be using an InputStream that can carry out the decompression itself?



This email and any attachments are confidential, protected by copyright and may be legally privileged.  If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.  Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened.  It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email.  Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Brookfield House, Green Lane, Ivinghoe, Leighton Buzzard, LU7 9ES.  Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.

Outbound email has been scanned for viruses and SPAM
Tony Burton 2012-03-14, 16:08
Joey Echeverria 2012-03-14, 19:02
Tony Burton 2012-03-16, 10:05