Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> decompressing bzip2 data with a custom InputFormat


+
Tony Burton 2012-03-12, 18:04
Copy link to this message
-
RE: decompressing bzip2 data with a custom InputFormat
Hi - sorry to bump this, but I'm having trouble resolving this.

Essentially the question is: If I create my own InputFormat by subclassing TextInputFormat, does the subclass have to handle its own streaming of compressed data? If so, can anyone point me at an example where this is done?

Thanks!

Tony

-----Original Message-----
From: Tony Burton [mailto:[EMAIL PROTECTED]]
Sent: 12 March 2012 18:05
To: [EMAIL PROTECTED]
Subject: decompressing bzip2 data with a custom InputFormat

 Hi,

I'm setting up a map-only job that reads large bzip2-compressed data files, parses the XML and writes out the same data in plain text format. My XML InputFormat extends TextInputFormat and has a RecordReader based upon the one you can see at http://xmlandhadoop.blogspot.com/ (my version of it works great for uncompressed XML input data). For compressed data, I've added io.compression.codecs to my core-site.xml and set it to o.a.h.io.compress.BZip2Codec. I'm using Hadoop 0.20.2.

Have I forgotten something basic when running a Hadoop job to read compressed data? Or, given that I've written my own InputFormat, should I be using an InputStream that can carry out the decompression itself?

Thanks

Tony
 
**********************************************************************
This email and any attachments are confidential, protected by copyright and may be legally privileged.  If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.  Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened.  It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email.  Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Brookfield House, Green Lane, Ivinghoe, Leighton Buzzard, LU7 9ES.  Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.

Outbound email has been scanned for viruses and SPAM
www.sportingindex.com
Inbound Email has been scanned for viruses and SPAM
**********************************************************************

This email and any attachments are confidential, protected by copyright and may be legally privileged.  If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.  Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened.  It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email.  Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Brookfield House, Green Lane, Ivinghoe, Leighton Buzzard, LU7 9ES.  Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.

Outbound email has been scanned for viruses and SPAM
+
Joey Echeverria 2012-03-14, 19:02
+
Tony Burton 2012-03-16, 10:05
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB