Does map-reduce work well with binary contents in the file? This
binary content is basically some CAD files and map reduce program need
to read these files using some proprietry tool extract values and do
some processing. Wondering if there are others doing similar type of
processing. Best practices etc.

On Wed, 31 Aug 2011 08:44:42 -0700
Mohit Anchlia <[EMAIL PROTECTED]> wrote:

> Does map-reduce work well with binary contents in the file? This
> binary content is basically some CAD files and map reduce program need
> to read these files using some proprietry tool extract values and do
> some processing. Wondering if there are others doing similar type of
> processing. Best practices etc.

yes, it works.  you just need to select the right input format.
Personally i store all my binary files into a sequencefile (because my binary files are small)

Dieter

On Thu, Sep 1, 2011 at 1:25 AM, Dieter Plaetinck
<[EMAIL PROTECTED]> wrote:
> On Wed, 31 Aug 2011 08:44:42 -0700
> Mohit Anchlia <[EMAIL PROTECTED]> wrote:
>
>> Does map-reduce work well with binary contents in the file? This
>> binary content is basically some CAD files and map reduce program need
>> to read these files using some proprietry tool extract values and do
>> some processing. Wondering if there are others doing similar type of
>> processing. Best practices etc.
>
> yes, it works.  you just need to select the right input format.
> Personally i store all my binary files into a sequencefile (because my binary files are small)

Thanks! Is there a specific tutorial I can focus on to see how it could be done?
>
> Dieter
>

On Thu, Sep 1, 2011 at 8:37 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

Thanks! Is there a specific tutorial I can focus on to see how it could be
> done?
>

Take the word count example and change its output format to be
SequenceFileOutputFormat.

job.setOutputFormatClass(SequenceFileOutputFormat.class);

and it will generate SequenceFiles instead of text. There is
SequenceFileInputFormat for reading.

-- Owen

Mohit,

"Hadoop: The Definitive Guide" (Chapter 3 - Hadoop I/O) has a section on
SequenceFile and is worth reading.

http://oreilly.com/catalog/9780596521981

Thanks,
Praveen

On Thu, Sep 1, 2011 at 9:15 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> On Thu, Sep 1, 2011 at 8:37 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >wrote:
>
> Thanks! Is there a specific tutorial I can focus on to see how it could be
> > done?
> >
>
> Take the word count example and change its output format to be
> SequenceFileOutputFormat.
>
> job.setOutputFormatClass(SequenceFileOutputFormat.class);
>
> and it will generate SequenceFiles instead of text. There is
> SequenceFileInputFormat for reading.
>
> -- Owen
>