Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> RE: hadoop 1.0.3 equivalent of MultipleTextOutputFormat


Copy link to this message
-
RE: hadoop 1.0.3 equivalent of MultipleTextOutputFormat
Thanks Harsh! Will try it out and report back later.
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: 29 August 2012 11:12
To: [EMAIL PROTECTED]
Subject: Re: hadoop 1.0.3 equivalent of MultipleTextOutputFormat

Hi Tony,

Seeing your new question, I recalled Tom's post to a user once, here:
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pdyVyydt5Ys/1CaLukt4v1AJ

This specific call allows you to specify / characters in your name,
that gets translated into creation of directories automatically:
http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html#write(KEYOUT,%20VALUEOUT,%20java.lang.String)
(The last argument is where you will need to specify the path)

Try it out and let us know!

On Tue, Aug 28, 2012 at 7:06 PM, Tony Burton <[EMAIL PROTECTED]> wrote:
> Hi Harsh
>
> Thanks for the reply - my understanding is that with MultipleOutputs I can write differently named files into the same target directory. With MultipleTextOutputFormat I was able to override the target directory name to perform the segmentation, by overriding generateFileNameForKeyValue().
>
> Does the 1.0.3 MultipleOutputs give me the ability to alter the target directory name as well as the file name?
>
> Thanks,
>
> Tony
>
>
>
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: 28 August 2012 13:44
> To: [EMAIL PROTECTED]
> Subject: Re: hadoop 1.0.3 equivalent of MultipleTextOutputFormat
>
> The Multiple*OutputFormat have been deprecated in favor of the generic
> MultipleOutputs API. Would using that instead work for you?
>
> On Tue, Aug 28, 2012 at 6:05 PM, Tony Burton <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I've seen that org.apache.hadoop.mapred.lib.MultipleTextOutputFormat is good for writing results into (for example) different directories created on the fly. However, now I'm implementing a MapReduce job using Hadoop 1.0.3, I see that the new API no longer supports MultipleTextOutputFormat. Is there an equivalent that I can use, or will it be supported in a future release?
>>
>> Thanks,
>>
>> Tony
>>
>>
>> **********************************************************************
>> This email and any attachments are confidential, protected by copyright and may be legally privileged.  If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.  Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened.  It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email.  Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP.  Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001).  Any financial promotion contained herein has been issued
>> and approved by Sporting Index Ltd.
>>
>> Outbound email has been scanned for viruses and SPAM
>>
>
>
>
> --
> Harsh J
> www.sportingindex.com
> Inbound Email has been scanned for viruses and SPAM
> **********************************************************************
> This email and any attachments are confidential, protected by copyright and may be legally privileged.  If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.  Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened.  It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email.  Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP.  Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001).  Any financial promotion contained herein has been issued

Harsh J
www.sportingindex.com
Inbound Email has been scanned for viruses and SPAM
**********************************************************************

This email and any attachments are confidential, protected by copyright and may be legally privileged.  If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.  Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened.  It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email.  Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP.  Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001).  Any financial promotion contained herein has been issued
and approved by Sporting I
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB