Mohit Anchlia
2012-02-27, 22:18
Harsh J
2012-02-28, 05:53
Mohit Anchlia
2012-02-28, 06:07
madhu phatak
2012-02-28, 08:37
Subir S
2012-02-28, 08:49
Harsh J
2012-02-28, 13:41
-
Handling bad recordsMohit Anchlia 2012-02-27, 22:18
What's the best way to write records to a different file? I am doing xml
processing and during processing I might come accross invalid xml format.
Current I have it under try catch block and writing to log4j. But I think
it would be better to just write it to an output file that just contains
errors.
processing and during processing I might come accross invalid xml format.
Current I have it under try catch block and writing to log4j. But I think
it would be better to just write it to an output file that just contains
errors.
-
Re: Handling bad recordsHarsh J 2012-02-28, 05:53
Mohit,
Use the MultipleOutputs API:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
to have a named output of bad records. There is an example of use
detailed on the link.
On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> What's the best way to write records to a different file? I am doing xml
> processing and during processing I might come accross invalid xml format.
> Current I have it under try catch block and writing to log4j. But I think
> it would be better to just write it to an output file that just contains
> errors.
--
Harsh J
Use the MultipleOutputs API:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
to have a named output of bad records. There is an example of use
detailed on the link.
On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> What's the best way to write records to a different file? I am doing xml
> processing and during processing I might come accross invalid xml format.
> Current I have it under try catch block and writing to log4j. But I think
> it would be better to just write it to an output file that just contains
> errors.
--
Harsh J
-
Re: Handling bad recordsMohit Anchlia 2012-02-28, 06:07
Thanks that's helpful. In that example what is "A" and "B" referring to? Is
that the output file name?
mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Mohit,
>
> Use the MultipleOutputs API:
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> to have a named output of bad records. There is an example of use
> detailed on the link.
>
> On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> > What's the best way to write records to a different file? I am doing xml
> > processing and during processing I might come accross invalid xml format.
> > Current I have it under try catch block and writing to log4j. But I think
> > it would be better to just write it to an output file that just contains
> > errors.
>
>
>
> --
> Harsh J
>
that the output file name?
mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Mohit,
>
> Use the MultipleOutputs API:
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> to have a named output of bad records. There is an example of use
> detailed on the link.
>
> On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> > What's the best way to write records to a different file? I am doing xml
> > processing and during processing I might come accross invalid xml format.
> > Current I have it under try catch block and writing to log4j. But I think
> > it would be better to just write it to an output file that just contains
> > errors.
>
>
>
> --
> Harsh J
>
-
Re: Handling bad recordsmadhu phatak 2012-02-28, 08:37
Hi Mohit ,
A and B refers to two different output files (multipart name). The file
names will be seq-A* and seq-B*. Its similar to "r" in part-r-00000
On Tue, Feb 28, 2012 at 11:37 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
> Thanks that's helpful. In that example what is "A" and "B" referring to? Is
> that the output file name?
>
> mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
> mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
>
>
> On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
> > Mohit,
> >
> > Use the MultipleOutputs API:
> >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> > to have a named output of bad records. There is an example of use
> > detailed on the link.
> >
> > On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> > wrote:
> > > What's the best way to write records to a different file? I am doing
> xml
> > > processing and during processing I might come accross invalid xml
> format.
> > > Current I have it under try catch block and writing to log4j. But I
> think
> > > it would be better to just write it to an output file that just
> contains
> > > errors.
> >
> >
> >
> > --
> > Harsh J
> >
>
--
Join me at http://hadoopworkshop.eventbrite.com/
A and B refers to two different output files (multipart name). The file
names will be seq-A* and seq-B*. Its similar to "r" in part-r-00000
On Tue, Feb 28, 2012 at 11:37 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
> Thanks that's helpful. In that example what is "A" and "B" referring to? Is
> that the output file name?
>
> mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
> mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
>
>
> On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
> > Mohit,
> >
> > Use the MultipleOutputs API:
> >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> > to have a named output of bad records. There is an example of use
> > detailed on the link.
> >
> > On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> > wrote:
> > > What's the best way to write records to a different file? I am doing
> xml
> > > processing and during processing I might come accross invalid xml
> format.
> > > Current I have it under try catch block and writing to log4j. But I
> think
> > > it would be better to just write it to an output file that just
> contains
> > > errors.
> >
> >
> >
> > --
> > Harsh J
> >
>
--
Join me at http://hadoopworkshop.eventbrite.com/
-
Re: Handling bad recordsSubir S 2012-02-28, 08:49
Can multiple output be used with Hadoop Streaming?
On Tue, Feb 28, 2012 at 2:07 PM, madhu phatak <[EMAIL PROTECTED]> wrote:
> Hi Mohit ,
> A and B refers to two different output files (multipart name). The file
> names will be seq-A* and seq-B*. Its similar to "r" in part-r-00000
>
> On Tue, Feb 28, 2012 at 11:37 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks that's helpful. In that example what is "A" and "B" referring to?
> Is
> > that the output file name?
> >
> > mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
> > mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
> >
> >
> > On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> > > Mohit,
> > >
> > > Use the MultipleOutputs API:
> > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> > > to have a named output of bad records. There is an example of use
> > > detailed on the link.
> > >
> > > On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >
> > > wrote:
> > > > What's the best way to write records to a different file? I am doing
> > xml
> > > > processing and during processing I might come accross invalid xml
> > format.
> > > > Current I have it under try catch block and writing to log4j. But I
> > think
> > > > it would be better to just write it to an output file that just
> > contains
> > > > errors.
> > >
> > >
> > >
> > > --
> > > Harsh J
> > >
> >
>
>
>
> --
> Join me at http://hadoopworkshop.eventbrite.com/
>
On Tue, Feb 28, 2012 at 2:07 PM, madhu phatak <[EMAIL PROTECTED]> wrote:
> Hi Mohit ,
> A and B refers to two different output files (multipart name). The file
> names will be seq-A* and seq-B*. Its similar to "r" in part-r-00000
>
> On Tue, Feb 28, 2012 at 11:37 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks that's helpful. In that example what is "A" and "B" referring to?
> Is
> > that the output file name?
> >
> > mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
> > mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
> >
> >
> > On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> > > Mohit,
> > >
> > > Use the MultipleOutputs API:
> > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> > > to have a named output of bad records. There is an example of use
> > > detailed on the link.
> > >
> > > On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >
> > > wrote:
> > > > What's the best way to write records to a different file? I am doing
> > xml
> > > > processing and during processing I might come accross invalid xml
> > format.
> > > > Current I have it under try catch block and writing to log4j. But I
> > think
> > > > it would be better to just write it to an output file that just
> > contains
> > > > errors.
> > >
> > >
> > >
> > > --
> > > Harsh J
> > >
> >
>
>
>
> --
> Join me at http://hadoopworkshop.eventbrite.com/
>
-
Re: Handling bad recordsHarsh J 2012-02-28, 13:41
Subir,
No, not unless you use a specialized streaming library (pydoop, dumbo,
etc. for python, for example).
On Tue, Feb 28, 2012 at 2:19 PM, Subir S <[EMAIL PROTECTED]> wrote:
> Can multiple output be used with Hadoop Streaming?
>
> On Tue, Feb 28, 2012 at 2:07 PM, madhu phatak <[EMAIL PROTECTED]> wrote:
>
>> Hi Mohit ,
>> A and B refers to two different output files (multipart name). The file
>> names will be seq-A* and seq-B*. Its similar to "r" in part-r-00000
>>
>> On Tue, Feb 28, 2012 at 11:37 AM, Mohit Anchlia <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Thanks that's helpful. In that example what is "A" and "B" referring to?
>> Is
>> > that the output file name?
>> >
>> > mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
>> > mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
>> >
>> >
>> > On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >
>> > > Mohit,
>> > >
>> > > Use the MultipleOutputs API:
>> > >
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
>> > > to have a named output of bad records. There is an example of use
>> > > detailed on the link.
>> > >
>> > > On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]
>> >
>> > > wrote:
>> > > > What's the best way to write records to a different file? I am doing
>> > xml
>> > > > processing and during processing I might come accross invalid xml
>> > format.
>> > > > Current I have it under try catch block and writing to log4j. But I
>> > think
>> > > > it would be better to just write it to an output file that just
>> > contains
>> > > > errors.
>> > >
>> > >
>> > >
>> > > --
>> > > Harsh J
>> > >
>> >
>>
>>
>>
>> --
>> Join me at http://hadoopworkshop.eventbrite.com/
>>
--
Harsh J
No, not unless you use a specialized streaming library (pydoop, dumbo,
etc. for python, for example).
On Tue, Feb 28, 2012 at 2:19 PM, Subir S <[EMAIL PROTECTED]> wrote:
> Can multiple output be used with Hadoop Streaming?
>
> On Tue, Feb 28, 2012 at 2:07 PM, madhu phatak <[EMAIL PROTECTED]> wrote:
>
>> Hi Mohit ,
>> A and B refers to two different output files (multipart name). The file
>> names will be seq-A* and seq-B*. Its similar to "r" in part-r-00000
>>
>> On Tue, Feb 28, 2012 at 11:37 AM, Mohit Anchlia <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Thanks that's helpful. In that example what is "A" and "B" referring to?
>> Is
>> > that the output file name?
>> >
>> > mos.getCollector("seq", "A", reporter).collect(key, new Text("Bye"));
>> > mos.getCollector("seq", "B", reporter).collect(key, new Text("Chau"));
>> >
>> >
>> > On Mon, Feb 27, 2012 at 9:53 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >
>> > > Mohit,
>> > >
>> > > Use the MultipleOutputs API:
>> > >
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
>> > > to have a named output of bad records. There is an example of use
>> > > detailed on the link.
>> > >
>> > > On Tue, Feb 28, 2012 at 3:48 AM, Mohit Anchlia <[EMAIL PROTECTED]
>> >
>> > > wrote:
>> > > > What's the best way to write records to a different file? I am doing
>> > xml
>> > > > processing and during processing I might come accross invalid xml
>> > format.
>> > > > Current I have it under try catch block and writing to log4j. But I
>> > think
>> > > > it would be better to just write it to an output file that just
>> > contains
>> > > > errors.
>> > >
>> > >
>> > >
>> > > --
>> > > Harsh J
>> > >
>> >
>>
>>
>>
>> --
>> Join me at http://hadoopworkshop.eventbrite.com/
>>
--
Harsh J
