Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to figure out the range of a split that failed?


Copy link to this message
-
Re: how to figure out the range of a split that failed?
Dear Sharad,

Oh my God, thank you Sharad. You are my savior. Though the example you've
given me is not the exact Hadoop Stream I was looking for, it sure shed
light on my problem.
Thanks again!!!
And for the people who are wondering how to enable SkipBadRecords feature in
Hadoop Streaming, refer to this site:
http://hadoop.apache.org/common/docs/current/mapred-default.html
Search for "mapred.skip.attempts.to.start.skipping" and you will get the
answer.

Sincerely, Ed

2010/6/30 Sharad Agarwal <[EMAIL PROTECTED]>

> edward choi wrote:
>
>> Thanks for the quick response.
>> I know the SkipBadRecords feature but unfortunately I cannot use it since
>> I
>> am running my job on Hadoop Streaming.
>> I had asked if there were any way to use SkipBadRecords in Hadoop
>> Streaming
>> but never got an answer. I guess it is not possible at all.
>> Thanks for your concern.
>>
>>
> SkipBadRecords feature can be used for streaming as well. Perhaps the best
> example is the testcase
> ->
> http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java?view=markup
>
> Sharad
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB