Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Question about MapReduce


Copy link to this message
-
Re: Question about MapReduce
Bertrand Dechoux 2012-11-14, 08:51
That would be "mapred.task.timeout".
You should download hadoop source, that way you would have the conf files
with all the default configuration.
(You could also get them online but lots of links are broken.)

And the name of the property is only half the story.
The error says explicitly that no report status was sent for this period
which is the reason for the killing.
If you do send a report status, you don't need to change the
mapred.task.timeout value which is already pretty high (IMHO/YMMV).

The way you can send a status would depend on whether you are using mapred
or mapreduce API.

Regards

Bertrand

PS : I cced this discussion to the non hbase specific user mailing list.
Replies could be done only on this done because it is a non HBase specific
question.

On Wed, Nov 14, 2012 at 3:41 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> One more question about MapReduce.
>
> One of my servers is slower than the others. I don't have any time
> constraint for the job to finish.
>
> But I'm getting this message:
>
> "Task attempt_201211122318_0014_m_000021_0 failed to report status for
> 601 seconds. Killing!"
>
> Where can I chance this timeout to something like 1800 seconds? Is it
> on the mapred-site.xml file? If so, which property should I insert?
>
> Thanks,
>
> JM
>
> 2012/11/2, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> > That was my initial plan too, but I was wondering if there was any
> > other best practice about the delete.  So I will go that way.
> >
> > Thanks,
> >
> > JM
> >
> > 2012/11/2, Shrijeet Paliwal <[EMAIL PROTECTED]>:
> >> Not sure what exactly is happening in your job. But in one of the delete
> >> jobs I wrote I was creating an instance of HTable in setup method of my
> >> mapper
> >>
> >> delTab = new HTable(conf, conf.get(TABLE_NAME));
> >>
> >> And performing delete in map() call using delTab. So no, you do not have
> >> access to table directly *usually*.
> >>
> >>
> >> -Shrijeet
> >>
> >>
> >> On Fri, Nov 2, 2012 at 12:47 PM, Jean-Marc Spaggiari <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >>> Sorry, one last question.
> >>>
> >>> On the map method, I have access to the row using the values
> >>> parameter. Now, based on the value content, I might want to delete it.
> >>> Do I have access to the table directly from one of the parameters? Or
> >>> should I call the delete using an HTableInterface from my pool?
> >>>
> >>> Thanks,
> >>>
> >>> JM
> >>>
> >>> 2012/11/2, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> >>> > Yep, you perfectly got my question.
> >>> >
> >>> > I just tried and it's working perfectly!
> >>> >
> >>> > Thanks a lot! I now have a lot to play with.
> >>> >
> >>> > JM
> >>> >
> >>> > 2012/11/2, Shrijeet Paliwal <[EMAIL PROTECTED]>:
> >>> >> JM,
> >>> >>
> >>> >> I personally would chose to put it neither hadoop libs nor hbase
> >>> >> libs.
> >>> >> Have
> >>> >> them go to your application's own install directory.
> >>> >>
> >>> >> Then you could sent the variable HADOOP_CLASSPATH to have your jar
> >>> >> (also
> >>> >> include hbase jars, hbase dependencies and dependencies your program
> >>> >> needs)
> >>> >> And to execute fire 'hadoop jar' command.
> >>> >>
> >>> >> An example[1]:
> >>> >>
> >>> >> Set classpath:
> >>> >> export HADOOP_CLASSPATH=`hbase
> >>> classpath`:mycool.jar:mycooldependency.jar
> >>> >>
> >>> >> Fire following to launch your job:
> >>> >> hadoop jar mycool.jar hbase.experiments.MyCoolProgram
> >>> >> -Dmapred.running.map.limit=50
> >>> >> -Dmapred.map.tasks.speculative.execution=false aCommandLineArg
> >>> >>
> >>> >>
> >>> >> Did I get your question right?
> >>> >>
> >>> >> [1] In the example I gave `hbase classpath` gets you set with all
> >>> >> hbase
> >>> >> jars.
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Fri, Nov 2, 2012 at 11:56 AM, Jean-Marc Spaggiari <
> >>> >> [EMAIL PROTECTED]> wrote:
> >>> >>
> >>> >>> Hi Shrijeet,
> >>> >>>
> >>> >>> Helped a lot! Thanks!
> >>> >>>
> >>> >>> Now, the only think I need is to know where's the best place to put

Bertrand Dechoux