Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Modify the number of map tasks


Copy link to this message
-
Re: Modify the number of map tasks
Mohammad Tariq 2012-12-12, 13:22
I wonder how you are able to run the job without a JT. You must have this
on your mapred-site.xml file :
        <property>
<name>mapred.job.tracker</name>
 <value>localhost:9001</value>
</property>

Also add "hadoop.tmp.dir" in core-site.xml, and "dfs.name.dir" &
"dfs.data.dir" in hdfs-site.xml.

Regards,
    Mohammad Tariq

On Wed, Dec 12, 2012 at 6:46 PM, imen Megdiche <[EMAIL PROTECTED]>wrote:

> For mapred-site.xml :
>
> <configuration>
>
> <property>
> <name>mapred.map.tasks</name>
> <value>6</value>
> </property>
>
> </configuration>
>
> for core-site.xml :
> <configuration>
>
> <!-- <property>
> <name>fs.default.name</name>
> <value>hdfs://localhost:9100</value>
> </property> -->
>
> </configuration>
>
>  on hdfs-site.xml  nothing
>
>
>
>
>
> 2012/12/12 Mohammad Tariq <[EMAIL PROTECTED]>
>
>> Can I have a look at your config files?
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Wed, Dec 12, 2012 at 6:31 PM, imen Megdiche <[EMAIL PROTECTED]>wrote:
>>
>>> i run the start-all.sh and all daemons starts without problems. But i
>>> the log of the tasktracker look like this :
>>>
>>>
>>> 2012-12-12 13:53:45,495 INFO org.apache.hadoop.mapred.TaskTracker:
>>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting TaskTracker
>>> STARTUP_MSG:   host = megdiche-OptiPlex-GX280/127.0.1.1
>>> STARTUP_MSG:   args = []
>>> STARTUP_MSG:   version = 1.0.4
>>> STARTUP_MSG:   build >>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2012-12-12 13:53:47,009 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2012-12-12 13:53:47,331 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2012-12-12 13:53:47,336 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2012-12-12 13:53:47,336 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>>> system started
>>> 2012-12-12 13:53:48,165 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2012-12-12 13:53:48,192 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2012-12-12 13:53:48,513 ERROR org.apache.hadoop.mapred.TaskTracker: Can
>>> not start task tracker because java.lang.IllegalArgumentException: Does not
>>> contain a valid host:port authority: local
>>>     at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
>>>     at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
>>>     at
>>> org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2560)
>>>     at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1426)
>>>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
>>>
>>> 2012-12-12 13:53:48,519 INFO org.apache.hadoop.mapred.TaskTracker:
>>> SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down TaskTracker at megdiche-OptiPlex-GX280/
>>> 127.0.1.1
>>> ************************************************************/
>>>
>>>
>>>
>>>
>>> 2012/12/12 Mohammad Tariq <[EMAIL PROTECTED]>
>>>
>>>> I would check if all the daemons are running properly or not, before
>>>> anything else. If some problem is found, next place to track is the log of
>>>> each daemon.
>>>>
>>>> The correct command to check the status of a job from command line is :
>>>> hadoop job -status jobID.
>>>> (Mind the 'space' after job and remove 'command' from the statement)
>>>>
>>>> HTH
>>>>
>>>> Regards,
>>>>     Mohammad Tariq
>>>>
>>>>
>>>>
>>>> On Wed, Dec 12, 2012 at 6:14 PM, imen Megdiche <[EMAIL PROTECTED]
>>>> > wrote:
>>>>
>>>>> My goal is to analyze the response time of MapReduce depending on the size