Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Re: YARN hangs while computing pig?


Copy link to this message
-
Re: YARN hangs while computing pig?
Mark Grover 2012-11-29, 18:11
Redirecting to Apache pig user list

On Thu, Nov 29, 2012 at 1:01 AM, Johnny Kowalski
<[EMAIL PROTECTED]>wrote:

> Another hint? Is is some permission issue?
>
> MY EXAMPLE:
> in = LOAD '/user/myUser/aTest' USING PigStorage();
> DUMP in;
>
> When I try to run above pig in grunt shell with "pig -x local" it
> successfully prints output.
> *
> And when I do it in mapreduce pig grundshell it hangs... on this line:"
>
> 2012-11-28 15:24:27,709 [main] INFO  org.apache.pig.backend.hadoop.
> *
> ***executionengine.****mapReduceLayer.****MapReduceLauncher - 0% complete"
> *
>
>
> W dniu środa, 28 listopada 2012 15:44:22 UTC+1 użytkownik Johnny Kowalski
> napisał:
>
>> Hi, after configuring whole stack I've got another issue. My pig jobs
>> hangs.
>> I've created completly new user added a dir for him
>> sudo -u hdfs hadoop fs -mkdir /user/newUser
>> sudo -u hdfs hadoop fs -chown newUser:newUser /user/newUser
>>
>> and wanted to run pig that runs fine on another yarn configuration
>> cluster. And got something like this:
>>
>>
>> PIG CONSOLE OUTPUT
>>
>> 2012-11-28 15:24:26,759 [Thread-4] INFO  org.apache.hadoop.mapreduce.**lib.input.FileInputFormat
>> - Total input paths to process : 1
>> 2012-11-28 15:24:26,760 [Thread-4] INFO  org.apache.pig.backend.hadoop.**
>> executionengine.util.**MapRedUtil - Total input paths to process : 1
>> 2012-11-28 15:24:26,783 [Thread-4] INFO  org.apache.pig.backend.hadoop.**
>> executionengine.util.**MapRedUtil - Total input paths (combined) to
>> process : 1
>> 2012-11-28 15:24:26,872 [Thread-4] INFO  org.apache.hadoop.mapreduce.**JobSubmitter
>> - number of splits:1
>> 2012-11-28 15:24:26,889 [Thread-4] WARN  org.apache.hadoop.conf.**Configuration
>> - fs.default.name is deprecated. Instead, use fs.defaultFS
>> 2012-11-28 15:24:26,890 [Thread-4] WARN  org.apache.hadoop.conf.**Configuration
>> - mapreduce.job.counters.limit is deprecated. Instead, use
>> mapreduce.job.counters.max
>> 2012-11-28 15:24:26,891 [Thread-4] WARN  org.apache.hadoop.conf.**Configuration
>> - mapred.job.tracker is deprecated. Instead, use
>> mapreduce.jobtracker.address
>> 2012-11-28 15:24:26,892 [Thread-4] WARN  org.apache.hadoop.conf.**Configuration
>> - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
>> 2012-11-28 15:24:26,893 [Thread-4] WARN  org.apache.hadoop.conf.**Configuration
>> - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
>> 2012-11-28 15:24:26,894 [Thread-4] WARN  org.apache.hadoop.conf.**Configuration
>> - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
>> 2012-11-28 15:24:27,563 [Thread-4] INFO  org.apache.hadoop.mapred.**ResourceMgrDelegate
>> - Submitted application application_1354100898821_0011 to ResourceManager
>> at userver/192.168.56.101:8032
>> 2012-11-28 15:24:27,629 [Thread-4] INFO  org.apache.hadoop.mapreduce.**Job
>> - The url to track the job: http://userver:8088/proxy/**
>> application_1354100898821_**0011/<http://userver:8088/proxy/application_1354100898821_0011/>
>> 2012-11-28 15:24:27,709 [main] INFO  org.apache.pig.backend.hadoop.**
>> executionengine.**mapReduceLayer.**MapReduceLauncher - 0% complete
>>
>> Here I've noticed logs keep growing while mine job is in status RUNNING
>>
>> /var/log/hadoop-yarn# less hadoop-cmf-yarn1-NODEMANAGER-**userver.log.out
>>
>> 2012-11-28 15:39:16,227 INFO org.apache.hadoop.yarn.server.**nodemanager.
>> **NodeStatusUpdaterImpl: Sending out status for container: container_id
>> {, app_attempt_id {, application_id {, id: 11, cluster_timestamp:
>> 1354100898821, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics:
>> "", exit_status: -1000,
>> 2012-11-28 15:39:17,230 INFO org.apache.hadoop.yarn.server.**nodemanager.
>> **NodeStatusUpdaterImpl: Sending out status for container: container_id
>> {, app_attempt_id {, application_id {, id: 11, cluster_timestamp:
>> 1354100898821, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics:
>> "", exit_status: -1000,
>> 2012-11-28 15:39:18,233 INFO org.apache.hadoop.yarn.server.**nodemanager.