Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Job setup for a pig run takes ages


Copy link to this message
-
Re: Job setup for a pig run takes ages
Dmitriy Ryaboy 2012-06-18, 23:26
Can you do a few successive ones?
Also, please let us know which version of pig you are using, and which loaders.

D

On Mon, Jun 18, 2012 at 2:51 PM, Danfeng Li <[EMAIL PROTECTED]> wrote:
> This is the jstack output during the setup time, not exactly sure how to interoperate it.
>
> Thanks.
> Dan
>
> [dli@hmaster run]$ jstack 15640
> 2012-06-18 17:32:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x0000000055dcb800 nid=0x431d waiting on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Low Memory Detector" daemon prio=10 tid=0x0000000055105000 nid=0x3d3b runnable [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x0000000055103000 nid=0x3d3a waiting on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x0000000055100000 nid=0x3d39 waiting on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00000000550fe000 nid=0x3d38 runnable [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00000000550de800 nid=0x3d37 in Object.wait() [0x0000000041d7a000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab48a3cf8> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>        - locked <0x00002aaab48a3cf8> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00000000550dc800 nid=0x3d36 in Object.wait() [0x0000000041093000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab48a3cb0> (a java.lang.ref.Reference$Lock)
>        at java.lang.Object.wait(Object.java:485)
>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>        - locked <0x00002aaab48a3cb0> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x0000000055065800 nid=0x3d25 runnable [0x0000000041653000]
>   java.lang.Thread.State: RUNNABLE
>        at org.apache.pig.newplan.logical.expression.ProjectExpression.getFieldSchema(ProjectExpression.java:164)
>        at org.apache.pig.newplan.logical.relational.LOInnerLoad.getSchema(LOInnerLoad.java:59)
>        at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
>        at org.apache.pig.newplan.logical.relational.LOInnerLoad.accept(LOInnerLoad.java:109)
>        at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>        at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:94)
>        at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:71)
>        at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
>        at org.apache.pig.newplan.logical.optimizer.SchemaPatcher.transformed(SchemaPatcher.java:43)
>        at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:113)
>        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
>        at org.apache.pig.PigServer.compilePp(PigServer.java:1365)
>        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1207)
>        at org.apache.pig.PigServer.execute(PigServer.java:1201)
>        at org.apache.pig.PigServer.access$100(PigServer.java:129)
>        at org.apache.pig.PigServer$Graph.execute(PigServer.java:1528)
>        at org.apache.pig.PigServer.executeBatchEx(PigServer.java:373)
>        at org.apache.pig.PigServer.executeBatch(PigServer.java:340)