Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Job setup for a pig run takes ages


Copy link to this message
-
Re: Job setup for a pig run takes ages
Can you do a few successive ones?
Also, please let us know which version of pig you are using, and which loaders.

D

On Mon, Jun 18, 2012 at 2:51 PM, Danfeng Li <[EMAIL PROTECTED]> wrote:
> This is the jstack output during the setup time, not exactly sure how to interoperate it.
>
> Thanks.
> Dan
>
> [dli@hmaster run]$ jstack 15640
> 2012-06-18 17:32:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x0000000055dcb800 nid=0x431d waiting on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Low Memory Detector" daemon prio=10 tid=0x0000000055105000 nid=0x3d3b runnable [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x0000000055103000 nid=0x3d3a waiting on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x0000000055100000 nid=0x3d39 waiting on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00000000550fe000 nid=0x3d38 runnable [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00000000550de800 nid=0x3d37 in Object.wait() [0x0000000041d7a000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab48a3cf8> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>        - locked <0x00002aaab48a3cf8> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00000000550dc800 nid=0x3d36 in Object.wait() [0x0000000041093000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab48a3cb0> (a java.lang.ref.Reference$Lock)
>        at java.lang.Object.wait(Object.java:485)
>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>        - locked <0x00002aaab48a3cb0> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x0000000055065800 nid=0x3d25 runnable [0x0000000041653000]
>   java.lang.Thread.State: RUNNABLE
>        at org.apache.pig.newplan.logical.expression.ProjectExpression.getFieldSchema(ProjectExpression.java:164)
>        at org.apache.pig.newplan.logical.relational.LOInnerLoad.getSchema(LOInnerLoad.java:59)
>        at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
>        at org.apache.pig.newplan.logical.relational.LOInnerLoad.accept(LOInnerLoad.java:109)
>        at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>        at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:94)
>        at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:71)
>        at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
>        at org.apache.pig.newplan.logical.optimizer.SchemaPatcher.transformed(SchemaPatcher.java:43)
>        at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:113)
>        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
>        at org.apache.pig.PigServer.compilePp(PigServer.java:1365)
>        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1207)
>        at org.apache.pig.PigServer.execute(PigServer.java:1201)
>        at org.apache.pig.PigServer.access$100(PigServer.java:129)
>        at org.apache.pig.PigServer$Graph.execute(PigServer.java:1528)
>        at org.apache.pig.PigServer.executeBatchEx(PigServer.java:373)
>        at org.apache.pig.PigServer.executeBatch(PigServer.java:340)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB