Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Help with PIG 0.7 and JOINs


Copy link to this message
-
Re: Help with PIG 0.7 and JOINs
You would find the log for MR jobs that failed either at -
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
Or
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119

You can forward the error message if you need further help with this.

-Thejas

On 8/20/10 4:22 PM, "Raman Yakkala" <[EMAIL PROTECTED]> wrote:

Hi Guys,

I am trying to do join tow data sets and the jobs are failing. There are
some warnings reported and I am not good at understanding them. I am seeking
your help in adjusting any parameters to the job so that the job might
succeed.

Here is the grunt shell output:

grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
USING PigStorage();
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
odl: $1, $2, $4, $5, $6, $8
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for odl
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
$19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
$36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
$53, $55
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for oil
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for qif
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for qif
2010-08-20 14:43:47,221 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
- 1-799 Operator Key: 1-799)
*2010-08-20 14:43:47,269 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
2010-08-20 14:43:47,274 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning MULTI_LEAF_MAP 1 time(s).
*2010-08-20 14:43:47,284 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2010-08-20 14:43:47,333 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:48,645 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:48,715 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:49,884 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:49,896 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 2 map-reduce job(s) waiting for submission.
*2010-08-20 14:43:49,902 [Thread-13] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
*2010-08-20 14:43:50,398 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-20 14:43:51,082 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 5800
2010-08-20 14:44:02,525 [Thread-13] WARN  org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 14:44:02,871 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 950
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7120
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
2010-08-20 14:48:41,613 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 20% complete
2010-08-20 14:52:11,487 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2010-08-20 14:56:47,404 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 38% complete
2010-08-20 14:58:45,888 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 42% complete
2010-08-20 15:09:23,195 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 15:09:24,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 15:09:24,344 [main] INFO
org.apach
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB