-
Re: Help with PIG 0.7 and JOINsThejas M Nair 2010-08-20, 23:40
You would find the log for MR jobs that failed either at -
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
Or
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
You can forward the error message if you need further help with this.
-Thejas
On 8/20/10 4:22 PM, "Raman Yakkala" <[EMAIL PROTECTED]> wrote:
Hi Guys,
I am trying to do join tow data sets and the jobs are failing. There are
some warnings reported and I am not good at understanding them. I am seeking
your help in adjusting any parameters to the job so that the job might
succeed.
Here is the grunt shell output:
grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
USING PigStorage();
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
odl: $1, $2, $4, $5, $6, $8
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for odl
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
$19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
$36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
$53, $55
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for oil
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for qif
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for qif
2010-08-20 14:43:47,221 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
- 1-799 Operator Key: 1-799)
*2010-08-20 14:43:47,269 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
2010-08-20 14:43:47,274 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning MULTI_LEAF_MAP 1 time(s).
*2010-08-20 14:43:47,284 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2010-08-20 14:43:47,333 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:48,645 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:48,715 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:49,884 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:49,896 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 2 map-reduce job(s) waiting for submission.
*2010-08-20 14:43:49,902 [Thread-13] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
*2010-08-20 14:43:50,398 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-20 14:43:51,082 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 5800
2010-08-20 14:44:02,525 [Thread-13] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 14:44:02,871 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 950
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7120
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
2010-08-20 14:48:41,613 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 20% complete
2010-08-20 14:52:11,487 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2010-08-20 14:56:47,404 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 38% complete
2010-08-20 14:58:45,888 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 42% complete
2010-08-20 15:09:23,195 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 15:09:24,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 15:09:24,344 [main] INFO
org.apach
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
Or
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
You can forward the error message if you need further help with this.
-Thejas
On 8/20/10 4:22 PM, "Raman Yakkala" <[EMAIL PROTECTED]> wrote:
Hi Guys,
I am trying to do join tow data sets and the jobs are failing. There are
some warnings reported and I am not good at understanding them. I am seeking
your help in adjusting any parameters to the job so that the job might
succeed.
Here is the grunt shell output:
grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
USING PigStorage();
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
odl: $1, $2, $4, $5, $6, $8
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for odl
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
$19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
$36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
$53, $55
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for oil
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for qif
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for qif
2010-08-20 14:43:47,221 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
- 1-799 Operator Key: 1-799)
*2010-08-20 14:43:47,269 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
2010-08-20 14:43:47,274 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning MULTI_LEAF_MAP 1 time(s).
*2010-08-20 14:43:47,284 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2010-08-20 14:43:47,333 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:48,645 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:48,715 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:49,884 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:49,896 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 2 map-reduce job(s) waiting for submission.
*2010-08-20 14:43:49,902 [Thread-13] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
*2010-08-20 14:43:50,398 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-20 14:43:51,082 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 5800
2010-08-20 14:44:02,525 [Thread-13] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 14:44:02,871 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 950
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7120
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
2010-08-20 14:48:41,613 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 20% complete
2010-08-20 14:52:11,487 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2010-08-20 14:56:47,404 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 38% complete
2010-08-20 14:58:45,888 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 42% complete
2010-08-20 15:09:23,195 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 15:09:24,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 15:09:24,344 [main] INFO
org.apach
