I am getting a error while running a Pig Script on a 400MB compressed file.
But the script works fine with a sample input file with 1000 lines. The
error details are given below. Any thoughts ?

2011-06-01 12:12:22,152 [Thread-4] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2011-06-01 12:12:22,166 [Thread-4] INFO
 org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2011-06-01 12:12:22,168 [Thread-4] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2011-06-01 12:12:22,275 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2011-06-01 12:12:22,921 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201105271444_0064
2011-06-01 12:12:22,921 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://localhost:50030/jobdetails.jsp?jobid=job_201105271444_0064
2011-06-01 12:12:31,109 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2011-06-01 12:12:46,088 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:14:43,279 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:15:18,404 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:19:08,924 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:37:46,492 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:37:51,541 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
 2011-06-01 12:39:41,488 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201105271444_0064 has failed! Stop running all dependent jobs
2011-06-01 12:39:41,493 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/impressions/imp1.log.gz
2011-06-01 12:39:41,494 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/matchtables/ad1.log.gz
2011-06-01 12:39:41,494 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-06-01 12:39:41,523 [main] ERROR org.apache.pig.tools.pigstats.PigStats
- ERROR 2997: Unable to recreate exception from backed error:
java.io.IOException: Spill failed
2011-06-01 12:39:41,524 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-06-01 12:39:41,525 [main] INFO  org.apache.pig.tools.pigstats.PigStats
- Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u0 0.8.0-cdh3u0 root 2011-06-01 12:12:17 2011-06-01 12:39:41
HASH_JOIN,FILTER

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_201105271444_0064 advertiser_match,joined,logs,out HASH_JOIN Message:
Job failed! Error - NA hdfs://localhost/user/root/pigdbck/resultimp2,

Input(s):
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/impressions/imp.log.gz"
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/matchtables/adv .log.gz"

Output(s):
Failed to produce result in "hdfs://localhost/user/root/pigdbck/resul

Backend error message
---------------------
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1069)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1050)
at java.io.DataOutputStream.writeBoolean(DataOutputStream.java:122)
at
org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:122)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:917)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:573)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not
find any valid local directory for output/spill46.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:376)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputF

2011-06-01 13:53:31,347 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 16% complete
2011-06-01 13:53:39,179 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 13:53:49,953 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 44% complete
2011-06-01 14:04:06,008 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 44% complete
  2011-06-01 14:19:12,734 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201105271444_0067 has failed! Stop running all dependent jobs
2011-06-01 14:19:12,739 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/matchtables/ad.log.gz
2011-06-01 14:19:12,741 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/impressions/Net.log.gz
2011-06-01 14:19:12,741 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/matchtables/adv.log.gz
2011-06-01 14:19:12,741 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-06-01 14:19:12,757 [main] ERROR org.apache.pig.tools.pigstats.PigStats
- ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/file.out
2011-06-01 14:19:12,757 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-06-01 14:19:12,758 [main] INFO  org.apache.pig.tools.pigstats.PigStats
- Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u0 0.8.0-cdh3u0 root 2011-06-01 13:53:14 2011-06-01 14:19:12
HASH_JOIN,FILTER

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_201105271444_0067 ad_match,advertiser_match,joined,logs,out
HASH_JOIN Message:
Job failed! Error - NA hdfs://localhost/user/root/pigdbck/resultimpnet4,

Input(s):
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/matchtables/ad.log.gz"
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/impressions/Net.log.gz"
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/matchtables/adv.log.gz"

Output(s):
Failed to produce result in
"hdfs://localhost/user/root/pigdbck/resultimpnet4"

Counters:
Total records written : 0
cat /root/pigdbck/scripts/pig_1306936392697.log
Backend error message
---------------------
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/file.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:376)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
at
org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:69)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1494)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1179)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:581)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)

Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/file.out

org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to
recreate exception from backed error:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/file.out
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.execute(PigServer.java:1190)
at org.apache.pig.PigServer.access$100(PigServer.java:128)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
at
org.apache.pig.tools.grunt.GruntParser.processFsCommand(GruntParser.java:898)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:117)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:406)
at org.apache.pig.Main.main(Main.java:107)
On 1 June 2011 14:00, Subhramanian, Deepak <
[EMAIL PROTECTED]> wrote:
"Please consider the environment before printing this e-mail"
 
The Newspaper Marketing Agency: Opening Up Newspapers:
 
http://www.nmauk.co.uk/
 
This e-mail and any attachments are confidential, may be legally privileged and are the proper

Do you have enough disk space on each node ? It looks like MR is having problem writing/finding a disk to write.
Are you seeing this problem for all pig/MR jobs or just one of them ?

Thanks,
Thejas

On 6/1/11 6:00 AM, "Subhramanian, Deepak" <[EMAIL PROTECTED]> wrote:

I am getting a error while running a Pig Script on a 400MB compressed file.
But the script works fine with a sample input file with 1000 lines. The
error details are given below. Any thoughts ?

2011-06-01 12:12:22,152 [Thread-4] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2011-06-01 12:12:22,166 [Thread-4] INFO
 org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2011-06-01 12:12:22,168 [Thread-4] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2011-06-01 12:12:22,275 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2011-06-01 12:12:22,921 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201105271444_0064
2011-06-01 12:12:22,921 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://localhost:50030/jobdetails.jsp?jobid=job_201105271444_0064
2011-06-01 12:12:31,109 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2011-06-01 12:12:46,088 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:14:43,279 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:15:18,404 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:19:08,924 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:37:46,492 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:37:51,541 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
 2011-06-01 12:39:41,488 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201105271444_0064 has failed! Stop running all dependent jobs
2011-06-01 12:39:41,493 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/impressions/imp1.log.gz
2011-06-01 12:39:41,494 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/matchtables/ad1.log.gz
2011-06-01 12:39:41,494 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-06-01 12:39:41,523 [main] ERROR org.apache.pig.tools.pigstats.PigStats
- ERROR 2997: Unable to recreate exception from backed error:
java.io.IOException: Spill failed
2011-06-01 12:39:41,524 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-06-01 12:39:41,525 [main] INFO  org.apache.pig.tools.pigstats.PigStats
- Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u0 0.8.0-cdh3u0 root 2011-06-01 12:12:17 2011-06-01 12:39:41
HASH_JOIN,FILTER

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_201105271444_0064 advertiser_match,joined,logs,out HASH_JOIN Message:
Job failed! Error - NA hdfs://localhost/user/root/pigdbck/resultimp2,

Input(s):
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/impressions/imp.log.gz"
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/matchtables/adv .log.gz"

Output(s):
Failed to produce result in "hdfs://localhost/user/root/pigdbck/resul

Backend error message
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1069)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1050)
at java.io.DataOutputStream.writeBoolean(DataOutputStream.java:122)
at
org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:122)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:917)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:573)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not
find any valid local directory for output/spill46.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator

How big is the output of the join expected to be ? (for example, if you have large number of join keys with same value in both files, the output could be very large). Are you using replicated join ?

Thanks,
Thejas
On 6/1/11 8:33 AM, "Subhramanian, Deepak" <[EMAIL PROTECTED]> wrote:

It looks like I have enough space  in the ec2 server  I am using. I am trying to join a 300mb zipped file with another 2mb zipped file. The pig script worked  when the no of columns in the output was less.

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   7571832   2225088  78% /
devtmpfs               3944716        52   3944664   1% /dev
tmpfs                  3944716         0   3944716   0% /dev/shm
On 1 June 2011 15:53, Thejas M Nair <[EMAIL PROTECTED]> wrote:
Do you have enough disk space on each node ? It looks like MR is having problem writing/finding a disk to write.
Are you seeing this problem for all pig/MR jobs or just one of them ?

Thanks,
Thejas
On 6/1/11 6:00 AM, "Subhramanian, Deepak" <[EMAIL PROTECTED]> wrote:

I am getting a error while running a Pig Script on a 400MB compressed file.
But the script works fine with a sample input file with 1000 lines. The
error details are given below. Any thoughts ?

2011-06-01 12:12:22,152 [Thread-4] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2011-06-01 12:12:22,166 [Thread-4] INFO
 org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2011-06-01 12:12:22,168 [Thread-4] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2011-06-01 12:12:22,275 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2011-06-01 12:12:22,921 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201105271444_0064
2011-06-01 12:12:22,921 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://localhost:50030/jobdetails.jsp?jobid=job_201105271444_0064
2011-06-01 12:12:31,109 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2011-06-01 12:12:46,088 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:14:43,279 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:15:18,404 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:19:08,924 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:37:46,492 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-06-01 12:37:51,541 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
 2011-06-01 12:39:41,488 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201105271444_0064 has failed! Stop running all dependent jobs
2011-06-01 12:39:41,493 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/impressions/imp1.log.gz
2011-06-01 12:39:41,494 [main] WARN  org.apache.pig.tools.pigstats.JobStats
- unable to get input counter for
hdfs://localhost/user/root/pigdbck/data/matchtables/ad1.log.gz
2011-06-01 12:39:41,494 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-06-01 12:39:41,523 [main] ERROR org.apache.pig.tools.pigstats.PigStats
- ERROR 2997: Unable to recreate exception from backed error:
java.io.IOException: Spill failed
2011-06-01 12:39:41,524 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-06-01 12:39:41,525 [main] INFO  org.apache.pig.tools.pigstats.PigStats
- Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u0 0.8.0-cdh3u0 root 2011-06-01 12:12:17 2011-06-01 12:39:41
HASH_JOIN,FILTER

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_201105271444_0064 advertiser_match,joined,logs,out HASH_JOIN Message:
Job failed! Error - NA hdfs://localhost/user/root/pigdbck/resultimp2,

Input(s):
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/impressions/imp.log.gz"
Failed to read data from
"hdfs://localhost/user/root/pigdbck/data/matchtables/adv .log.gz"

Output(s):
Failed to produce result in "hdfs://localhost/user/root/pigdbck/resul

Backend error message
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1069)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1050)
at java.io.DataOutputStream.writeBoolean(DataOutputStream.java:122)
at
org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:122)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:917)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:573)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:23

If I use an S3 volume with a bigger memory as the datanode , can I get rid
of the diskspace issue . Or Hadoop internally uses the space in the local
server for temporary files ?

On 1 June 2011 18:55, Subhramanian, Deepak <
[EMAIL PROTECTED]> wrote:

> Hi Tejas,
>
> It looks it is a disk space issue.
> Thanks for pointing out.
>
> Does hadoop internally unzip the file when it process the file.
>
> ip-10-49-15-188:~ # df -k
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda1             10321208   7585836   2211084  78% /
> devtmpfs               3944716        52   3944664   1% /dev
> tmpfs                  3944716         0   3944716   0% /dev/shm
>
>
> ip-10-49-15-188:~ # df -k
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda1             10321208   9599652    197268  98% /
> devtmpfs               3944716        52   3944664   1% /dev
> tmpfs                  3944716         0   3944716   0% /dev/shm
>
> On 1 June 2011 17:12, Subhramanian, Deepak <
> [EMAIL PROTECTED]> wrote:
>
>>
>>
>>
>> The output will be around 300 mb as I am removing 2 columns from the input
>> file and adding two columns using a left outer join. I am running the script
>> in a pseudo installation .
>>
>> This is the join query.
>> JOIN logs BY AdvertiserID LEFT, advertiser_match BY AdvertiserID ;
>>
>> On 1 June 2011 16:46, Thejas M Nair <[EMAIL PROTECTED]> wrote:
>>
>>>  How big is the output of the join expected to be ? (for example, if you
>>> have large number of join keys with same value in both files, the output
>>> could be very large). Are you using replicated join ?
>>>
>>> Thanks,
>>> Thejas
>>>
>>>
>>>
>>>
>>>
>>> On 6/1/11 8:33 AM, "Subhramanian, Deepak" <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>> It looks like I have enough space  in the ec2 server  I am using. I am
>>> trying to join a 300mb zipped file with another 2mb zipped file. The pig
>>> script worked  when the no of columns in the output was less.
>>>
>>> Filesystem           1K-blocks      Used Available Use% Mounted on
>>> /dev/sda1             10321208   7571832   2225088  78% /
>>> devtmpfs               3944716        52   3944664   1% /dev
>>> tmpfs                  3944716         0   3944716   0% /dev/shm
>>>
>>>
>>> On 1 June 2011 15:53, Thejas M Nair <[EMAIL PROTECTED]> wrote:
>>>
>>> Do you have enough disk space on each node ? It looks like MR is having
>>> problem writing/finding a disk to write.
>>> Are you seeing this problem for all pig/MR jobs or just one of them ?
>>>
>>> Thanks,
>>> Thejas
>>>
>>>
>>>
>>>
>>> On 6/1/11 6:00 AM, "Subhramanian, Deepak" <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>> I am getting a error while running a Pig Script on a 400MB compressed
>>> file.
>>> But the script works fine with a sample input file with 1000 lines. The
>>> error details are given below. Any thoughts ?
>>>
>>> 2011-06-01 12:12:22,152 [Thread-4] INFO
>>>  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-01 12:12:22,166 [Thread-4] INFO
>>>  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
>>> paths
>>> to process : 1
>>> 2011-06-01 12:12:22,168 [Thread-4] INFO
>>>  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-01 12:12:22,275 [main] INFO
>>>
>>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>>> - 0% complete
>>> 2011-06-01 12:12:22,921 [main] INFO
>>>
>>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>>> - HadoopJobId: job_201105271444_0064
>>> 2011-06-01 12:12:22,921 [main] INFO
>>>
>>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>>> - More information at:
>>> http://localhost:50030/jobdetails.jsp?jobid=job_201105271444_0064
>>> 2011-06-01 12:12:31,109 [main] INFO
>>>
>>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
Deepak Subhramanian
Data & Analytics
News International, Digital Technology
Email: [EMAIL PROTECTED]

"Please consider the environment before printing this e-mail"
 
The Newspaper Marketing Agency: Opening Up Newspapers:
 
http://www.nmauk.co.uk/
 
This e-mail and any attachments are confidential, may be legally privileged and are the property of NI Group Limited (which is the holding company for the News International group, is registered in England under number 81701 and whose registered office is 3 Thomas More Square, London E98 1XY, VAT number GB 243 8054 69), on whose systems they were generated. If you have received this e-mail in error, please notify the sender immediately and do not use, distribute, store or copy it in any way. Statements or opinions in this e-mail or any attachment are those of the author and are not necessarily agreed or authorised by NI Group Limited or any member of its group. NI Group Limited may monitor outgoing or incoming emails as permitted by law. It accepts no liability for viruses introduced by this e-mail or attachments.

This message and its attachments may contain legally privileged or confidential information. It is intended solely for the named addressee. If you are not the addressee indicated in this message (or responsible for delivery of the message to the addressee), you may not copy or deliver this message or its attachments to anyone. Rather, you should permanently delete this message and its attachments and kindly notify the sender by reply e-mail. Any content of this message and its attachments that does not relate to the official business of News America Incorporated or its subsidiaries must be taken not to have been sent or endorsed by any of them. No representation is made that this email or its attachments are without defect.