Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Running pig application doing geometric data expansion received Java.nio.channels.ClosedChannelException using hadoop-1.1.1


Copy link to this message
-
Running pig application doing geometric data expansion received Java.nio.channels.ClosedChannelException using hadoop-1.1.1
Hi All,
During running pig application doing geometric data expansion, "Java.nio.channels.ClosedChannelException" error happened. Could you please help to see this issue and give your suggestions? Thanks

Pig statement:
A = LOAD '$tmpfiledir1' AS (f1:chararray);

B = LOAD '$tmpfiledir1' AS (f1:chararray);

C = UNION A, B;

STORE C INTO '$tmpfiledir2';

The output file $tmpfiledir2 will be the input again as $tmpfiledir1. And the original size of $tmpfiledir1 is 1GB. Then the output will be:
2GB -> 4GB -> 8GB -> 16GB -> 32GB -> 64GB -> 128GB -> 256GB -> 512GB -> 1TB -> 2TB -> 4TB -> 8TB -> 16TB
Error from tasktracker:
07-14 17:50:08,261 WARN org.apache.hadoop.mapred.TaskTracker: Failed validating JVM
java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201307141733_0015_m_001616_0, with JvmId: jvm_201307141733_0015_m_-1094010055
at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3435)
at org.apache.hadoop.mapred.TaskTracker.statusUpdate(TaskTracker.java:3505)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
2013-07-14 17:50:08,262 WARN org.apache.hadoop.ipc.Server: IPC Server Responder, call statusUpdate(attempt_201307141733_0015_m_001616_0, org.apache.hadoop.mapred.MapTaskStatus@33fb33fb, org.apache.hadoop.mapred.JvmContext@40354035) from 127.0.0.1:15007: output error
2013-07-14 17:50:08,262 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54168 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:147)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:338)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1705)
at org.apache.hadoop.ipc.Server.access$2000(Server.java:95)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:741)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:805)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1421)
2013-07-14 17:50:08,265 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /hadoop/sdc/mapred/local/ttprivate/taskTracker/biadmin/jobcache/job_201307141733_0015/attempt_201307141733_0015_m_002271_0/taskjvm.sh
2013-07-14 17:50:08,266 WARN org.apache.hadoop.mapred.TaskTracker: Failed validating JVM
java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201307141733_0015_m_001641_0, with JvmId: jvm_201307141733_0015_m_22934798
at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3435)
at org.apache.hadoop.mapred.TaskTracker.statusUpdate(TaskTracker.java:3505)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
2013-07-14 17:50:08,297 WARN org.apache.hadoop.mapred.LinuxTaskController: Exit code from task is : 66
2013-07-14 17:50:08,297 WARN org.apache.hadoop.mapred.LinuxTaskController: Exception thrown while launching task JVM : org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.launchTask(LinuxTaskController.java:238)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)