Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Running pig application doing geometric data expansion received Java.nio.channels.ClosedChannelException using hadoop-1.1.1


Copy link to this message
-
Running pig application doing geometric data expansion received Java.nio.channels.ClosedChannelException using hadoop-1.1.1
Hi All,
During running pig application doing geometric data expansion, "Java.nio.channels.ClosedChannelException" error happened. Could you please help to see this issue and give your suggestions? Thanks

Pig statement:
A = LOAD '$tmpfiledir1' AS (f1:chararray);

B = LOAD '$tmpfiledir1' AS (f1:chararray);

C = UNION A, B;

STORE C INTO '$tmpfiledir2';

The output file $tmpfiledir2 will be the input again as $tmpfiledir1. And the original size of $tmpfiledir1 is 1GB. Then the output will be:
2GB -> 4GB -> 8GB -> 16GB -> 32GB -> 64GB -> 128GB -> 256GB -> 512GB -> 1TB -> 2TB -> 4TB -> 8TB -> 16TB
Error from tasktracker:
07-14 17:50:08,261 WARN org.apache.hadoop.mapred.TaskTracker: Failed validating JVM
java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201307141733_0015_m_001616_0, with JvmId: jvm_201307141733_0015_m_-1094010055
at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3435)
at org.apache.hadoop.mapred.TaskTracker.statusUpdate(TaskTracker.java:3505)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
2013-07-14 17:50:08,262 WARN org.apache.hadoop.ipc.Server: IPC Server Responder, call statusUpdate(attempt_201307141733_0015_m_001616_0, org.apache.hadoop.mapred.MapTaskStatus@33fb33fb, org.apache.hadoop.mapred.JvmContext@40354035) from 127.0.0.1:15007: output error
2013-07-14 17:50:08,262 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54168 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:147)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:338)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1705)
at org.apache.hadoop.ipc.Server.access$2000(Server.java:95)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:741)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:805)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1421)
2013-07-14 17:50:08,265 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /hadoop/sdc/mapred/local/ttprivate/taskTracker/biadmin/jobcache/job_201307141733_0015/attempt_201307141733_0015_m_002271_0/taskjvm.sh
2013-07-14 17:50:08,266 WARN org.apache.hadoop.mapred.TaskTracker: Failed validating JVM
java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201307141733_0015_m_001641_0, with JvmId: jvm_201307141733_0015_m_22934798
at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3435)
at org.apache.hadoop.mapred.TaskTracker.statusUpdate(TaskTracker.java:3505)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
2013-07-14 17:50:08,297 WARN org.apache.hadoop.mapred.LinuxTaskController: Exit code from task is : 66
2013-07-14 17:50:08,297 WARN org.apache.hadoop.mapred.LinuxTaskController: Exception thrown while launching task JVM : org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.launchTask(LinuxTaskController.java:238)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)
at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB