Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> CopyTable utility fails on larger tables


Copy link to this message
-
CopyTable utility fails on larger tables
Hello,

I can copy relatively small tables (10gb, 7million rows) using the built-in
HBase (0.92.1-cdh4.0.1) CopyTable utility but copying larger tables, say
150gb, 100million rows does not work.

The failed CopyTable job required 128 mappers according to the Job Tracker
UI, all of these failed in the first attempt after 15 minutes, the job then
ran another 1 hour while remaining at 0%. However, according  to the
counters many rows apparently had been mapped and emitted. Checking with
HBase shell, I could not perform any action on the destination table (scan,
get, count) and the HBase Master Web UI showed only one region for the
destination table. I checked the log file on this region server and saw
attached log record (extract).

What precautions should I take when copying tables? Do certain settings
need to be de-activated for the duration of the job?

Thank you,

/David
2012-12-05 15:50:40,406 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
_xxxxxxx_EH_xxx,{\xF0\xE4\xA2?!EQ\xB8\xC9tE\x19\x92
\x08,1354713876229.a75fba31d9883ed7be4ed4a7be0e592f. due to global heap
pressure
2012-12-05 15:50:49,086 INFO org.apache.hadoop.hbase.regionserver.Store:
Added hdfs://
xxxxx-1.xxxxxx.net:8020/hbase/_xxxxxxx_EH_xxx/a75fba31d9883ed7be4ed4a7be0e592f/t/1788b9f6f9594e2e9efe4ea5230d134c,
entries=418152, sequenceid=1440152048, memsize=217.0m, filesize=145.9m
2012-12-05 15:50:49,088 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~217.8m/228416264, currentsize=33.0m/34555320
for region _xxxxxxx_EH_xxx,{\xF0\xE4\xA2?!EQ\xB8\xC9tE\x19\x92
\x08,1354713876229.a75fba31d9883ed7be4ed4a7be0e592f. in 8682ms,
sequenceid=1440152048, compaction requested=true
2012-12-05 15:50:49,088 WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region
_xxxxxxx_EH_xxx,,1354713876229.37825c623850b16013ab0bf902d02746. has too
many store files; delaying flush up to 90000ms
2012-12-05 15:50:49,760 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@44848967),
rpc version=1, client version=29, methodsFingerPrint=54742778 from
5.39.67.13:56290: output error
2012-12-05 15:50:49,760 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1663)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:934)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1013)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:419)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1356)

2012-12-05 15:50:49,763 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: readAndProcess threw exception java.io.IOException:
Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1686)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1130)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:713)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:505)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:480)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2012-12-05 15:50:49,792 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: readAndProcess threw exception java.io.IOException:
Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1686)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1130)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:713)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:505)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:480)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2012-12-05 15:50:49,851 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@304fa425),
rpc version=1, client version=29, methodsFingerPrint=54742778 from
5.39.67.13:56289: output error
2012-12-05 15:50:49,851 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1663)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:934)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1013)
at
org.ap
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB