Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Heavy Writes Block Reads to RegionServer


Copy link to this message
-
Heavy Writes Block Reads to RegionServer
We are running a job that does heavy writes into a new table.  The table is
not pre-split so it has 1 region.  I know this is not recommended; we were
doing it partially to test this particular case.

Here's what we're seeing:
   1. Reads are entirely blocked.  No reads to any region on that server
   make it through.
   2. Writes are insanely slow.  Some writes appear to be taking over 10
   minutes.
   3. All of the box's resources are quiet:  Around < 20% CPU usage, plenty
   of memory to spare, iostat looked normal
   4. ngrep showed only writes coming through.  no reads
   5. The logs showed lots of WARN org.apache.hadoop.ipc.HBaseServer: IPC
   Server Responder, call
   multi(org.apache.hadoop.hbase.client.MultiAction@714cd947) from
   10.211.117.161:34380: output error

Any ideas what's up?  Is there some sort of global lock that might halt
reads during heavy writes?  Anything else we can look for during this?  We
can rerun the job to reproduce this, as this is a test cluster which can
afford to be brought down.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB