Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Heavy Writes Block Reads to RegionServer

Copy link to this message
Heavy Writes Block Reads to RegionServer
We are running a job that does heavy writes into a new table.  The table is
not pre-split so it has 1 region.  I know this is not recommended; we were
doing it partially to test this particular case.

Here's what we're seeing:
   1. Reads are entirely blocked.  No reads to any region on that server
   make it through.
   2. Writes are insanely slow.  Some writes appear to be taking over 10
   3. All of the box's resources are quiet:  Around < 20% CPU usage, plenty
   of memory to spare, iostat looked normal
   4. ngrep showed only writes coming through.  no reads
   5. The logs showed lots of WARN org.apache.hadoop.ipc.HBaseServer: IPC
   Server Responder, call
   multi(org.apache.hadoop.hbase.client.MultiAction@714cd947) from output error

Any ideas what's up?  Is there some sort of global lock that might halt
reads during heavy writes?  Anything else we can look for during this?  We
can rerun the job to reproduce this, as this is a test cluster which can
afford to be brought down.