-Heavy Writes Block Reads to RegionServer
Bryan Beaudreault 2012-05-15, 21:38
We are running a job that does heavy writes into a new table. The table is
not pre-split so it has 1 region. I know this is not recommended; we were
doing it partially to test this particular case.
Here's what we're seeing:
1. Reads are entirely blocked. No reads to any region on that server
make it through.
2. Writes are insanely slow. Some writes appear to be taking over 10
3. All of the box's resources are quiet: Around < 20% CPU usage, plenty
of memory to spare, iostat looked normal
4. ngrep showed only writes coming through. no reads
5. The logs showed lots of WARN org.apache.hadoop.ipc.HBaseServer: IPC
Server Responder, call
10.211.117.161:34380: output error
Any ideas what's up? Is there some sort of global lock that might halt
reads during heavy writes? Anything else we can look for during this? We
can rerun the job to reproduce this, as this is a test cluster which can
afford to be brought down.