Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - How is DataXceiver been used?


Copy link to this message
-
RE: How is DataXceiver been used?
Liu, Raymond 2013-01-18, 01:15
Hi Tariq

Thanks, this blog is very informational!

I roughly figure out the usage of xceiver. While the weird thing is even according to this blog, I should have far enough xceiever in my full table scan job. Especially if it count the number of storefile instead of real blk_xxxx tfile on nativeFS. Then I only have around 24 storefiles for 24 region since I do major compact them. And then it exceeding 1024 limit during full table scan....

Might need more investigation into my case to check out what wrong with it.
>
> Hello Raymond,
>
>             You might find this
> link<http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/>helpful.
> It explains the problem
> and solution in great detail.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Jan 17, 2013 at 12:14 PM, Liu, Raymond
> <[EMAIL PROTECTED]>wrote:
>
> > Hi
> >
> >         I have a table with about 24 region on the one regionserver,
> > and each region have about 20 block files on hdfs.
> >         The xceiverCount is set to 1024, I have thought that this is
> > quite enough since at most 480 blocks will be opened.
> >         While when I do a MR job to scan the table, with 24 map task
> > each open and scan a different region at the same time, it turn out
> > that the DataXceiver is run out...
> >
> >         I am a little bit puzzled, those blocks will only be read by
> > one task, then shouldn't region server scan blocks one by one? And
> > since there are 480 blocks at most, how can it use up dataXceiver?
> >
> > Best Regards,
> > Raymond Liu
> >
> >