Liu, Raymond 2013-01-17, 06:44
Mohammad Tariq 2013-01-17, 09:46
Thanks, this blog is very informational!
I roughly figure out the usage of xceiver. While the weird thing is even according to this blog, I should have far enough xceiever in my full table scan job. Especially if it count the number of storefile instead of real blk_xxxx tfile on nativeFS. Then I only have around 24 storefiles for 24 region since I do major compact them. And then it exceeding 1024 limit during full table scan....
Might need more investigation into my case to check out what wrong with it.
> Hello Raymond,
> You might find this
> It explains the problem
> and solution in great detail.
> Warm Regards,
> On Thu, Jan 17, 2013 at 12:14 PM, Liu, Raymond
> <[EMAIL PROTECTED]>wrote:
> > Hi
> > I have a table with about 24 region on the one regionserver,
> > and each region have about 20 block files on hdfs.
> > The xceiverCount is set to 1024, I have thought that this is
> > quite enough since at most 480 blocks will be opened.
> > While when I do a MR job to scan the table, with 24 map task
> > each open and scan a different region at the same time, it turn out
> > that the DataXceiver is run out...
> > I am a little bit puzzled, those blocks will only be read by
> > one task, then shouldn't region server scan blocks one by one? And
> > since there are 480 blocks at most, how can it use up dataXceiver?
> > Best Regards,
> > Raymond Liu