jeff whiting 2010-06-04, 15:56
Todd Lipcon 2010-06-04, 16:01
Jeff Whiting 2010-06-04, 16:37
Todd Lipcon 2010-06-04, 19:03
Alex Kozlov 2010-06-04, 19:18
Allen Wittenauer 2010-06-04, 19:19
Jeff Whiting 2010-06-07, 17:02
Andrew Purtell 2010-06-07, 19:53
Gokulakannan M 2010-06-08, 05:31
-RE: Lots of Different Kind of Datanode Errors
Andrew Purtell 2010-06-08, 16:54
From: Gokulakannan M <[EMAIL PROTECTED]>
Subject: RE: Lots of Different Kind of Datanode Errors
To: [EMAIL PROTECTED], [EMAIL PROTECTED]
Date: Monday, June 7, 2010, 10:31 PM
What is the reference of that fix?
From: Andrew Purtell
Sent: Tuesday, June 08, 2010 1:24
To: [EMAIL PROTECTED]
Subject: Re: Lots of Different
Kind of Datanode Errors
Current synchronization on FSDataset seems not
quite right. Doing what amounted to applying Todd's patch that modifies
FSDataSet to use reentrant rwlocks cleared up that type of problem for
From: Jeff Whiting <[EMAIL PROTECTED]>
Subject: Re: Lots of Different Kind of Datanode Errors
To: [EMAIL PROTECTED]
Date: Monday, June 7, 2010, 10:02 AM
Thanks for the replies. I have turned off swap
on all the machines to prevent any swap problems. I was pounding my
hard drives quite hard. I had a simulated 60 clients loading data as
fast as I could into hbase with a map reduce export job going at the same
time. Would that scenario explain some of the errors I was seeing?
Over the weekend under more of a normal load I haven't not any exception
except for about 6 of these:
2010-06-05 03:46:41,229 ERROR datanode.DataNode (DataXceiver.java:run(131)) -
blk_-1677111232590888964_4471547 is valid, and cannot be written to.
The reason the config shows 4096 is because I increased the xceiver account
after the first email message in this thread.
Allen Wittenauer wrote:
On Jun 4, 2010, at 12:03 PM, Todd Lipcon wrote:
Hi Jeff, That seems like a reasonable config, but the error message you pasted indicated xceivers was set to 2048 instead of 4096. Also, in my experience SocketTimeoutExceptions are usually due to swapping. Verify that your machines aren't swapping when you're under load.
Or doing any other heavy disk IO.
-- Jeff WhitingQualtrics Senior Software [EMAIL PROTECTED]