Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Occasional regionserver crashes following socket errors writing to HDFS


Copy link to this message
-
Re: Occasional regionserver crashes following socket errors writing to HDFS
Eran,

see my response inline...

On May 10, 2012, at 2:17 PM, Eran Kutner wrote:

> Michale I appreciate the feedback but I'd have to disagree.
> In my case for example, I need to look at a complete set of data produced
> by the map phase in order to make a decision and write it to Hbase. So sure
> I could write all the mappers output to hbase then have another map only
> job to scan the output of the previous one do the calculation then write
> the output to another table. I don't really see why would that be better
> than using a reducer.

You disagree without actually benchmarking the two?
That's pretty bold. :-)

2 things.
First Reducers are expensive.
Second, writing sorted records in to HBase is also more expensive than if you're writing records in random order.

Here's a caveat. I don't know what you're attempting to do, so I can only say in general, I've found it faster to write 2 mappers and avoid using reducers.

> As for the other tips, I agree the files are too large, so I increased the
> file size, but I don't really see why is that relevant to the error we're
> talking about. Why having many regions cause timeouts on HDFS?
> I do have mslabs configured and GC tuneups.
> I do run multiple reducers, I suspect that's aggravating the problem not
> helping it.
> As far as I can tell dfs.balance.bandwidthPerSec is relevant only for
> balancing done with the balancer, not for the initial replication.
>
>
With respect to the number of regions... you'd probably get a better answer St.Ack or JD.

With respect to the bandwidth issue...
We set it higher to something like 10% of the available pipe. Not that its going to be used all the time, but the smaller the pipe, the longer it takes to copy a file from one node to another.
How much of an impact it has on your performance... Not sure. But its always something to check and think about.  

BTW, I did a quick read on your problem. You didn't say which release/version of HBase you were running....
 

> -eran
>
>
>
> On Thu, May 10, 2012 at 9:59 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>> Sigh.
>>
>> Dave,
>> I really think you need to think more about the problem.
>>
>> Think about what a reduce does and then think about what happens in side
>> of HBase.
>>
>> Then think about which runs faster... a job with two mappers writing the
>> intermediate and final results in HBase,
>> or a M/R job that writes its output to HBase.
>>
>> If you really truly think about the problem, you will start to understand
>> why I say you really don't want to use a reducer when you're working w
>> HBase.
>>
>>
>> On May 10, 2012, at 1:41 PM, Dave Revell wrote:
>>
>>> Some examples of when you'd want a reducer:
>>> http://static.usenix.org/event/osdi04/tech/full_papers/dean/dean.pdf
>>>
>>> On Thu, May 10, 2012 at 11:30 AM, Michael Segel
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>> Dave, do you really want to go there?
>>>>
>>>> OP has a couple of issues and he was going down a rabbit hole.
>>>> (You can choose if that's a reference to 'the Matrix, Jefferson
>> Starship,
>>>> Alice in Wonderland... or all of the above)
>>>>
>>>> So to put him on the correct path, I recommended the following, not in
>> any
>>>> order...
>>>>
>>>> 1) Increase his region size for this table only.
>>>> 2) Look to decreasing the number of regions managed by a RS (which is
>> why
>>>> you increase region size)
>>>> 3) Up the dfs.balance.bandwidthPerSec. (How often does HBase move
>> regions
>>>> and how exactly do they move regions ?)
>>>> 4) Look at implementing MSLABS and GC tuning. This cuts down on the
>>>> overhead.
>>>> 5) Refactoring his job....
>>>>
>>>> Oops.
>>>> Ok I didn't put that in the list.
>>>> But that was the last thing I wrote as a separate statement.
>>>> Clearly you didn't take my advice and think about the problem....
>>>>
>>>> To prove a point.... you wrote:
>>>> 'Many mapreduce algorithms require a reduce phase (e.g. sorting)'
>>>>
>>>> Ok. So tell me why you would want to sort your input in to HBase and if
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB