Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


Copy link to this message
-
Re: Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
Cheng Su 2012-11-01, 09:16
Sorry, my mistake. Ignore about the "max store size of a single CF" please.

m(_ _)m

On Thu, Nov 1, 2012 at 4:43 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:
> Thanks Cheng. I'll try increasing my max region size limit.
>
> However I am not clear with this math:
>
> "Since you set the max file size to 2G, you can only store 2XN G data
> into a single CF."
>
> Why is that? My assumption is, even though single region can only be 2 GB,
> I can still have hundreds of regions, and hence can store 200GB+ data in
> single CF on my 10 machine cluster.
>
> Ameya
>
>
> On Thu, Nov 1, 2012 at 1:19 AM, Cheng Su <[EMAIL PROTECTED]> wrote:
>
>> I met same problem these days.
>> I'm not very sure the error log is exactly same, but I do have the
>> same exception
>>
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action: NotServingRegionException: 1 time, servers with
>> issues: smartdeals-hbase8-snc1.snc1:60020,
>>
>> and the table is also neither enabled nor disabled, thus I can't drop it.
>>
>> I guess the problem is the total store size.
>> How many region server do you have?
>> Since you set the max file size to 2G, you can only store 2XN G data
>> into a single CF.
>> (N is the number of your region servers)
>>
>> You might want to increase the max file size or region servers.
>>
>> On Thu, Nov 1, 2012 at 3:29 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:
>> > One more thing, the Hbase table in question is neither enabled, nor
>> > disabled:
>> >
>> > hbase(main):006:0> is_disabled 'userTable1'
>> > false
>> >
>> > 0 row(s) in 0.0040 seconds
>> >
>> > hbase(main):007:0> is_enabled 'userTable1'
>> > false
>> >
>> > 0 row(s) in 0.0040 seconds
>> >
>> > Ameya
>> >
>> > On Thu, Nov 1, 2012 at 12:02 AM, Ameya Kantikar <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I am trying to load lot of data (around 1.5 TB) into a single Hbase
>> table.
>> >> I have setup region size at 2 GB. I also
>> >> set hbase.regionserver.handler.count at 30.
>> >>
>> >> When I start loading data via MR, after a while, tasks start failing
>> with
>> >> following error:
>> >>
>> >> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action: NotServingRegionException: 1 time, servers with issues:
>> smartdeals-hbase8-snc1.snc1:60020,
>> >>       at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641)
>> >>       at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
>> >>       at
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
>> >>       at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
>> >>       at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
>> >>       at
>> com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:83)
>> >>       at
>> com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:33)
>> >>       at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
>> >>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
>> >>       at org.apache.hadoop.mapred.MapTask.run(MapTask.j
>> >>
>> >> On the hbase8 machine I see following in logs:
>> >>
>> >> ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Error while
>> syncing, requesting close of hlog
>> >> java.io.IOException: Reflection
>> >>         at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:230)
>> >>         at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1109)
>> >>         at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1213)
>> >>         at
>> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1071)
>> >>         at java.lang.Thread.run(Thread.java:662)
>> >> Caused by: java.lang.reflect.InvocationTargetException
>> >>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
Regards,
Cheng Su