Hi all,

I'm stuck. I can't insert any valuable peace of data into hbase.

Data is something around ~20mil rows (20G). I try to insert them into
nondistributed hbase with 4 parallel jobs.
MR job run until all memory given to hbase is exhaused and then
hbase produces hprof file. As profiler shows, all memory accumulated
in MemStore.kvset.
I don't understand, why hbase doesn't block untill flush memstore.
The same if I give hbase 6Gb or RAM.

6GB gc log http://paste.ubuntu.com/502577/

hadoop: 0.20.2+320
hbase: stumbleupon-20100830

nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).

Andrey.

Data is simple table with two column families.
info: json object (small)
rows: ~300 bigdecimal columns per row.

Weird indeed, even after the WAL was rolled 4 times (theoretically
256MB of data) I don't even see a flush request... although you're
running at INFO level instead of DEBUG. Could you switch that and send
us just the full log.

Thanks a lot!

J-D

On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I'm stuck. I can't insert any valuable peace of data into hbase.
>
> Data is something around ~20mil rows (20G). I try to insert them into
> nondistributed hbase with 4 parallel jobs.
> MR job run until all memory given to hbase is exhaused and then
> hbase produces hprof file. As profiler shows, all memory accumulated
> in MemStore.kvset.
> I don't understand, why hbase doesn't block untill flush memstore.
> The same if I give hbase 6Gb or RAM.
>
> 6GB gc log http://paste.ubuntu.com/502577/
>
> hadoop: 0.20.2+320
> hbase: stumbleupon-20100830
>
> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>
>
>
> Andrey.
>

Ok I found the bug, I think it's only in our distro.

Stay tuned!

J-D

On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
> Weird indeed, even after the WAL was rolled 4 times (theoretically
> 256MB of data) I don't even see a flush request... although you're
> running at INFO level instead of DEBUG. Could you switch that and send
> us just the full log.
>
> Thanks a lot!
>
> J-D
>
> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>> Hi all,
>>
>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>
>> Data is something around ~20mil rows (20G). I try to insert them into
>> nondistributed hbase with 4 parallel jobs.
>> MR job run until all memory given to hbase is exhaused and then
>> hbase produces hprof file. As profiler shows, all memory accumulated
>> in MemStore.kvset.
>> I don't understand, why hbase doesn't block untill flush memstore.
>> The same if I give hbase 6Gb or RAM.
>>
>> 6GB gc log http://paste.ubuntu.com/502577/
>>
>> hadoop: 0.20.2+320
>> hbase: stumbleupon-20100830
>>
>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>
>>
>>
>> Andrey.
>>
>

wow. i'll wait. thanks for reply.

2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
> Ok I found the bug, I think it's only in our distro.
>
> Stay tuned!
>
> J-D
>
> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>> 256MB of data) I don't even see a flush request... although you're
>> running at INFO level instead of DEBUG. Could you switch that and send
>> us just the full log.
>>
>> Thanks a lot!
>>
>> J-D
>>
>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>> Hi all,
>>>
>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>
>>> Data is something around ~20mil rows (20G). I try to insert them into
>>> nondistributed hbase with 4 parallel jobs.
>>> MR job run until all memory given to hbase is exhaused and then
>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>> in MemStore.kvset.
>>> I don't understand, why hbase doesn't block untill flush memstore.
>>> The same if I give hbase 6Gb or RAM.
>>>
>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>
>>> hadoop: 0.20.2+320
>>> hbase: stumbleupon-20100830
>>>
>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>
>>>
>>>
>>> Andrey.
>>>
>>
>

The fix is here: http://pastebin.com/zuL23e0U

We're going to do a push to github later today, along with other
patches that require more testing.

J-D

On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
> wow. i'll wait. thanks for reply.
>
> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>> Ok I found the bug, I think it's only in our distro.
>>
>> Stay tuned!
>>
>> J-D
>>
>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>> 256MB of data) I don't even see a flush request... although you're
>>> running at INFO level instead of DEBUG. Could you switch that and send
>>> us just the full log.
>>>
>>> Thanks a lot!
>>>
>>> J-D
>>>
>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>> Hi all,
>>>>
>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>
>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>> nondistributed hbase with 4 parallel jobs.
>>>> MR job run until all memory given to hbase is exhaused and then
>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>> in MemStore.kvset.
>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>> The same if I give hbase 6Gb or RAM.
>>>>
>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>
>>>> hadoop: 0.20.2+320
>>>> hbase: stumbleupon-20100830
>>>>
>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>
>>>>
>>>>
>>>> Andrey.
>>>>
>>>
>>
>

Against what version this patch.  I can't find "Only 1 KV, does" in
any stumbleupon or
upstream repositories
2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
> The fix is here: http://pastebin.com/zuL23e0U
>
> We're going to do a push to github later today, along with other
> patches that require more testing.
>
> J-D
>
> On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>> wow. i'll wait. thanks for reply.
>>
>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>> Ok I found the bug, I think it's only in our distro.
>>>
>>> Stay tuned!
>>>
>>> J-D
>>>
>>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>>> 256MB of data) I don't even see a flush request... although you're
>>>> running at INFO level instead of DEBUG. Could you switch that and send
>>>> us just the full log.
>>>>
>>>> Thanks a lot!
>>>>
>>>> J-D
>>>>
>>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>>
>>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>>> nondistributed hbase with 4 parallel jobs.
>>>>> MR job run until all memory given to hbase is exhaused and then
>>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>>> in MemStore.kvset.
>>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>>> The same if I give hbase 6Gb or RAM.
>>>>>
>>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>>
>>>>> hadoop: 0.20.2+320
>>>>> hbase: stumbleupon-20100830
>>>>>
>>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>>
>>>>>
>>>>>
>>>>> Andrey.
>>>>>
>>>>
>>>
>>
>

http://github.com/stumbleupon/hbase/blob/master/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1418

On Thu, Sep 30, 2010 at 1:17 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
> Against what version this patch.  I can't find "Only 1 KV, does" in
> any stumbleupon or
> upstream repositories
>
>
> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>> The fix is here: http://pastebin.com/zuL23e0U
>>
>> We're going to do a push to github later today, along with other
>> patches that require more testing.
>>
>> J-D
>>
>> On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>> wow. i'll wait. thanks for reply.
>>>
>>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>>> Ok I found the bug, I think it's only in our distro.
>>>>
>>>> Stay tuned!
>>>>
>>>> J-D
>>>>
>>>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>>>> 256MB of data) I don't even see a flush request... although you're
>>>>> running at INFO level instead of DEBUG. Could you switch that and send
>>>>> us just the full log.
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>> J-D
>>>>>
>>>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>>>
>>>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>>>> nondistributed hbase with 4 parallel jobs.
>>>>>> MR job run until all memory given to hbase is exhaused and then
>>>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>>>> in MemStore.kvset.
>>>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>>>> The same if I give hbase 6Gb or RAM.
>>>>>>
>>>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>>>
>>>>>> hadoop: 0.20.2+320
>>>>>> hbase: stumbleupon-20100830
>>>>>>
>>>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Andrey.
>>>>>>
>>>>>
>>>>
>>>
>>
>

--
Alexey Kovyrin
http://kovyrin.net/

ohh... sorry. I find it. Thanks for patch.

2010/9/30 Andrey Stepachev <[EMAIL PROTECTED]>:
> Against what version this patch.  I can't find "Only 1 KV, does" in
> any stumbleupon or
> upstream repositories
>
>
> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>> The fix is here: http://pastebin.com/zuL23e0U
>>
>> We're going to do a push to github later today, along with other
>> patches that require more testing.
>>
>> J-D
>>
>> On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>> wow. i'll wait. thanks for reply.
>>>
>>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>>> Ok I found the bug, I think it's only in our distro.
>>>>
>>>> Stay tuned!
>>>>
>>>> J-D
>>>>
>>>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>>>> 256MB of data) I don't even see a flush request... although you're
>>>>> running at INFO level instead of DEBUG. Could you switch that and send
>>>>> us just the full log.
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>> J-D
>>>>>
>>>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>>>
>>>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>>>> nondistributed hbase with 4 parallel jobs.
>>>>>> MR job run until all memory given to hbase is exhaused and then
>>>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>>>> in MemStore.kvset.
>>>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>>>> The same if I give hbase 6Gb or RAM.
>>>>>>
>>>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>>>
>>>>>> hadoop: 0.20.2+320
>>>>>> hbase: stumbleupon-20100830
>>>>>>
>>>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Andrey.
>>>>>>
>>>>>
>>>>
>>>
>>
>

Thanks. I found. I looked in upstream branch instead of stumbleupon
branch in my local repo.

2010/9/30 Alexey Kovyrin <[EMAIL PROTECTED]>:
> http://github.com/stumbleupon/hbase/blob/master/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1418
>

Thanx once more. Now works fine.

2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
> The fix is here: http://pastebin.com/zuL23e0U
>
> We're going to do a push to github later today, along with other
> patches that require more testing.
>
> J-D
>
> On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>> wow. i'll wait. thanks for reply.
>>
>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>> Ok I found the bug, I think it's only in our distro.
>>>
>>> Stay tuned!
>>>
>>> J-D
>>>
>>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>>> 256MB of data) I don't even see a flush request... although you're
>>>> running at INFO level instead of DEBUG. Could you switch that and send
>>>> us just the full log.
>>>>
>>>> Thanks a lot!
>>>>
>>>> J-D
>>>>
>>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>>
>>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>>> nondistributed hbase with 4 parallel jobs.
>>>>> MR job run until all memory given to hbase is exhaused and then
>>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>>> in MemStore.kvset.
>>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>>> The same if I give hbase 6Gb or RAM.
>>>>>
>>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>>
>>>>> hadoop: 0.20.2+320
>>>>> hbase: stumbleupon-20100830
>>>>>
>>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>>
>>>>>
>>>>>
>>>>> Andrey.
>>>>>
>>>>
>>>
>>
>

Sorry for that :)

I still didn't push to github the new version as I'm still testing a
few unrelated improvements that are coming in (nothing fancy, as it's
already in apache's SVN).

J-D

On Thu, Sep 30, 2010 at 12:01 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
> Thanx once more. Now works fine.
>
> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>> The fix is here: http://pastebin.com/zuL23e0U
>>
>> We're going to do a push to github later today, along with other
>> patches that require more testing.
>>
>> J-D
>>
>> On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>> wow. i'll wait. thanks for reply.
>>>
>>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>>> Ok I found the bug, I think it's only in our distro.
>>>>
>>>> Stay tuned!
>>>>
>>>> J-D
>>>>
>>>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>>>> 256MB of data) I don't even see a flush request... although you're
>>>>> running at INFO level instead of DEBUG. Could you switch that and send
>>>>> us just the full log.
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>> J-D
>>>>>
>>>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>>>
>>>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>>>> nondistributed hbase with 4 parallel jobs.
>>>>>> MR job run until all memory given to hbase is exhaused and then
>>>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>>>> in MemStore.kvset.
>>>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>>>> The same if I give hbase 6Gb or RAM.
>>>>>>
>>>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>>>
>>>>>> hadoop: 0.20.2+320
>>>>>> hbase: stumbleupon-20100830
>>>>>>
>>>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Andrey.
>>>>>>
>>>>>
>>>>
>>>
>>
>

Ok. I applied patch to internal git. 10 hours of mr job and still looks fine.

2010/9/30 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
> Sorry for that :)
>
> I still didn't push to github the new version as I'm still testing a
> few unrelated improvements that are coming in (nothing fancy, as it's
> already in apache's SVN).
>
> J-D
>
> On Thu, Sep 30, 2010 at 12:01 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>> Thanx once more. Now works fine.
>>
>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>> The fix is here: http://pastebin.com/zuL23e0U
>>>
>>> We're going to do a push to github later today, along with other
>>> patches that require more testing.
>>>
>>> J-D
>>>
>>> On Wed, Sep 29, 2010 at 10:54 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>> wow. i'll wait. thanks for reply.
>>>>
>>>> 2010/9/29 Jean-Daniel Cryans <[EMAIL PROTECTED]>:
>>>>> Ok I found the bug, I think it's only in our distro.
>>>>>
>>>>> Stay tuned!
>>>>>
>>>>> J-D
>>>>>
>>>>> On Wed, Sep 29, 2010 at 9:26 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>>>>> Weird indeed, even after the WAL was rolled 4 times (theoretically
>>>>>> 256MB of data) I don't even see a flush request... although you're
>>>>>> running at INFO level instead of DEBUG. Could you switch that and send
>>>>>> us just the full log.
>>>>>>
>>>>>> Thanks a lot!
>>>>>>
>>>>>> J-D
>>>>>>
>>>>>> On Wed, Sep 29, 2010 at 4:25 AM, Andrey Stepachev <[EMAIL PROTECTED]> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm stuck. I can't insert any valuable peace of data into hbase.
>>>>>>>
>>>>>>> Data is something around ~20mil rows (20G). I try to insert them into
>>>>>>> nondistributed hbase with 4 parallel jobs.
>>>>>>> MR job run until all memory given to hbase is exhaused and then
>>>>>>> hbase produces hprof file. As profiler shows, all memory accumulated
>>>>>>> in MemStore.kvset.
>>>>>>> I don't understand, why hbase doesn't block untill flush memstore.
>>>>>>> The same if I give hbase 6Gb or RAM.
>>>>>>>
>>>>>>> 6GB gc log http://paste.ubuntu.com/502577/
>>>>>>>
>>>>>>> hadoop: 0.20.2+320
>>>>>>> hbase: stumbleupon-20100830
>>>>>>>
>>>>>>> nothing in hbase-site.xml (except hbase.rootdir and zookeeper.quorum).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Andrey.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>