Matt Parker
2012-08-02, 17:16
Marc Parisi
2012-08-02, 17:25
Matt Parker
2012-08-02, 17:34
John Vines
2012-08-02, 17:44
Matt Parker
2012-08-04, 20:37
William Slacum
2012-08-02, 17:49
Matt Parker
2012-08-03, 15:04
Marc Parisi
2012-08-02, 17:53
Matt Parker
2012-08-02, 17:54
David Medinets
2012-08-03, 00:46
Josh Elser
2012-08-03, 02:02
Adam Fuchs
2012-08-03, 11:45
Matt Parker
2012-08-02, 17:51
-
Accumulo 1.4 Memory IssuesMatt Parker 2012-08-02, 17:16
I setup a single instance Accumulo server.
I can load 32K rows of image metadata without issue.
I have another set of routines that build a dynamic spatial index, where
nodes are inserted/updated/deleted over time.
These operations are typically done one at a time, where each batchwriter
are closed after use.
It loads maybe a couple hundred operations, and then it dies with an
OutOfMemory error when trying to close a batchwriter.
I tried uping the memery settings on my client and on the tserver, but the
results were the same.
Outside of Accumulo, I can build the whole index in memory without any
special JVM memory settings. I was wondering whether anyone else had run
into a similar issue?
I can load 32K rows of image metadata without issue.
I have another set of routines that build a dynamic spatial index, where
nodes are inserted/updated/deleted over time.
These operations are typically done one at a time, where each batchwriter
are closed after use.
It loads maybe a couple hundred operations, and then it dies with an
OutOfMemory error when trying to close a batchwriter.
I tried uping the memery settings on my client and on the tserver, but the
results were the same.
Outside of Accumulo, I can build the whole index in memory without any
special JVM memory settings. I was wondering whether anyone else had run
into a similar issue?
+
Matt Parker 2012-08-02, 17:16
-
Re: Accumulo 1.4 Memory IssuesMarc Parisi 2012-08-02, 17:25
are you using native maps? if so, are they being used?
On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
> I setup a single instance Accumulo server.
>
> I can load 32K rows of image metadata without issue.
>
> I have another set of routines that build a dynamic spatial index, where
> nodes are inserted/updated/deleted over time.
> These operations are typically done one at a time, where each batchwriter
> are closed after use.
>
> It loads maybe a couple hundred operations, and then it dies with an
> OutOfMemory error when trying to close a batchwriter.
>
> I tried uping the memery settings on my client and on the tserver, but the
> results were the same.
>
> Outside of Accumulo, I can build the whole index in memory without any
> special JVM memory settings. I was wondering whether anyone else had run
> into a similar issue?
>
On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
> I setup a single instance Accumulo server.
>
> I can load 32K rows of image metadata without issue.
>
> I have another set of routines that build a dynamic spatial index, where
> nodes are inserted/updated/deleted over time.
> These operations are typically done one at a time, where each batchwriter
> are closed after use.
>
> It loads maybe a couple hundred operations, and then it dies with an
> OutOfMemory error when trying to close a batchwriter.
>
> I tried uping the memery settings on my client and on the tserver, but the
> results were the same.
>
> Outside of Accumulo, I can build the whole index in memory without any
> special JVM memory settings. I was wondering whether anyone else had run
> into a similar issue?
>
+
Marc Parisi 2012-08-02, 17:25
-
Re: Accumulo 1.4 Memory IssuesMatt Parker 2012-08-02, 17:34
for my small test case, I'm storing some basic data in three tables:
nodes - spatial index (id, list of child nodes, whether it's a leaf node )
image metadata - (id, bounding box coordinates, a text string of the
bounding box)
link - linking table that tells which images correspond to specific nodes.
The image data isn't being stored in Accumulo, yet.
On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
> are you using native maps? if so, are they being used?
>
>
> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
>
>> I setup a single instance Accumulo server.
>>
>> I can load 32K rows of image metadata without issue.
>>
>> I have another set of routines that build a dynamic spatial index, where
>> nodes are inserted/updated/deleted over time.
>> These operations are typically done one at a time, where each batchwriter
>> are closed after use.
>>
>> It loads maybe a couple hundred operations, and then it dies with an
>> OutOfMemory error when trying to close a batchwriter.
>>
>> I tried uping the memery settings on my client and on the tserver, but
>> the results were the same.
>>
>> Outside of Accumulo, I can build the whole index in memory without any
>> special JVM memory settings. I was wondering whether anyone else had run
>> into a similar issue?
>>
>
>
nodes - spatial index (id, list of child nodes, whether it's a leaf node )
image metadata - (id, bounding box coordinates, a text string of the
bounding box)
link - linking table that tells which images correspond to specific nodes.
The image data isn't being stored in Accumulo, yet.
On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
> are you using native maps? if so, are they being used?
>
>
> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
>
>> I setup a single instance Accumulo server.
>>
>> I can load 32K rows of image metadata without issue.
>>
>> I have another set of routines that build a dynamic spatial index, where
>> nodes are inserted/updated/deleted over time.
>> These operations are typically done one at a time, where each batchwriter
>> are closed after use.
>>
>> It loads maybe a couple hundred operations, and then it dies with an
>> OutOfMemory error when trying to close a batchwriter.
>>
>> I tried uping the memery settings on my client and on the tserver, but
>> the results were the same.
>>
>> Outside of Accumulo, I can build the whole index in memory without any
>> special JVM memory settings. I was wondering whether anyone else had run
>> into a similar issue?
>>
>
>
+
Matt Parker 2012-08-02, 17:34
-
Re: Accumulo 1.4 Memory IssuesJohn Vines 2012-08-02, 17:44
What dies, the tserver or the client?
If it's the client, then you either need to decrease the size of your
batchwriter when you create it or you need to up the heap space for your
client code.
If it's the tserver, make sure you A. picked a configuration setup who's
footprint is large enough for your system + other things running, including
zookeeper, hadoop, and your client code.
John
On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
> for my small test case, I'm storing some basic data in three tables:
>
> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
> image metadata - (id, bounding box coordinates, a text string of the
> bounding box)
> link - linking table that tells which images correspond to specific nodes.
>
> The image data isn't being stored in Accumulo, yet.
>
>
>
> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> are you using native maps? if so, are they being used?
>>
>>
>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> I setup a single instance Accumulo server.
>>>
>>> I can load 32K rows of image metadata without issue.
>>>
>>> I have another set of routines that build a dynamic spatial index, where
>>> nodes are inserted/updated/deleted over time.
>>> These operations are typically done one at a time, where each
>>> batchwriter are closed after use.
>>>
>>> It loads maybe a couple hundred operations, and then it dies with an
>>> OutOfMemory error when trying to close a batchwriter.
>>>
>>> I tried uping the memery settings on my client and on the tserver, but
>>> the results were the same.
>>>
>>> Outside of Accumulo, I can build the whole index in memory without any
>>> special JVM memory settings. I was wondering whether anyone else had run
>>> into a similar issue?
>>>
>>
>>
>
If it's the client, then you either need to decrease the size of your
batchwriter when you create it or you need to up the heap space for your
client code.
If it's the tserver, make sure you A. picked a configuration setup who's
footprint is large enough for your system + other things running, including
zookeeper, hadoop, and your client code.
John
On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
> for my small test case, I'm storing some basic data in three tables:
>
> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
> image metadata - (id, bounding box coordinates, a text string of the
> bounding box)
> link - linking table that tells which images correspond to specific nodes.
>
> The image data isn't being stored in Accumulo, yet.
>
>
>
> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> are you using native maps? if so, are they being used?
>>
>>
>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> I setup a single instance Accumulo server.
>>>
>>> I can load 32K rows of image metadata without issue.
>>>
>>> I have another set of routines that build a dynamic spatial index, where
>>> nodes are inserted/updated/deleted over time.
>>> These operations are typically done one at a time, where each
>>> batchwriter are closed after use.
>>>
>>> It loads maybe a couple hundred operations, and then it dies with an
>>> OutOfMemory error when trying to close a batchwriter.
>>>
>>> I tried uping the memery settings on my client and on the tserver, but
>>> the results were the same.
>>>
>>> Outside of Accumulo, I can build the whole index in memory without any
>>> special JVM memory settings. I was wondering whether anyone else had run
>>> into a similar issue?
>>>
>>
>>
>
+
John Vines 2012-08-02, 17:44
-
Re: Accumulo 1.4 Memory IssuesMatt Parker 2012-08-04, 20:37
I did write a set of tests to randomly insert, delete, update (delete and
insert again) records, which had no issues into 100K operations and using
large, random column family keys.
I finally tracked down the error to a BatchScanner that wasn't being
closed, which consumed every ounce client memory when hundreds of these
were waiting around for something to do. So everything is well with the
world again. Thanks for all the suggestions.
On Thu, Aug 2, 2012 at 1:44 PM, John Vines <[EMAIL PROTECTED]> wrote:
> What dies, the tserver or the client?
>
> If it's the client, then you either need to decrease the size of your
> batchwriter when you create it or you need to up the heap space for your
> client code.
>
> If it's the tserver, make sure you A. picked a configuration setup who's
> footprint is large enough for your system + other things running, including
> zookeeper, hadoop, and your client code.
>
> John
>
>
> On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
insert again) records, which had no issues into 100K operations and using
large, random column family keys.
I finally tracked down the error to a BatchScanner that wasn't being
closed, which consumed every ounce client memory when hundreds of these
were waiting around for something to do. So everything is well with the
world again. Thanks for all the suggestions.
On Thu, Aug 2, 2012 at 1:44 PM, John Vines <[EMAIL PROTECTED]> wrote:
> What dies, the tserver or the client?
>
> If it's the client, then you either need to decrease the size of your
> batchwriter when you create it or you need to up the heap space for your
> client code.
>
> If it's the tserver, make sure you A. picked a configuration setup who's
> footprint is large enough for your system + other things running, including
> zookeeper, hadoop, and your client code.
>
> John
>
>
> On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
+
Matt Parker 2012-08-04, 20:37
-
Re: Accumulo 1.4 Memory IssuesWilliam Slacum 2012-08-02, 17:49
Is it the TServer bombing out or your client or both? How often are you
flushing your writer?
On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]> wrote:
> for my small test case, I'm storing some basic data in three tables:
>
> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
> image metadata - (id, bounding box coordinates, a text string of the
> bounding box)
> link - linking table that tells which images correspond to specific nodes.
>
> The image data isn't being stored in Accumulo, yet.
>
>
>
> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> are you using native maps? if so, are they being used?
>>
>>
>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> I setup a single instance Accumulo server.
>>>
>>> I can load 32K rows of image metadata without issue.
>>>
>>> I have another set of routines that build a dynamic spatial index, where
>>> nodes are inserted/updated/deleted over time.
>>> These operations are typically done one at a time, where each
>>> batchwriter are closed after use.
>>>
>>> It loads maybe a couple hundred operations, and then it dies with an
>>> OutOfMemory error when trying to close a batchwriter.
>>>
>>> I tried uping the memery settings on my client and on the tserver, but
>>> the results were the same.
>>>
>>> Outside of Accumulo, I can build the whole index in memory without any
>>> special JVM memory settings. I was wondering whether anyone else had run
>>> into a similar issue?
>>>
>>
>>
>
flushing your writer?
On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]> wrote:
> for my small test case, I'm storing some basic data in three tables:
>
> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
> image metadata - (id, bounding box coordinates, a text string of the
> bounding box)
> link - linking table that tells which images correspond to specific nodes.
>
> The image data isn't being stored in Accumulo, yet.
>
>
>
> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> are you using native maps? if so, are they being used?
>>
>>
>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> I setup a single instance Accumulo server.
>>>
>>> I can load 32K rows of image metadata without issue.
>>>
>>> I have another set of routines that build a dynamic spatial index, where
>>> nodes are inserted/updated/deleted over time.
>>> These operations are typically done one at a time, where each
>>> batchwriter are closed after use.
>>>
>>> It loads maybe a couple hundred operations, and then it dies with an
>>> OutOfMemory error when trying to close a batchwriter.
>>>
>>> I tried uping the memery settings on my client and on the tserver, but
>>> the results were the same.
>>>
>>> Outside of Accumulo, I can build the whole index in memory without any
>>> special JVM memory settings. I was wondering whether anyone else had run
>>> into a similar issue?
>>>
>>
>>
>
+
William Slacum 2012-08-02, 17:49
-
Re: Accumulo 1.4 Memory IssuesMatt Parker 2012-08-03, 15:04
I'm getting low memory warnings on the tserver, and the client dies with an
OutOfMemory error when trying to close a writer.
I flush and close the writer basically after every change.
On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
[EMAIL PROTECTED]> wrote:
> Is it the TServer bombing out or your client or both? How often are you
> flushing your writer?
>
>
> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
OutOfMemory error when trying to close a writer.
I flush and close the writer basically after every change.
On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
[EMAIL PROTECTED]> wrote:
> Is it the TServer bombing out or your client or both? How often are you
> flushing your writer?
>
>
> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
+
Matt Parker 2012-08-03, 15:04
-
Re: Accumulo 1.4 Memory IssuesMarc Parisi 2012-08-02, 17:53
Check your log ( /logs/tserver_<<hostname>>.log ) and see if you can find
reference to memory failures, either in the native map interface or the JVM
( perhaps there is a misconfiguration on the tserver causing the JVM to
fail )?
On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
[EMAIL PROTECTED]> wrote:
> Is it the TServer bombing out or your client or both? How often are you
> flushing your writer?
>
>
> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
reference to memory failures, either in the native map interface or the JVM
( perhaps there is a misconfiguration on the tserver causing the JVM to
fail )?
On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
[EMAIL PROTECTED]> wrote:
> Is it the TServer bombing out or your client or both? How often are you
> flushing your writer?
>
>
> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
+
Marc Parisi 2012-08-02, 17:53
-
Re: Accumulo 1.4 Memory IssuesMatt Parker 2012-08-02, 17:54
I'm flushing/closing the writer after every small "transaction" (i.e. a
nodes get updated or inserted, or a link record is moved
(deleted/reinserted).
The client will throw the Out of Memory error, but I can restart the client
and rerun the same set of operatoins again. So I would assume the TServer
seems to be uneffected.
On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
[EMAIL PROTECTED]> wrote:
> Is it the TServer bombing out or your client or both? How often are you
> flushing your writer?
>
>
> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
nodes get updated or inserted, or a link record is moved
(deleted/reinserted).
The client will throw the Out of Memory error, but I can restart the client
and rerun the same set of operatoins again. So I would assume the TServer
seems to be uneffected.
On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
[EMAIL PROTECTED]> wrote:
> Is it the TServer bombing out or your client or both? How often are you
> flushing your writer?
>
>
> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>
>> for my small test case, I'm storing some basic data in three tables:
>>
>> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
>> image metadata - (id, bounding box coordinates, a text string of the
>> bounding box)
>> link - linking table that tells which images correspond to specific nodes.
>>
>> The image data isn't being stored in Accumulo, yet.
>>
>>
>>
>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> are you using native maps? if so, are they being used?
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>
>>>> I setup a single instance Accumulo server.
>>>>
>>>> I can load 32K rows of image metadata without issue.
>>>>
>>>> I have another set of routines that build a dynamic spatial index,
>>>> where nodes are inserted/updated/deleted over time.
>>>> These operations are typically done one at a time, where each
>>>> batchwriter are closed after use.
>>>>
>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>> OutOfMemory error when trying to close a batchwriter.
>>>>
>>>> I tried uping the memery settings on my client and on the tserver, but
>>>> the results were the same.
>>>>
>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>> into a similar issue?
>>>>
>>>
>>>
>>
>
+
Matt Parker 2012-08-02, 17:54
-
Re: Accumulo 1.4 Memory IssuesDavid Medinets 2012-08-03, 00:46
Maybe you can download the trial version of JProbe to see what objects
are being created and retained in memory?
are being created and retained in memory?
+
David Medinets 2012-08-03, 00:46
-
Re: Accumulo 1.4 Memory IssuesJosh Elser 2012-08-03, 02:02
FYI, you could also use jvisualvm/jmap/jhat which will already be on
your system if you've installed Oracle/Sun JDK
On 08/02/2012 07:46 PM, David Medinets wrote:
> Maybe you can download the trial version of JProbe to see what objects
> are being created and retained in memory?
your system if you've installed Oracle/Sun JDK
On 08/02/2012 07:46 PM, David Medinets wrote:
> Maybe you can download the trial version of JProbe to see what objects
> are being created and retained in memory?
+
Josh Elser 2012-08-03, 02:02
-
Re: Accumulo 1.4 Memory IssuesAdam Fuchs 2012-08-03, 11:45
Matt,
It's possible that calling flush on the BatchWriter after every add of a
small mutation could cause memory usage to spike and the java garbage
collector to fall behind. I'm not sure this is in our standard test cases.
There are two things we should do to move forwards:
1. We should try to get your application working given these constraints.
Is there a reason you're flushing the writer after every mutation? Can you
limit the calls to flush to groups of mutations? Usually flushing too
frequently it's an indicator that you're trying to do a read-modify-write
loop, and there is often a better performing alternative. If you do need to
flush frequently (e.g. maybe you're using cells as locks) then we'll
probably have to skip to #2.
2. We should try to build a test that replicates this problem in a
simplified environment. Can you share done source code that we can use to
build that test?
Adam
On Aug 2, 2012 1:54 PM, "Matt Parker" <[EMAIL PROTECTED]> wrote:
> I'm flushing/closing the writer after every small "transaction" (i.e. a
> nodes get updated or inserted, or a link record is moved
> (deleted/reinserted).
>
> The client will throw the Out of Memory error, but I can restart the
> client and rerun the same set of operatoins again. So I would assume the
> TServer seems to be uneffected.
>
> On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
> [EMAIL PROTECTED]> wrote:
>
>> Is it the TServer bombing out or your client or both? How often are you
>> flushing your writer?
>>
>>
>> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> for my small test case, I'm storing some basic data in three tables:
>>>
>>> nodes - spatial index (id, list of child nodes, whether it's a leaf node
>>> )
>>> image metadata - (id, bounding box coordinates, a text string of the
>>> bounding box)
>>> link - linking table that tells which images correspond to specific
>>> nodes.
>>>
>>> The image data isn't being stored in Accumulo, yet.
>>>
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>>
>>>> are you using native maps? if so, are they being used?
>>>>
>>>>
>>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> I setup a single instance Accumulo server.
>>>>>
>>>>> I can load 32K rows of image metadata without issue.
>>>>>
>>>>> I have another set of routines that build a dynamic spatial index,
>>>>> where nodes are inserted/updated/deleted over time.
>>>>> These operations are typically done one at a time, where each
>>>>> batchwriter are closed after use.
>>>>>
>>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>>> OutOfMemory error when trying to close a batchwriter.
>>>>>
>>>>> I tried uping the memery settings on my client and on the tserver, but
>>>>> the results were the same.
>>>>>
>>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>>> into a similar issue?
>>>>>
>>>>
>>>>
>>>
>>
>
It's possible that calling flush on the BatchWriter after every add of a
small mutation could cause memory usage to spike and the java garbage
collector to fall behind. I'm not sure this is in our standard test cases.
There are two things we should do to move forwards:
1. We should try to get your application working given these constraints.
Is there a reason you're flushing the writer after every mutation? Can you
limit the calls to flush to groups of mutations? Usually flushing too
frequently it's an indicator that you're trying to do a read-modify-write
loop, and there is often a better performing alternative. If you do need to
flush frequently (e.g. maybe you're using cells as locks) then we'll
probably have to skip to #2.
2. We should try to build a test that replicates this problem in a
simplified environment. Can you share done source code that we can use to
build that test?
Adam
On Aug 2, 2012 1:54 PM, "Matt Parker" <[EMAIL PROTECTED]> wrote:
> I'm flushing/closing the writer after every small "transaction" (i.e. a
> nodes get updated or inserted, or a link record is moved
> (deleted/reinserted).
>
> The client will throw the Out of Memory error, but I can restart the
> client and rerun the same set of operatoins again. So I would assume the
> TServer seems to be uneffected.
>
> On Thu, Aug 2, 2012 at 1:49 PM, William Slacum <
> [EMAIL PROTECTED]> wrote:
>
>> Is it the TServer bombing out or your client or both? How often are you
>> flushing your writer?
>>
>>
>> On Thu, Aug 2, 2012 at 10:34 AM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> for my small test case, I'm storing some basic data in three tables:
>>>
>>> nodes - spatial index (id, list of child nodes, whether it's a leaf node
>>> )
>>> image metadata - (id, bounding box coordinates, a text string of the
>>> bounding box)
>>> link - linking table that tells which images correspond to specific
>>> nodes.
>>>
>>> The image data isn't being stored in Accumulo, yet.
>>>
>>>
>>>
>>> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>>
>>>> are you using native maps? if so, are they being used?
>>>>
>>>>
>>>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> I setup a single instance Accumulo server.
>>>>>
>>>>> I can load 32K rows of image metadata without issue.
>>>>>
>>>>> I have another set of routines that build a dynamic spatial index,
>>>>> where nodes are inserted/updated/deleted over time.
>>>>> These operations are typically done one at a time, where each
>>>>> batchwriter are closed after use.
>>>>>
>>>>> It loads maybe a couple hundred operations, and then it dies with an
>>>>> OutOfMemory error when trying to close a batchwriter.
>>>>>
>>>>> I tried uping the memery settings on my client and on the tserver, but
>>>>> the results were the same.
>>>>>
>>>>> Outside of Accumulo, I can build the whole index in memory without any
>>>>> special JVM memory settings. I was wondering whether anyone else had run
>>>>> into a similar issue?
>>>>>
>>>>
>>>>
>>>
>>
>
+
Adam Fuchs 2012-08-03, 11:45
-
Re: Accumulo 1.4 Memory IssuesMatt Parker 2012-08-02, 17:51
I think I understand your question now.
We haven't compiled them in, but we'll give it try and see if the results
differ.
Thanks for your help.
On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
> for my small test case, I'm storing some basic data in three tables:
>
> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
> image metadata - (id, bounding box coordinates, a text string of the
> bounding box)
> link - linking table that tells which images correspond to specific nodes.
>
> The image data isn't being stored in Accumulo, yet.
>
>
>
> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> are you using native maps? if so, are they being used?
>>
>>
>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> I setup a single instance Accumulo server.
>>>
>>> I can load 32K rows of image metadata without issue.
>>>
>>> I have another set of routines that build a dynamic spatial index, where
>>> nodes are inserted/updated/deleted over time.
>>> These operations are typically done one at a time, where each
>>> batchwriter are closed after use.
>>>
>>> It loads maybe a couple hundred operations, and then it dies with an
>>> OutOfMemory error when trying to close a batchwriter.
>>>
>>> I tried uping the memery settings on my client and on the tserver, but
>>> the results were the same.
>>>
>>> Outside of Accumulo, I can build the whole index in memory without any
>>> special JVM memory settings. I was wondering whether anyone else had run
>>> into a similar issue?
>>>
>>
>>
>
We haven't compiled them in, but we'll give it try and see if the results
differ.
Thanks for your help.
On Thu, Aug 2, 2012 at 1:34 PM, Matt Parker <[EMAIL PROTECTED]> wrote:
> for my small test case, I'm storing some basic data in three tables:
>
> nodes - spatial index (id, list of child nodes, whether it's a leaf node )
> image metadata - (id, bounding box coordinates, a text string of the
> bounding box)
> link - linking table that tells which images correspond to specific nodes.
>
> The image data isn't being stored in Accumulo, yet.
>
>
>
> On Thu, Aug 2, 2012 at 1:25 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> are you using native maps? if so, are they being used?
>>
>>
>> On Thu, Aug 2, 2012 at 1:16 PM, Matt Parker <[EMAIL PROTECTED]>wrote:
>>
>>> I setup a single instance Accumulo server.
>>>
>>> I can load 32K rows of image metadata without issue.
>>>
>>> I have another set of routines that build a dynamic spatial index, where
>>> nodes are inserted/updated/deleted over time.
>>> These operations are typically done one at a time, where each
>>> batchwriter are closed after use.
>>>
>>> It loads maybe a couple hundred operations, and then it dies with an
>>> OutOfMemory error when trying to close a batchwriter.
>>>
>>> I tried uping the memery settings on my client and on the tserver, but
>>> the results were the same.
>>>
>>> Outside of Accumulo, I can build the whole index in memory without any
>>> special JVM memory settings. I was wondering whether anyone else had run
>>> into a similar issue?
>>>
>>
>>
>
+
Matt Parker 2012-08-02, 17:51
