Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo, mail # user - Running Accumulo straight from Memory


+
Moore, Matthew J. 2012-09-11, 16:02
+
Adam Fuchs 2012-09-11, 21:29
+
Moore, Matthew J. 2012-09-12, 16:32
+
dlmarion@... 2012-09-12, 16:56
+
David Medinets 2012-09-12, 16:57
Copy link to this message
-
RE: Running Accumulo straight from Memory
Adam Fuchs 2012-09-12, 20:53
Even if you are just using memory, minor and major compactions are
important to get compression, handle deletes, get sequential access (cache
line efficiency), use iterators, and introduce locality groups.

Adam
On Sep 12, 2012 12:33 PM, "Moore, Matthew J." <[EMAIL PROTECTED]>
wrote:

> Adam,****
>
> It does look like we are the first to try this.  We are trying to keep
> everything in memory and as a result there is no minor compactions, and
> probably major compactions to make tables larger.  We tried this on SSDs
> using a file system and we were not getting the processing speeds that we
> had wanted.****
>
> ** **
>
> Matt****
>
> ** **
>
> ** **
>
> *From:* [EMAIL PROTECTED][mailto:
> [EMAIL PROTECTED]] *On Behalf
> Of *Adam Fuchs
> *Sent:* Tuesday, September 11, 2012 5:30 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Running Accumulo straight from Memory****
>
> ** **
>
> Matthew,****
>
> ** **
>
> I don't know of anyone who has done this, but I believe you could:****
>
> 1. mount a RAM disk****
>
> 2. point the hdfs core-site.xml fs.default.name property to file:///****
>
> 3. point the accumulo-site.xml instance.dfs.dir property to a directory on
> the RAM disk****
>
> 4. disable the WAL for all tables by setting the accumulo-site.xml
> table.walog.enabled to false****
>
> 5. initialize and start up accumulo as you regularly would and cross your
> fingers
>
> Of course, the "you may lose data" and "this is not an officially
> supported configuration" caveats apply. Out of curiosity, what would you be
> trying to accomplish with this configuration?****
>
> ** **
>
> Adam****
>
> ** **
>
> ** **
>
> On Tue, Sep 11, 2012 at 12:02 PM, Moore, Matthew J. <
> [EMAIL PROTECTED]> wrote:****
>
> Has anyone run Accumulo on a single server straight from memory?  Probably
> using something like a Fusion  IO drive.  We are trying to use it without
> using an SSD or any spinning discs.****
>
>  ****
>
> *Matthew Moore*****
>
> Systems Engineer****
>
> SAIC, ISBU****
>
> Columbia, MD****
>
> 410-312-2542****
>
>  ****
>
> ** **
>
+
David Medinets 2012-09-12, 21:20
+
Adam Fuchs 2012-09-12, 21:27
+
Moore, Matthew J. 2012-09-13, 12:47
+
Keith Turner 2012-09-13, 16:32
+
Eric Newton 2012-09-11, 16:19
+
Moore, Matthew J. 2012-09-11, 16:55
+
Eric Newton 2012-09-11, 17:59
+
William Slacum 2012-09-11, 16:30
+
William Slacum 2012-09-11, 16:30