Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # user >> RE: EXTERNAL: Re: Large files in Accumulo


+
Cardon, Tejay E 2012-08-23, 20:37
+
Eric Newton 2012-08-23, 21:05
Copy link to this message
-
Re: EXTERNAL: Re: Large files in Accumulo
The filedata example shows one way to split a file into multiple Values.

Billie
On Thu, Aug 23, 2012 at 2:05 PM, Eric Newton <[EMAIL PROTECTED]> wrote:

> An entire mutation needs to fit in memory several times, so you should not
> attempt to push in a single mutation larger than a 100MB unless you have a
> lot of memory in your tserver/logger.
>
> And while I'm at it, large keys will create large indexes, so try to keep
> your (row,cf,cq,cv) under 100K.
>
> -Eric
>
>
> On Thu, Aug 23, 2012 at 4:37 PM, Cardon, Tejay E <[EMAIL PROTECTED]>wrote:
>
>>  In my case I’ll be doing a document based index store (like the
>> wikisearch example), but my documents may be as large as several GB.  I
>> just wanted to pick the collective brain of the group to see if I’m walking
>> into a major headache.  If it’s never been tried before, then I’ll give it
>> a shot and report back.****
>>
>>
>> Tejay****
>>
>> ** **
>>
>> *From:* William Slacum [mailto:[EMAIL PROTECTED]]
>> *Sent:* Thursday, August 23, 2012 2:07 PM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* EXTERNAL: Re: Large files in Accumulo****
>>
>> ** **
>>
>> Are these RFiles as a whole? I know at some point HBase needed to have
>> entire rows fit into memory; Accumulo does not have this restriction.****
>>
>> On Thu, Aug 23, 2012 at 12:55 PM, Cardon, Tejay E <
>> [EMAIL PROTECTED]> wrote:****
>>
>> Alright, this one’s a quick question.  I’ve been told that HBase does not
>> perform well if large (> 100MB) files are stored in it).  Does Accumulo
>> have similar trouble?  If so, can it be overcome by storing the large files
>> in their own locality group?****
>>
>>  ****
>>
>> Thanks,****
>>
>> Tejay****
>>
>> ** **
>>
>
>
+
Cardon, Tejay E 2012-08-23, 21:34
+
Christopher Tubbs 2012-08-23, 22:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB