Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> compressing values returned to scanner


Copy link to this message
-
Re: compressing values returned to scanner
Someone can correct me if I'm wrong, but I believe the file compression
option you quoted is for the RFiles in HDFS. You can enable compression
there and will still see some benefit even if you compress the values on
ingest.

On Mon, Oct 1, 2012 at 12:40 PM, ameet kini <[EMAIL PROTECTED]> wrote:

> That is exactly my use case (ingest once, serve often, no server-side
> iterators).
>
> And I'm doing pre-compression on ingest. I was just looking to do away
> with app-level compression code. Not a biggie.
>
> Ameet
>
>
> On Mon, Oct 1, 2012 at 3:32 PM, William Slacum <
> [EMAIL PROTECTED]> wrote:
>
>> If you aren't often looking at the data in the value on the tablet server
>> (like in an iterator), you can also pre-compress your values on ingest.
>>
>>
>> On Mon, Oct 1, 2012 at 12:19 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> You could compress the data in the value, and decompress the data upon
>>> receipt by the scanner.
>>>
>>>
>>> On Mon, Oct 1, 2012 at 3:03 PM, ameet kini <[EMAIL PROTECTED]> wrote:
>>>
>>>>
>>>> My understanding of compression in Accumulo 1.4.1 is that it is on by
>>>> default and that data is decompressed by the tablet server, so data on the
>>>> wire between server/client is decompressed. Is there a way to shift the
>>>> decompression from happening on the server to the client? I have a use case
>>>> where each Value in my table is relatively large (~ 8MB) and I can benefit
>>>> from compression over the wire. I don't have any server side iterators, so
>>>> the values don't need to be decompressed by the tablet server. Also, each
>>>> scan returns a few rows, so client-side decompression can be fast.
>>>>
>>>> The only way I can think of now is to disable compression on that
>>>> table, and handle compression/decompression in the application. But if
>>>> there is a way to do this in Accumulo, I'd prefer that.
>>>>
>>>> Thanks,
>>>> Ameet
>>>>
>>>
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB