Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> AgeOffFilter + moving data to a different table


Copy link to this message
-
Re: AgeOffFilter + moving data to a different table
Can you change the storage process so that each new set of data goes to a
different table? Then you could sweep information to a aggregation table
without any conflicts.
On Wed, Sep 4, 2013 at 1:21 PM, John Vines <[EMAIL PROTECTED]> wrote:

> No, the iterator structures, which filters are built on, do not provide a
> mechanism for writing to another table. While you could put a writer in
> there, it is HIGHLY not recommended because it can potentially cause
> deadlocking. You will need an external process to manage writing of old
> records to a new table.
>
>
> On Wed, Sep 4, 2013 at 1:11 PM, Slater, David M. <[EMAIL PROTECTED]>wrote:
>
>> Hey,****
>>
>> ** **
>>
>> I was wondering if it was possible (and a good idea) to modify an
>> AgeOffFilter so that when a record is aged off, it gets sent to a different
>> table. ****
>>
>> ** **
>>
>> Basically, I am storing netflow data and aggregating that information
>> (through a Combiner) into a single entry. Updates to the aggregated record
>> come regularly when the flow is active, but when the no updates have come
>> in a certain amount of time (e.g. 30 seconds), I assume that the flow is
>> done and want to migrate that to a different table (with further indexing).
>> It is important that I migrate it from the current table so that the
>> combiner doesn’t combine two different flows (separated sufficiently in
>> time) with the same Key (ips, ports, and protocol). ****
>>
>> ** **
>>
>> Is there a way to do this with AgeOffFilters? Would I need an external
>> program running that would periodically scan the table with an ageofffilter
>> and then re-ingest those netflows into a new table? ****
>>
>> ** **
>>
>> Regards,
>> David****
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB