Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> AgeOffFilter + moving data to a different table


Copy link to this message
-
Re: AgeOffFilter + moving data to a different table
Can you change the storage process so that each new set of data goes to a
different table? Then you could sweep information to a aggregation table
without any conflicts.
On Wed, Sep 4, 2013 at 1:21 PM, John Vines <[EMAIL PROTECTED]> wrote:

> No, the iterator structures, which filters are built on, do not provide a
> mechanism for writing to another table. While you could put a writer in
> there, it is HIGHLY not recommended because it can potentially cause
> deadlocking. You will need an external process to manage writing of old
> records to a new table.
>
>
> On Wed, Sep 4, 2013 at 1:11 PM, Slater, David M. <[EMAIL PROTECTED]>wrote:
>
>> Hey,****
>>
>> ** **
>>
>> I was wondering if it was possible (and a good idea) to modify an
>> AgeOffFilter so that when a record is aged off, it gets sent to a different
>> table. ****
>>
>> ** **
>>
>> Basically, I am storing netflow data and aggregating that information
>> (through a Combiner) into a single entry. Updates to the aggregated record
>> come regularly when the flow is active, but when the no updates have come
>> in a certain amount of time (e.g. 30 seconds), I assume that the flow is
>> done and want to migrate that to a different table (with further indexing).
>> It is important that I migrate it from the current table so that the
>> combiner doesn’t combine two different flows (separated sufficiently in
>> time) with the same Key (ips, ports, and protocol). ****
>>
>> ** **
>>
>> Is there a way to do this with AgeOffFilters? Would I need an external
>> program running that would periodically scan the table with an ageofffilter
>> and then re-ingest those netflows into a new table? ****
>>
>> ** **
>>
>> Regards,
>> David****
>>
>
>