Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> AgeOffFilter + moving data to a different table


Copy link to this message
-
AgeOffFilter + moving data to a different table
Hey,

I was wondering if it was possible (and a good idea) to modify an AgeOffFilter so that when a record is aged off, it gets sent to a different table.

Basically, I am storing netflow data and aggregating that information (through a Combiner) into a single entry. Updates to the aggregated record come regularly when the flow is active, but when the no updates have come in a certain amount of time (e.g. 30 seconds), I assume that the flow is done and want to migrate that to a different table (with further indexing). It is important that I migrate it from the current table so that the combiner doesn't combine two different flows (separated sufficiently in time) with the same Key (ips, ports, and protocol).

Is there a way to do this with AgeOffFilters? Would I need an external program running that would periodically scan the table with an ageofffilter and then re-ingest those netflows into a new table?

Regards,
David
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB