Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> RE: shipping logs to s3 or other servers for backups


Copy link to this message
-
RE: shipping logs to s3 or other servers for backups
Hi,

In my application, I am storing user events, and I want to partition the
storage by day.

So at the end of a day, I want to take that "file" and ship it to s3 or
another server as a backup.  This way I can replay the events for a
specific day if needed.

These events also have to be in order.

How should I structure things in kafka to ensure that these user events all
belong to the same file (or set of files if the file gets large I believe
kafka splits it into multiple files).

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB