Kafka, mail # user - RE: shipping logs to s3 or other servers for backups - 2013-06-13, 16:15
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
RE: shipping logs to s3 or other servers for backups
Hi,

In my application, I am storing user events, and I want to partition the
storage by day.

So at the end of a day, I want to take that "file" and ship it to s3 or
another server as a backup.  This way I can replay the events for a
specific day if needed.

These events also have to be in order.

How should I structure things in kafka to ensure that these user events all
belong to the same file (or set of files if the file gets large I believe
kafka splits it into multiple files).

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB