Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - RE: shipping logs to s3 or other servers for backups


Copy link to this message
-
RE: shipping logs to s3 or other servers for backups
S Ahmed 2013-06-13, 16:15
Hi,

In my application, I am storing user events, and I want to partition the
storage by day.

So at the end of a day, I want to take that "file" and ship it to s3 or
another server as a backup.  This way I can replay the events for a
specific day if needed.

These events also have to be in order.

How should I structure things in kafka to ensure that these user events all
belong to the same file (or set of files if the file gets large I believe
kafka splits it into multiple files).