Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Feedback on real world production experience with Flume


Copy link to this message
-
Re: Feedback on real world production experience with Flume
Hi,

in my former job:
productive, Germany, Web portal. Throughput 600 mb/minute. Logfiles from Windows IIS, Apache. Used in a usual way, no own decorators or sinks. Simply syslog -> bucketing (1 minute rollover) -> hdfs splitted into minutes (YYYYMMDDHHMM).

Stable, some issues (you'll found on the mailing list), but works well if you know what is to do when anything will happen. Btw, NG 1.1.0 is more stable as flume pre 1.x and runs in some productive environments.

- Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

On Apr 21, 2012, at 12:14 AM, Karl Hennig wrote:

> I am investigating automated methods of moving our data from the web tier into HDFS for processing, a process that's performed periodically.
>
> I am looking for feedback from anyone who has actually used Flume in a production setup (redundant, failover) successfully.  I understand it is now being largely rearchitected during its incubation as Apache Flume-NG, so I don't have full confidence in the old, stable releases.
>
> The other option would be to write our own tools.  What methods are you using for these kinds of tasks?  Did you write your own or does Flume (or something else) work for you?
>
> I'm also on the Flume mailing list, but I wanted to ask these questions here because I'm interested in Flume _and_ alternatives.
>
> Thank you!
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB