Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Fwd: Flume workflow design


Copy link to this message
-
Fwd: Flume workflow design
Flavio Pompermaier 2013-07-18, 22:37
Hi to all,
I'm new to Flume but I'm very excited about it!
I'd like to use it to gather some data, process received messages and then
indexing to solr.
Any suggestion about how to do that with Flume?
I've already tested an Avro source that sends data to HBase,
but my use case requires those messages to be saved in HBase but also
processed and then indexed in Solr (obviously I also need to convert the
object structure to convert them).
I think the first part is quite simple (I just use 2 sinks, one that store
in HBase) and another one that forward to another Avro instance, right?
If messages are sent during a map/reduce job, is the avro source the best
option to send documents to index to my sink (i.e. that is my first part of
the flow that up to now I simulated with an avro source..)?
Best,
Flavio