Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - [NEW FEATURE] - FLUME-1687 - Solr Sink for Apache Flume Now In Beta


Copy link to this message
-
[NEW FEATURE] - FLUME-1687 - Solr Sink for Apache Flume Now In Beta
Israel Ekpo 2013-04-20, 17:43
Fellow Flume Users,

I have just created an Apache Solr sink for Flume against version 1.3.1

This has been tested and it works fine.

This sink is a great alternative to the ElasticSearchSink.

There are some flume users that have experience with Apache Solr but do not
necessarily understand how to get ElasticSearch up and running.

Having a SolrSink as an alternative could be very helpful in creating a
user interface for searching through event and log data collected with
Flume using Apache Solr.

This sink essentially picks up events from the channel and uses the
ConcurrentUpdateSolrServer client from SolrJ to send events in batches as
SolrInputDocuments using multipe worker threads to Apache Solr.

The number of documents/events per batch and the number of worker threads
can be configured.

There is a default serializer as well.

If you have the chance please try it out and let me know what you think.

It would be great to gather some feedback from users before finalizing the
features and configuration options for this sink.

Instructions are available here on how to set up and configure this sink.

https://cwiki.apache.org/confluence/display/FLUME/How+to+Setup+Solr+Sink+for+Flume

JAR files for the feature as well as its dependencies are available here

https://issues.apache.org/jira/browse/FLUME-1687