Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> ElasticSearchSink does not work


Copy link to this message
-
Re: ElasticSearchSink does not work
Matt, thanks. I had mentioned in an early mail that the problem was resolved, in the same way as you mentioned.
 
Shushuai 
 

________________________________
 From: Matt Wise <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; shushuai zhu <[EMAIL PROTECTED]>
Sent: Tuesday, June 18, 2013 2:36 PM
Subject: Re: ElasticSearchSink does not work
  
In order for me to use the elasticsearch sink, I had to install the ElasticSearch JAR packages onto my Flume nodes and add them to the environment through the flume-env.sh script. Here's our puppet flume-env.sh template:

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
JAVA_OPTS="-Xms100m -Xmx<%= flume_max_mem.to_i %>m -Dcom.sun.management.jmxremote -Dflume.monitoring.type=http -Dflume.monitoring.port=<%= flume_monitoring_port %>"

>
# Note that the Flume conf directory is always included in the classpath.
FLUME_CLASSPATH="<%= elasticsearch_dest %>/lib/*"

--Matt

On Jun 12, 2013, at 8:04 AM, shushuai zhu <[EMAIL PROTECTED]> wrote:

Hi, just a quick update. Found some other site to download apache-flume-1.3.1-bin.tar.gz like:

>http://apache.mesi.com.ar/flume/1.3.1/apache-flume-1.3.1-bin.tar.gz
>
>After the installation and running, still got the same class not found exception (see below log messages). I suspect some additional jar file is needed for ElasticSearchSink. Anyone have any idea?
>
>Thanks.
>
>Shushuai
>
>
>
>________________________________
> From: shushuai zhu <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Sent: Wednesday, June 12, 2013 10:35 AM
>Subject: Re: ElasticSearchSink does not work
>  
>
>
>Allan, thanks for the reply. In my case, I only used one channel and one sink at the same time.

>About 10 minutes after the data were sent to the Flume agent, some messages were logged in flume.log (see below). It says class org/elasticsearch/common/transport/TransportAddress was not found. This seems indicating that the Cloudera version of Flume does not support ElasticSearchSink. Anyway to add the missing class or some jar file?

>I also tried to download the flume from Flume site:

>http://flume.apache.org/download.html
>http://www.apache.org/dyn/closer.cgi/flume/1.3.1/apache-flume-1.3.1-bin.tar.gz

>But the downloaded apache-flume-1.3.1-bin.tar.gz is complained as not a gzip file nor a tar file on my Linux box (Red Hat 5). Can anyone let me know the exact downloading process? If possible, please provide some step-by-step instruction for downloading and installation.

>Thanks.

>Shushuai


>-------------------------------------------------------------------------------------------------------------
>11 Jun 2013 19:40:37,082 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting
>11 Jun 2013 19:40:37,114 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:conf/flume.conf
>11 Jun 2013 19:40:37,121 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -
 Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: k1 Agent: agent1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
 (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent1]
 sink: k1, type: org.apache.flume.sink.elasticsearch.ElasticSearchSink
 org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
 Method)
 avro-source writes to both channels, you should see data going to both locations.
 on success