Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> ElasticSearchSink does not work


+
shushuai zhu 2013-06-12, 02:37
+
Allan Feid 2013-06-12, 14:09
+
shushuai zhu 2013-06-12, 14:35
+
shushuai zhu 2013-06-12, 15:04
+
Matt Wise 2013-06-18, 18:36
Copy link to this message
-
Re: ElasticSearchSink does not work
Matt, thanks. I had mentioned in an early mail that the problem was resolved, in the same way as you mentioned.
 
Shushuai 
 

________________________________
 From: Matt Wise <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; shushuai zhu <[EMAIL PROTECTED]>
Sent: Tuesday, June 18, 2013 2:36 PM
Subject: Re: ElasticSearchSink does not work
  
In order for me to use the elasticsearch sink, I had to install the ElasticSearch JAR packages onto my Flume nodes and add them to the environment through the flume-env.sh script. Here's our puppet flume-env.sh template:

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
JAVA_OPTS="-Xms100m -Xmx<%= flume_max_mem.to_i %>m -Dcom.sun.management.jmxremote -Dflume.monitoring.type=http -Dflume.monitoring.port=<%= flume_monitoring_port %>"

>
# Note that the Flume conf directory is always included in the classpath.
FLUME_CLASSPATH="<%= elasticsearch_dest %>/lib/*"

--Matt

On Jun 12, 2013, at 8:04 AM, shushuai zhu <[EMAIL PROTECTED]> wrote:

Hi, just a quick update. Found some other site to download apache-flume-1.3.1-bin.tar.gz like:

>http://apache.mesi.com.ar/flume/1.3.1/apache-flume-1.3.1-bin.tar.gz
>
>After the installation and running, still got the same class not found exception (see below log messages). I suspect some additional jar file is needed for ElasticSearchSink. Anyone have any idea?
>
>Thanks.
>
>Shushuai
>
>
>
>________________________________
> From: shushuai zhu <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Sent: Wednesday, June 12, 2013 10:35 AM
>Subject: Re: ElasticSearchSink does not work
>  
>
>
>Allan, thanks for the reply. In my case, I only used one channel and one sink at the same time.

>About 10 minutes after the data were sent to the Flume agent, some messages were logged in flume.log (see below). It says class org/elasticsearch/common/transport/TransportAddress was not found. This seems indicating that the Cloudera version of Flume does not support ElasticSearchSink. Anyway to add the missing class or some jar file?

>I also tried to download the flume from Flume site:

>http://flume.apache.org/download.html
>http://www.apache.org/dyn/closer.cgi/flume/1.3.1/apache-flume-1.3.1-bin.tar.gz

>But the downloaded apache-flume-1.3.1-bin.tar.gz is complained as not a gzip file nor a tar file on my Linux box (Red Hat 5). Can anyone let me know the exact downloading process? If possible, please provide some step-by-step instruction for downloading and installation.

>Thanks.

>Shushuai


>-------------------------------------------------------------------------------------------------------------
>11 Jun 2013 19:40:37,082 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting
>11 Jun 2013 19:40:37,114 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:conf/flume.conf
>11 Jun 2013 19:40:37,121 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -
 Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: k1 Agent: agent1
>11 Jun 2013 19:40:37,122 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
 (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent1]
 sink: k1, type: org.apache.flume.sink.elasticsearch.ElasticSearchSink
 org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
 Method)
 avro-source writes to both channels, you should see data going to both locations.
 on success
+
Edward Sargisson 2013-06-12, 16:10
+
shushuai zhu 2013-06-13, 01:01
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB