Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume HbaseSink ZK woes


+
iain wright 2012-10-08, 19:29
Copy link to this message
-
Re: Flume HbaseSink ZK woes
Hi Iain,

I am not too sure of this issue. It looks like something to do with the HBaseClient. Can you try pinging HBase user list and see if this is a known issue? Did you try the async hbase sink? That is the recommended sink, I's suggest using that.

Thanks,
Hari
--
Hari Shreedharan
On Monday, October 8, 2012 at 12:29 PM, iain wright wrote:

> We're having some trouble with the Flume & the HbaseSink. Seems we cannot hang on to zookeeper sessions. Using 1.3 ng, hbase 0.94. Hbase & Zoo both look fine, Don't think we are hitting our hbase.zookeeper.property.maxClientCnxns as we only have about 95 sesions and I believe it defaults to 2k. I can establish new sessions using the CLI from the same server and idle there for 10 minutes without getting dropped.
>
> Flume & zookeeper logs below, using the same hadoop & hbase directories from our regionserver's.
>
> Searched the list, this user appeared to have the same problem, not sure how he fixed it though:
> http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%[EMAIL PROTECTED]%3E
>
> startup cmd
> /app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf -f conf/brian.properties  -Dflume.root.logger=INFO,console
>
> flume-env.sh (http://flume-env.sh)
> $ cat conf/flume-env.sh (http://flume-env.sh)
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements.  See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership.  The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License.  You may obtain a copy of the License at
> #
> #     http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # If this file is placed at FLUME_CONF_DIR/flume-env.sh (http://flume-env.sh), it will be sourced
> # during Flume startup.
>
> # Enviroment variables can be set here.
>
> #JAVA_HOME=/usr/lib/jvm/java-6-sun
>
> # Give Flume more memory and pre-allocate, enable remote monitoring via JMX
> #JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
>
> # Note that the Flume conf directory is always included in the classpath.
> #FLUME_CLASSPATH="/app/flume/lib"
>
> FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"
>
> HBASE_HOME="/app/hbase-0.94.0"
>
> HADOOP_HOME="/app/hadoop-1.0.1"
>
>
> config
> $ cat conf/brian.properties
> #example.conf: A single-node Flume configuration
>
> # Name the components on this agent
> agent1.sources = source1
> agent1.sinks = sink1
> agent1.channels = channel1
>
> # Describe/configure source1
> agent1.sources.source1.type = exec
> agent1.sources.source1.command = tail -F /tank/log_dev.log
> agent1.sources.source1.batchSize = 1
> # Describe sink1
> #agent1.sinks.sink1.type = logger
> agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> agent1.sinks.sink1.table = brian_test
> agent1.sinks.sink1.columnFamily = f1
> #agent1.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
> #agent1.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
>
>
> # Use a channel which buffers events in memory
> agent1.channels.channel1.type = memory
> agent1.channels.channel1.capacity = 1000
> agent1.channels.channel1.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.source1.channels = channel1
> agent1.sinks.sink1.channel = channel1
>
>
> flume console log
> 2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
 (http://www.labctsi.org/)> This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
+
iain wright 2012-10-08, 21:14
+
iain wright 2012-10-08, 23:23
+
iain wright 2012-10-09, 00:03
+
Hari Shreedharan 2012-10-09, 18:17
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB