Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> flume+hbase+zookeeper, can't hang on to zookeeper session


Copy link to this message
-
flume+hbase+zookeeper, can't hang on to zookeeper session
We're having some trouble with the Flume & the HbaseSink working. Seems we
cannot hang on to zookeeper sessions. The flume list recommended I cross
post here as this seems to be something up with the hbaseclient &/or
zookeeper. In all honesty I am probably just doing something dumb and not
seeing it.

Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and
are in normal prod use. I don't think we are hitting our
hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions
in ZK DUMP and I believe it defaults to 2k. I can establish new sessions
using the CLI from the same server and idle there for 10 minutes without
getting dropped. When I start up flume I see the session show up in the ZK
DUMP, see active TCP connection on the port etc., it just keeps
dropping/reconnecting.

Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase
directories from one of our datanode+regionserver's.

*Searched the flume list, this user appeared to have the same problem, not
sure how he fixed it though:*
http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%[EMAIL PROTECTED]%3E

*startup cmd
*/app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf -f
conf/brian.properties  -Dflume.root.logger=INFO,console*
*
*flume-env.sh*
$ cat conf/flume-env.sh

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.

# Enviroment variables can be set here.

#JAVA_HOME=/usr/lib/jvm/java-6-sun

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
#JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH="/app/flume/lib"

FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"

HBASE_HOME="/app/hbase-0.94.0"

HADOOP_HOME="/app/hadoop-1.0.1"
*config*
$ cat conf/brian.properties
#example.conf: A single-node Flume configuration

# Name the components on this agent
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure source1
agent1.sources.source1.type = exec
agent1.sources.source1.command = tail -F /tank/log_dev.log
agent1.sources.source1.batchSize = 1
# Describe sink1
#agent1.sinks.sink1.type = logger
agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
agent1.sinks.sink1.table = brian_test
agent1.sinks.sink1.columnFamily = f1
#agent1.sinks.sink1.serializer org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
#agent1.sinks.sink1.serializer org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
*flume console log*
2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.io.tmpdir=/var/tmp/
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.compiler=<NA>
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.name=FreeBSD
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.arch=amd64
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.version=9.0-RELEASE
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.name=czhang
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.home=/users/czhang
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000
watcher=hconnection
2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)]
The identifier of this process is [EMAIL PROTECTED]
2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbase