Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> flume+hbase+zookeeper, can't hang on to zookeeper session


Copy link to this message
-
flume+hbase+zookeeper, can't hang on to zookeeper session
We're having some trouble with the Flume & the HbaseSink working. Seems we
cannot hang on to zookeeper sessions. The flume list recommended I cross
post here as this seems to be something up with the hbaseclient &/or
zookeeper. In all honesty I am probably just doing something dumb and not
seeing it.

Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and
are in normal prod use. I don't think we are hitting our
hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions
in ZK DUMP and I believe it defaults to 2k. I can establish new sessions
using the CLI from the same server and idle there for 10 minutes without
getting dropped. When I start up flume I see the session show up in the ZK
DUMP, see active TCP connection on the port etc., it just keeps
dropping/reconnecting.

Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase
directories from one of our datanode+regionserver's.

*Searched the flume list, this user appeared to have the same problem, not
sure how he fixed it though:*
http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%[EMAIL PROTECTED]%3E

*startup cmd
*/app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf -f
conf/brian.properties  -Dflume.root.logger=INFO,console*
*
*flume-env.sh*
$ cat conf/flume-env.sh

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.

# Enviroment variables can be set here.

#JAVA_HOME=/usr/lib/jvm/java-6-sun

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
#JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH="/app/flume/lib"

FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"

HBASE_HOME="/app/hbase-0.94.0"

HADOOP_HOME="/app/hadoop-1.0.1"
*config*
$ cat conf/brian.properties
#example.conf: A single-node Flume configuration

# Name the components on this agent
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure source1
agent1.sources.source1.type = exec
agent1.sources.source1.command = tail -F /tank/log_dev.log
agent1.sources.source1.batchSize = 1
# Describe sink1
#agent1.sinks.sink1.type = logger
agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
agent1.sinks.sink1.table = brian_test
agent1.sinks.sink1.columnFamily = f1
#agent1.sinks.sink1.serializer org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
#agent1.sinks.sink1.serializer org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
*flume console log*
2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.io.tmpdir=/var/tmp/
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.compiler=<NA>
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.name=FreeBSD
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.arch=amd64
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.version=9.0-RELEASE
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.name=czhang
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.home=/users/czhang
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000
watcher=hconnection
2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)]
The identifier of this process is [EMAIL PROTECTED]
2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbase
+
iain wright 2012-10-08, 22:56
+
anil gupta 2012-10-10, 01:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB