Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Constant Traffic on port 35872


+
James Stewart 2013-01-17, 01:06
+
Mike Percy 2013-01-17, 01:18
+
James Stewart 2013-01-17, 01:53
Copy link to this message
-
Re: Constant Traffic on port 35872
Mike Percy 2013-01-17, 03:37
I doubt it's the Thrift RPC layer. It's most likely the app.
On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]
> wrote:

> I thought it was only ‘heartbeats’ that were supposed to go via port
> 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec.
> According to the master it’s only seeing heartbeats every 60 sec now, and
> yet I still get constantly spammed with data on port 35872 from every node.
> ****
>
> ** **
>
> It does look like metric collection or config reporting of some kind, like
> it’s reporting the configuration of the sources/sinks and even data about
> the JVM:****
>
> ** **
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013...
> rt.vmname...!Java HotSpot(TM) 64-Bit Server
> VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun
> Microsystems Inc.****
>
> ..........max.....@
> ......mem.other.used................mem.other.committed.....s......mem.heap.max...........
> ****
>
> ** **
>
> But it’s just the same data over and over again every second. This traffic
> is travelling across a WAN and with a lot of nodes it’s a significant
> enough amount of data to be a problem.****
>
> ** **
>
> I don’t know much about Java, but could this be something to do with
> Thrift?****
>
> ** **
>
> ** **
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, 17 January 2013 12:19 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Constant Traffic on port 35872****
>
> ** **
>
> I know next to nothing about Flume OG but if I had to guess I'd say it's
> either a heartbeat or metrics collection. Why do you want it to stop?****
>
> ** **
>
> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <
> [EMAIL PROTECTED]> wrote:****
>
> Hello all,****
>
>  ****
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a
> position to upgrade at the moment due to custom decorators + sinks.****
>
>  ****
>
> I’m seeing constant traffic from my various flume nodes back to my master
> on port 35872. Even after increasing my timeout period to 60 sec and
> disabling all custom sources/sinks/decorators, I am still constantly
> receiving packets from all of my nodes back to my master. I have included a
> dump of the tcp packets below – I receive this same traffic from every node
> every 1-2 sec.****
>
>  ****
>
> Does anybody have any idea what is generating this traffic and how I can
> stop it?****
>
>  ****
>
> Many thanks,****
>
>  ****
>
> James****
>
>  ****
>
>  ****
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu
> Jan 01 10:00:00 EST
> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu
+
James Stewart 2013-01-17, 04:02
+
Alexander Alten-Lorenz 2013-01-17, 07:00
+
Jeong-shik Jang 2013-01-17, 10:31
+
James Stewart 2013-01-17, 21:46