James Stewart
2013-01-17, 01:06
Mike Percy
2013-01-17, 01:18
James Stewart
2013-01-17, 01:53
Mike Percy
2013-01-17, 03:37
James Stewart
2013-01-17, 04:02
Alexander Alten-Lorenz
2013-01-17, 07:00
Jeong-shik Jang
2013-01-17, 10:31
James Stewart
2013-01-17, 21:46
-
Constant Traffic on port 35872James Stewart 2013-01-17, 01:06
Hello all,
I'm using flume 0.9.4 - before anybody mentions it, we aren't in a position to upgrade at the moment due to custom decorators + sinks.
I'm seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below - I receive this same traffic from every node every 1-2 sec.
Does anybody have any idea what is generating this traffic and how I can stop it?
Many thanks,
James
V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<...
I..
B...w. ."..CC1L...........
...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....my_physical_node.my_domain.com
.......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number of bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber of events............
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202_apps.NullSink
............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
...........(pn-my_logical_node.aus.optiumber of bytes........
11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872: P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<..c
I..
..w. .".:CC1L...........
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-my_physical_node.my_domain.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]...........
________________________________
Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defects or that the integrity of this communication has been maintained. Any views expressed in this communication are those of the individual sender. Optiver does not accept liability for any loss or damage caused directly or indirectly by this communication or its use.
Please consider the environment before printing this email.
I'm using flume 0.9.4 - before anybody mentions it, we aren't in a position to upgrade at the moment due to custom decorators + sinks.
I'm seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below - I receive this same traffic from every node every 1-2 sec.
Does anybody have any idea what is generating this traffic and how I can stop it?
Many thanks,
James
V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<...
I..
B...w. ."..CC1L...........
...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....my_physical_node.my_domain.com
.......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number of bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber of events............
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202_apps.NullSink
............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
...........(pn-my_logical_node.aus.optiumber of bytes........
11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872: P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<..c
I..
..w. .".:CC1L...........
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-my_physical_node.my_domain.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]...........
________________________________
Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defects or that the integrity of this communication has been maintained. Any views expressed in this communication are those of the individual sender. Optiver does not accept liability for any loss or damage caused directly or indirectly by this communication or its use.
Please consider the environment before printing this email.
-
Re: Constant Traffic on port 35872Mike Percy 2013-01-17, 01:18
I know next to nothing about Flume OG but if I had to guess I'd say it's
either a heartbeat or metrics collection. Why do you want it to stop?
On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]
> wrote:
> Hello all,****
>
> ** **
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a
> position to upgrade at the moment due to custom decorators + sinks.****
>
> ** **
>
> I’m seeing constant traffic from my various flume nodes back to my master
> on port 35872. Even after increasing my timeout period to 60 sec and
> disabling all custom sources/sinks/decorators, I am still constantly
> receiving packets from all of my nodes back to my master. I have included a
> dump of the tcp packets below – I receive this same traffic from every node
> every 1-2 sec.****
>
> ** **
>
> Does anybody have any idea what is generating this traffic and how I can
> stop it?****
>
> ** **
>
> Many thanks,****
>
> ** **
>
> James****
>
> ** **
>
> ** **
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu
> Jan 01 10:00:00 EST
> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu
> Jan 17 11:45:44 EST 2013...****
>
> 11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872:
> . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935
> 3787992397>****
>
> E.....@.<...****
>
> I..****
>
> B...w. ."..CC1L...........****
>
>
> ...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....
> my_physical_node.my_domain.com****
>
> .......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number
> of
> bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber
> of events............****
>
> NullSource....type...****
>
> ...lSource****
>
> ............name....NullSink....type....NullSinksr0202_apps.NullSink****
>
> ............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource*
> ***
>
> NullSource....type...****
>
> ...lSource****
>
>
> ............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
> ****
>
> ...........(pn-my_logical_node.aus.optiumber of bytes........****
>
> 11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872:
> P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935
> 3787992397>****
>
> E.....@.<..c****
>
> I..****
>
> ..w. .".:CC1L...........****
either a heartbeat or metrics collection. Why do you want it to stop?
On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]
> wrote:
> Hello all,****
>
> ** **
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a
> position to upgrade at the moment due to custom decorators + sinks.****
>
> ** **
>
> I’m seeing constant traffic from my various flume nodes back to my master
> on port 35872. Even after increasing my timeout period to 60 sec and
> disabling all custom sources/sinks/decorators, I am still constantly
> receiving packets from all of my nodes back to my master. I have included a
> dump of the tcp packets below – I receive this same traffic from every node
> every 1-2 sec.****
>
> ** **
>
> Does anybody have any idea what is generating this traffic and how I can
> stop it?****
>
> ** **
>
> Many thanks,****
>
> ** **
>
> James****
>
> ** **
>
> ** **
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu
> Jan 01 10:00:00 EST
> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu
> Jan 17 11:45:44 EST 2013...****
>
> 11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872:
> . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935
> 3787992397>****
>
> E.....@.<...****
>
> I..****
>
> B...w. ."..CC1L...........****
>
>
> ...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....
> my_physical_node.my_domain.com****
>
> .......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number
> of
> bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber
> of events............****
>
> NullSource....type...****
>
> ...lSource****
>
> ............name....NullSink....type....NullSinksr0202_apps.NullSink****
>
> ............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource*
> ***
>
> NullSource....type...****
>
> ...lSource****
>
>
> ............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
> ****
>
> ...........(pn-my_logical_node.aus.optiumber of bytes........****
>
> 11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872:
> P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935
> 3787992397>****
>
> E.....@.<..c****
>
> I..****
>
> ..w. .".:CC1L...........****
-
RE: Constant Traffic on port 35872James Stewart 2013-01-17, 01:53
I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]...........
But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
I don’t know much about Java, but could this be something to do with Thrift?
From: Mike Percy [mailto:[EMAIL PROTECTED]]
Sent: Thursday, 17 January 2013 12:19 PM
To: [EMAIL PROTECTED]
Subject: Re: Constant Traffic on port 35872
I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hello all,
I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
Does anybody have any idea what is generating this traffic and how I can stop it?
Many thanks,
James
V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<<mailto:E.....@.%3c>...
I..
B...w. ."..CC1L...........
...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....my_physical_node.my_domain.com<http://my_physical_node.my_domain.com>
.......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number of bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber of events............
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202_apps.NullSink
............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
...........(pn-my_logical_node.aus.optiumber of bytes........
11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872: P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<..c<mailto:E.....@.%3c..c>
I..
..w. .".:CC1L...........
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-my_physical_node.my_domain.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
________________________________
Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defe
It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]...........
But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
I don’t know much about Java, but could this be something to do with Thrift?
From: Mike Percy [mailto:[EMAIL PROTECTED]]
Sent: Thursday, 17 January 2013 12:19 PM
To: [EMAIL PROTECTED]
Subject: Re: Constant Traffic on port 35872
I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hello all,
I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
Does anybody have any idea what is generating this traffic and how I can stop it?
Many thanks,
James
V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<<mailto:E.....@.%3c>...
I..
B...w. ."..CC1L...........
...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....my_physical_node.my_domain.com<http://my_physical_node.my_domain.com>
.......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number of bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber of events............
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202_apps.NullSink
............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
...........(pn-my_logical_node.aus.optiumber of bytes........
11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872: P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<..c<mailto:E.....@.%3c..c>
I..
..w. .".:CC1L...........
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-my_physical_node.my_domain.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
________________________________
Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defe
-
Re: Constant Traffic on port 35872Mike Percy 2013-01-17, 03:37
I doubt it's the Thrift RPC layer. It's most likely the app.
On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]
> wrote:
> I thought it was only ‘heartbeats’ that were supposed to go via port
> 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec.
> According to the master it’s only seeing heartbeats every 60 sec now, and
> yet I still get constantly spammed with data on port 35872 from every node.
> ****
>
> ** **
>
> It does look like metric collection or config reporting of some kind, like
> it’s reporting the configuration of the sources/sinks and even data about
> the JVM:****
>
> ** **
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013...
> rt.vmname...!Java HotSpot(TM) 64-Bit Server
> VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun
> Microsystems Inc.****
>
> ..........max.....@
> ......mem.other.used................mem.other.committed.....s......mem.heap.max...........
> ****
>
> ** **
>
> But it’s just the same data over and over again every second. This traffic
> is travelling across a WAN and with a lot of nodes it’s a significant
> enough amount of data to be a problem.****
>
> ** **
>
> I don’t know much about Java, but could this be something to do with
> Thrift?****
>
> ** **
>
> ** **
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, 17 January 2013 12:19 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Constant Traffic on port 35872****
>
> ** **
>
> I know next to nothing about Flume OG but if I had to guess I'd say it's
> either a heartbeat or metrics collection. Why do you want it to stop?****
>
> ** **
>
> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <
> [EMAIL PROTECTED]> wrote:****
>
> Hello all,****
>
> ****
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a
> position to upgrade at the moment due to custom decorators + sinks.****
>
> ****
>
> I’m seeing constant traffic from my various flume nodes back to my master
> on port 35872. Even after increasing my timeout period to 60 sec and
> disabling all custom sources/sinks/decorators, I am still constantly
> receiving packets from all of my nodes back to my master. I have included a
> dump of the tcp packets below – I receive this same traffic from every node
> every 1-2 sec.****
>
> ****
>
> Does anybody have any idea what is generating this traffic and how I can
> stop it?****
>
> ****
>
> Many thanks,****
>
> ****
>
> James****
>
> ****
>
> ****
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu
> Jan 01 10:00:00 EST
> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu
On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]
> wrote:
> I thought it was only ‘heartbeats’ that were supposed to go via port
> 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec.
> According to the master it’s only seeing heartbeats every 60 sec now, and
> yet I still get constantly spammed with data on port 35872 from every node.
> ****
>
> ** **
>
> It does look like metric collection or config reporting of some kind, like
> it’s reporting the configuration of the sources/sinks and even data about
> the JVM:****
>
> ** **
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013...
> rt.vmname...!Java HotSpot(TM) 64-Bit Server
> VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun
> Microsystems Inc.****
>
> ..........max.....@
> ......mem.other.used................mem.other.committed.....s......mem.heap.max...........
> ****
>
> ** **
>
> But it’s just the same data over and over again every second. This traffic
> is travelling across a WAN and with a lot of nodes it’s a significant
> enough amount of data to be a problem.****
>
> ** **
>
> I don’t know much about Java, but could this be something to do with
> Thrift?****
>
> ** **
>
> ** **
>
> *From:* Mike Percy [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, 17 January 2013 12:19 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Constant Traffic on port 35872****
>
> ** **
>
> I know next to nothing about Flume OG but if I had to guess I'd say it's
> either a heartbeat or metrics collection. Why do you want it to stop?****
>
> ** **
>
> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <
> [EMAIL PROTECTED]> wrote:****
>
> Hello all,****
>
> ****
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a
> position to upgrade at the moment due to custom decorators + sinks.****
>
> ****
>
> I’m seeing constant traffic from my various flume nodes back to my master
> on port 35872. Even after increasing my timeout period to 60 sec and
> disabling all custom sources/sinks/decorators, I am still constantly
> receiving packets from all of my nodes back to my master. I have included a
> dump of the tcp packets below – I receive this same traffic from every node
> every 1-2 sec.****
>
> ****
>
> Does anybody have any idea what is generating this traffic and how I can
> stop it?****
>
> ****
>
> Many thanks,****
>
> ****
>
> James****
>
> ****
>
> ****
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu
> Jan 01 10:00:00 EST
> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu
-
RE: Constant Traffic on port 35872James Stewart 2013-01-17, 04:02
Yeah, I’ve just realised that it’s*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
I can understand why this config data would be sent back to the master occasionally but I don’t understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
From: Mike Percy [mailto:[EMAIL PROTECTED]]
Sent: Thursday, 17 January 2013 2:37 PM
To: [EMAIL PROTECTED]
Subject: Re: Constant Traffic on port 35872
I doubt it's the Thrift RPC layer. It's most likely the app.
On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
I don’t know much about Java, but could this be something to do with Thrift?
From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Thursday, 17 January 2013 12:19 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Constant Traffic on port 35872
I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hello all,
I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
Does anybody have any idea what is generating this traffic and how I can stop it?
Many thanks,
James
V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<<mailto:E.....@.%3c>...
I..
B...w. ."..CC1L...........
...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....my_physical_node.my_domain.com<http://my_physical_node.my_domain.com>
.......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number of bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber of events............
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202_apps.NullSink
............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
...........(pn-my_logical_node.aus.optiumber of bytes........
11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872: P 333225
So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
I can understand why this config data would be sent back to the master occasionally but I don’t understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
From: Mike Percy [mailto:[EMAIL PROTECTED]]
Sent: Thursday, 17 January 2013 2:37 PM
To: [EMAIL PROTECTED]
Subject: Re: Constant Traffic on port 35872
I doubt it's the Thrift RPC layer. It's most likely the app.
On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
I don’t know much about Java, but could this be something to do with Thrift?
From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Thursday, 17 January 2013 12:19 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Constant Traffic on port 35872
I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hello all,
I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
Does anybody have any idea what is generating this traffic and how I can stop it?
Many thanks,
James
V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935 3787992397>
E.....@.<<mailto:E.....@.%3c>...
I..
B...w. ."..CC1L...........
...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....my_physical_node.my_domain.com<http://my_physical_node.my_domain.com>
.......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number of bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber of events............
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202_apps.NullSink
............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource
NullSource....type...
...lSource
............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
...........(pn-my_logical_node.aus.optiumber of bytes........
11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872: P 333225
-
Re: Constant Traffic on port 35872Alexander Alten-Lorenz 2013-01-17, 07:00
Depends on the architecture, since the nodes are configured per master's webui. The master calls regularly the in-memory config and spread them around. This is needed for HA, as example.
Flume 1.x up has another architecture.
- Alex
On Jan 17, 2013, at 5:02 AM, James Stewart <[EMAIL PROTECTED]> wrote:
> Yeah, I’ve just realised that it’s*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
>
> So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
>
> I can understand why this config data would be sent back to the master occasionally but I don’t understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
>
> From: Mike Percy [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, 17 January 2013 2:37 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Constant Traffic on port 35872
>
> I doubt it's the Thrift RPC layer. It's most likely the app.
>
> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
>
> It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
>
> But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
>
> I don’t know much about Java, but could this be something to do with Thrift?
>
>
> From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
> Sent: Thursday, 17 January 2013 12:19 PM
> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> Subject: Re: Constant Traffic on port 35872
>
> I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
>
> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Hello all,
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
>
> I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
>
> Does anybody have any idea what is generating this traffic and how I can stop it?
>
> Many thanks,
>
> James
>
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Flume 1.x up has another architecture.
- Alex
On Jan 17, 2013, at 5:02 AM, James Stewart <[EMAIL PROTECTED]> wrote:
> Yeah, I’ve just realised that it’s*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
>
> So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
>
> I can understand why this config data would be sent back to the master occasionally but I don’t understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
>
> From: Mike Percy [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, 17 January 2013 2:37 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Constant Traffic on port 35872
>
> I doubt it's the Thrift RPC layer. It's most likely the app.
>
> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
>
> It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
>
> But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
>
> I don’t know much about Java, but could this be something to do with Thrift?
>
>
> From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
> Sent: Thursday, 17 January 2013 12:19 PM
> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> Subject: Re: Constant Traffic on port 35872
>
> I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
>
> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Hello all,
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
>
> I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
>
> Does anybody have any idea what is generating this traffic and how I can stop it?
>
> Many thanks,
>
> James
>
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu Jan 01 10:00:00 EST 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu Jan 17 11:45:44 EST 2013...
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
-
Re: Constant Traffic on port 35872Jeong-shik Jang 2013-01-17, 10:31
You may want to check property named flume.reporter.poller.period in
flume-conf.xml or flume-site.xml; default value is 2000 (millis) in case
of not being defined.
if you cannot find the property in flume-conf.xml, add it to
flume-site.xml and set value to 60000 (60 sec).
I am not sure which version of flume-og you are using, but in what I am
using (v0.9.4), the related class seems to have been deprecated.
- JS
On 1/17/13 4:00 PM, Alexander Alten-Lorenz wrote:
> Depends on the architecture, since the nodes are configured per master's webui. The master calls regularly the in-memory config and spread them around. This is needed for HA, as example.
> Flume 1.x up has another architecture.
>
> - Alex
>
> On Jan 17, 2013, at 5:02 AM, James Stewart <[EMAIL PROTECTED]> wrote:
>
>> Yeah, I’ve just realised that it’s*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
>>
>> So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
>>
>> I can understand why this config data would be sent back to the master occasionally but I don’t understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, 17 January 2013 2:37 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Constant Traffic on port 35872
>>
>> I doubt it's the Thrift RPC layer. It's most likely the app.
>>
>> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
>>
>> It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
>>
>> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
>>
>> But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
>>
>> I don’t know much about Java, but could this be something to do with Thrift?
>>
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
>> Sent: Thursday, 17 January 2013 12:19 PM
>> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
>> Subject: Re: Constant Traffic on port 35872
>>
>> I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
>>
>> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Hello all,
>>
>> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
>>
>> I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
flume-conf.xml or flume-site.xml; default value is 2000 (millis) in case
of not being defined.
if you cannot find the property in flume-conf.xml, add it to
flume-site.xml and set value to 60000 (60 sec).
I am not sure which version of flume-og you are using, but in what I am
using (v0.9.4), the related class seems to have been deprecated.
- JS
On 1/17/13 4:00 PM, Alexander Alten-Lorenz wrote:
> Depends on the architecture, since the nodes are configured per master's webui. The master calls regularly the in-memory config and spread them around. This is needed for HA, as example.
> Flume 1.x up has another architecture.
>
> - Alex
>
> On Jan 17, 2013, at 5:02 AM, James Stewart <[EMAIL PROTECTED]> wrote:
>
>> Yeah, I’ve just realised that it’s*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
>>
>> So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
>>
>> I can understand why this config data would be sent back to the master occasionally but I don’t understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, 17 January 2013 2:37 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Constant Traffic on port 35872
>>
>> I doubt it's the Thrift RPC layer. It's most likely the app.
>>
>> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> I thought it was only ‘heartbeats’ that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it’s only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
>>
>> It does look like metric collection or config reporting of some kind, like it’s reporting the configuration of the sources/sinks and even data about the JVM:
>>
>> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
>>
>> But it’s just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it’s a significant enough amount of data to be a problem.
>>
>> I don’t know much about Java, but could this be something to do with Thrift?
>>
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
>> Sent: Thursday, 17 January 2013 12:19 PM
>> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
>> Subject: Re: Constant Traffic on port 35872
>>
>> I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
>>
>> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Hello all,
>>
>> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a position to upgrade at the moment due to custom decorators + sinks.
>>
>> I’m seeing constant traffic from my various flume nodes back to my master on port 35872. Even after increasing my timeout period to 60 sec and disabling all custom sources/sinks/decorators, I am still constantly receiving packets from all of my nodes back to my master. I have included a dump of the tcp packets below – I receive this same traffic from every node every 1-2 sec.
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
-
RE: Constant Traffic on port 35872James Stewart 2013-01-17, 21:46
That was it!
I added flume.reporter.poller.period in flume-conf.xml with a value of 60000 and the reporting traffic is now back to a sensible level...:-).
Thanks so much for your replies, they really helped me out.
-----Original Message-----
From: Jeong-shik Jang [mailto:[EMAIL PROTECTED]]
Sent: Thursday, 17 January 2013 9:32 PM
To: [EMAIL PROTECTED]
Subject: Re: Constant Traffic on port 35872
You may want to check property named flume.reporter.poller.period in flume-conf.xml or flume-site.xml; default value is 2000 (millis) in case of not being defined.
if you cannot find the property in flume-conf.xml, add it to flume-site.xml and set value to 60000 (60 sec).
I am not sure which version of flume-og you are using, but in what I am using (v0.9.4), the related class seems to have been deprecated.
- JS
On 1/17/13 4:00 PM, Alexander Alten-Lorenz wrote:
> Depends on the architecture, since the nodes are configured per master's webui. The master calls regularly the in-memory config and spread them around. This is needed for HA, as example.
> Flume 1.x up has another architecture.
>
> - Alex
>
> On Jan 17, 2013, at 5:02 AM, James Stewart <[EMAIL PROTECTED]> wrote:
>
>> Yeah, I've just realised that it's*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
>>
>> So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
>>
>> I can understand why this config data would be sent back to the master occasionally but I don't understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, 17 January 2013 2:37 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Constant Traffic on port 35872
>>
>> I doubt it's the Thrift RPC layer. It's most likely the app.
>>
>> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> I thought it was only 'heartbeats' that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it's only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
>>
>> It does look like metric collection or config reporting of some kind, like it's reporting the configuration of the sources/sinks and even data about the JVM:
>>
>> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
>>
>> But it's just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it's a significant enough amount of data to be a problem.
>>
>> I don't know much about Java, but could this be something to do with Thrift?
>>
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
>> Sent: Thursday, 17 January 2013 12:19 PM
>> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
>> Subject: Re: Constant Traffic on port 35872
>>
>> I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
>>
>> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Hello all,
>>
>> I'm using flume 0.9.4 - before anybody mentions it, we aren't in a position to upgrade at the moment due to custom decorators + sinks.
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
I added flume.reporter.poller.period in flume-conf.xml with a value of 60000 and the reporting traffic is now back to a sensible level...:-).
Thanks so much for your replies, they really helped me out.
-----Original Message-----
From: Jeong-shik Jang [mailto:[EMAIL PROTECTED]]
Sent: Thursday, 17 January 2013 9:32 PM
To: [EMAIL PROTECTED]
Subject: Re: Constant Traffic on port 35872
You may want to check property named flume.reporter.poller.period in flume-conf.xml or flume-site.xml; default value is 2000 (millis) in case of not being defined.
if you cannot find the property in flume-conf.xml, add it to flume-site.xml and set value to 60000 (60 sec).
I am not sure which version of flume-og you are using, but in what I am using (v0.9.4), the related class seems to have been deprecated.
- JS
On 1/17/13 4:00 PM, Alexander Alten-Lorenz wrote:
> Depends on the architecture, since the nodes are configured per master's webui. The master calls regularly the in-memory config and spread them around. This is needed for HA, as example.
> Flume 1.x up has another architecture.
>
> - Alex
>
> On Jan 17, 2013, at 5:02 AM, James Stewart <[EMAIL PROTECTED]> wrote:
>
>> Yeah, I've just realised that it's*exactly* the same data that is returned when you connect to http://my.flume.node:35862, (for monitoring etc). Even the order in which the metrics are sent is the same.
>>
>> So it seems that the node is generating this configuration data and pumping it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of largely unnecessary traffic per node, which soon adds up over a WAN.
>>
>> I can understand why this config data would be sent back to the master occasionally but I don't understand why it does so every 1-2 seconds, ignoring flume.config.heartbeat.period.
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, 17 January 2013 2:37 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Constant Traffic on port 35872
>>
>> I doubt it's the Thrift RPC layer. It's most likely the app.
>>
>> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> I thought it was only 'heartbeats' that were supposed to go via port 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According to the master it's only seeing heartbeats every 60 sec now, and yet I still get constantly spammed with data on port 35872 from every node.
>>
>> It does look like metric collection or config reporting of some kind, like it's reporting the configuration of the sources/sinks and even data about the JVM:
>>
>> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... rt.vmname...!Java HotSpot(TM) 64-Bit Server VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun Microsystems Inc.
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>...........
>>
>> But it's just the same data over and over again every second. This traffic is travelling across a WAN and with a lot of nodes it's a significant enough amount of data to be a problem.
>>
>> I don't know much about Java, but could this be something to do with Thrift?
>>
>>
>> From: Mike Percy [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
>> Sent: Thursday, 17 January 2013 12:19 PM
>> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
>> Subject: Re: Constant Traffic on port 35872
>>
>> I know next to nothing about Flume OG but if I had to guess I'd say it's either a heartbeat or metrics collection. Why do you want it to stop?
>>
>> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Hello all,
>>
>> I'm using flume 0.9.4 - before anybody mentions it, we aren't in a position to upgrade at the moment due to custom decorators + sinks.
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
