I am trying to get data into a single table. I am getting very high error
counts from all tablet servers and loggers of which there are 3
each.....all with the same error:

05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at
org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
at
org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
at
org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
at
org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)

I can ping/ssh/etc. freely across all slaves and master nodes.

Thoughts?  Thanks a million,

Fred

have you verified that the tracers are running?

look at /conf/tracers to make sure you have them configured and that they
are running on those servers.

On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <[EMAIL PROTECTED]>wrote:

> I am trying to get data into a single table. I am getting very high error
> counts from all tablet servers and loggers of which there are 3
> each.....all with the same error:
>
> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
> java.net.ConnectException: Connection refused
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>  at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
> at java.net.Socket.connect(Socket.java:546)
> at java.net.Socket.connect(Socket.java:495)
>  at
> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
> at
> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>  at
> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
> at
> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>  at java.util.TimerThread.mainLoop(Timer.java:534)
> at java.util.TimerThread.run(Timer.java:484)
>
> I can ping/ssh/etc. freely across all slaves and master nodes.
>
> Thoughts?  Thanks a million,
>
> Fred
>
>

Hey Marc,

I can't tell you how much I appreciate the quick response. I never touched
the tracers fie, so it still just says localhost. If I have a master and 3
slaves, which IP's go into that file? All, or just the slaves?

Thanks!

Fred

On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:

> have you verified that the tracers are running?
>
> look at /conf/tracers to make sure you have them configured and that they
> are running on those servers.
>
>
> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <[EMAIL PROTECTED]>wrote:
>
>> I am trying to get data into a single table. I am getting very high error
>> counts from all tablet servers and loggers of which there are 3
>> each.....all with the same error:
>>
>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>> java.net.ConnectException: Connection refused
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>  at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>> at java.net.Socket.connect(Socket.java:546)
>> at java.net.Socket.connect(Socket.java:495)
>>  at
>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>> at
>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>  at
>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>> at
>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>
>> Thoughts?  Thanks a million,
>>
>> Fred
>>
>>
>
--
Fred Wolfinger
Security Research Engineer
G2, Inc.

302 Sentinel Drive, Suite 300
Annapolis Junction, MD 20701
Office: 301-575-5142
Cell: 443-655-3322

Typically I've only used 1 tracer ( and usually ) on my master; however,
I'm not sure of any suggested guidance, especially in regards to
scalability.

Be aware of the following if you have a different password for the tracer

http://mail-archives.apache.org/mod_mbox/accumulo-user/201202.mbox/%3CCADxc9BmhA4vRqT5Nn6TUAWNSRxtJMA3+K+[EMAIL PROTECTED]%3E

On Wed, Sep 5, 2012 at 1:21 PM, Fred Wolfinger <[EMAIL PROTECTED]>wrote:

> Hey Marc,
>
> I can't tell you how much I appreciate the quick response. I never touched
> the tracers fie, so it still just says localhost. If I have a master and 3
> slaves, which IP's go into that file? All, or just the slaves?
>
> Thanks!
>
> Fred
>
>
> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> have you verified that the tracers are running?
>>
>> look at /conf/tracers to make sure you have them configured and that they
>> are running on those servers.
>>
>>
>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <[EMAIL PROTECTED]
>> > wrote:
>>
>>> I am trying to get data into a single table. I am getting very high
>>> error counts from all tablet servers and loggers of which there are 3
>>> each.....all with the same error:
>>>
>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>> java.net.ConnectException: Connection refused
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at
>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>  at
>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>> at
>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>> at java.net.Socket.connect(Socket.java:546)
>>> at java.net.Socket.connect(Socket.java:495)
>>>  at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>> at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>  at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>> at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>> at java.util.TimerThread.run(Timer.java:484)
>>>
>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>
>>> Thoughts?  Thanks a million,
>>>
>>> Fred
>>>
>>>
>>
>
>
> --
> Fred Wolfinger
> Security Research Engineer
> G2, Inc.
>
> 302 Sentinel Drive, Suite 300
> Annapolis Junction, MD 20701
> Office: 301-575-5142
> Cell: 443-655-3322
>
>

Fred,

One tracer is fine, and you can set that to be the same as the master node.
You also need to set the username and password for the tracer in
accumulo-site.xml if you haven't already.

Adam
On Sep 5, 2012 1:22 PM, "Fred Wolfinger" <[EMAIL PROTECTED]> wrote:

> Hey Marc,
>
> I can't tell you how much I appreciate the quick response. I never touched
> the tracers fie, so it still just says localhost. If I have a master and 3
> slaves, which IP's go into that file? All, or just the slaves?
>
> Thanks!
>
> Fred
>
> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> have you verified that the tracers are running?
>>
>> look at /conf/tracers to make sure you have them configured and that they
>> are running on those servers.
>>
>>
>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <[EMAIL PROTECTED]
>> > wrote:
>>
>>> I am trying to get data into a single table. I am getting very high
>>> error counts from all tablet servers and loggers of which there are 3
>>> each.....all with the same error:
>>>
>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>> java.net.ConnectException: Connection refused
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at
>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>  at
>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>> at
>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>> at java.net.Socket.connect(Socket.java:546)
>>> at java.net.Socket.connect(Socket.java:495)
>>>  at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>> at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>  at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>> at
>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>> at java.util.TimerThread.run(Timer.java:484)
>>>
>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>
>>> Thoughts?  Thanks a million,
>>>
>>> Fred
>>>
>>>
>>
>
>
> --
> Fred Wolfinger
> Security Research Engineer
> G2, Inc.
>
> 302 Sentinel Drive, Suite 300
> Annapolis Junction, MD 20701
> Office: 301-575-5142
> Cell: 443-655-3322
>
>

Interesting...

So, by that advice, I should have never had a problem since the tracers
file on the master node had "localhost" inside. While waiting for your
response, I went ahead and just put the three slaves in the file and
restarted accumulo. Tracers were then started on each of the three slaves.
My ingest just completed with zero errors.

Fred

On Wed, Sep 5, 2012 at 1:31 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:

> Typically I've only used 1 tracer ( and usually ) on my master; however,
> I'm not sure of any suggested guidance, especially in regards to
> scalability.
>
> Be aware of the following if you have a different password for the tracer
>
>
> http://mail-archives.apache.org/mod_mbox/accumulo-user/201202.mbox/%3CCADxc9BmhA4vRqT5Nn6TUAWNSRxtJMA3+K+[EMAIL PROTECTED]%3E
>
>
> On Wed, Sep 5, 2012 at 1:21 PM, Fred Wolfinger <[EMAIL PROTECTED]>wrote:
>
>> Hey Marc,
>>
>> I can't tell you how much I appreciate the quick response. I never
>> touched the tracers fie, so it still just says localhost. If I have a
>> master and 3 slaves, which IP's go into that file? All, or just the slaves?
>>
>> Thanks!
>>
>> Fred
>>
>>
>> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> have you verified that the tracers are running?
>>>
>>> look at /conf/tracers to make sure you have them configured and that
>>> they are running on those servers.
>>>
>>>
>>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> I am trying to get data into a single table. I am getting very high
>>>> error counts from all tablet servers and loggers of which there are 3
>>>> each.....all with the same error:
>>>>
>>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>>> java.net.ConnectException: Connection refused
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at
>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>>  at
>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>> at
>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>> at java.net.Socket.connect(Socket.java:546)
>>>> at java.net.Socket.connect(Socket.java:495)
>>>>  at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>>> at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>>  at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>>> at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>
>>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>>
>>>> Thoughts?  Thanks a million,
>>>>
>>>> Fred
>>>>
>>>>
>>>
>>
>>
>> --
>> Fred Wolfinger
>> Security Research Engineer
>> G2, Inc.
>>
>> 302 Sentinel Drive, Suite 300
>> Annapolis Junction, MD 20701
>> Office: 301-575-5142
>> Cell: 443-655-3322
>>
>>
>
--
Fred Wolfinger
Security Research Engineer
G2, Inc.

302 Sentinel Drive, Suite 300
Annapolis Junction, MD 20701
Office: 301-575-5142
Cell: 443-655-3322

Hey Adam,

Thanks for the response. I will be sure to add those settings. To be clear,
though, will the tracers password be the same as the one I set when I ran
"accumulo init"?

Fred

On Wed, Sep 5, 2012 at 1:37 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:

> Fred,
>
> One tracer is fine, and you can set that to be the same as the master
> node. You also need to set the username and password for the tracer in
> accumulo-site.xml if you haven't already.
>
> Adam
> On Sep 5, 2012 1:22 PM, "Fred Wolfinger" <[EMAIL PROTECTED]>
> wrote:
>
>> Hey Marc,
>>
>> I can't tell you how much I appreciate the quick response. I never
>> touched the tracers fie, so it still just says localhost. If I have a
>> master and 3 slaves, which IP's go into that file? All, or just the slaves?
>>
>> Thanks!
>>
>> Fred
>>
>> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> have you verified that the tracers are running?
>>>
>>> look at /conf/tracers to make sure you have them configured and that
>>> they are running on those servers.
>>>
>>>
>>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> I am trying to get data into a single table. I am getting very high
>>>> error counts from all tablet servers and loggers of which there are 3
>>>> each.....all with the same error:
>>>>
>>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>>> java.net.ConnectException: Connection refused
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at
>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>>  at
>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>> at
>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>> at java.net.Socket.connect(Socket.java:546)
>>>> at java.net.Socket.connect(Socket.java:495)
>>>>  at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>>> at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>>  at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>>> at
>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>
>>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>>
>>>> Thoughts?  Thanks a million,
>>>>
>>>> Fred
>>>>
>>>>
>>>
>>
>>
>> --
>> Fred Wolfinger
>> Security Research Engineer
>> G2, Inc.
>>
>> 302 Sentinel Drive, Suite 300
>> Annapolis Junction, MD 20701
>> Office: 301-575-5142
>> Cell: 443-655-3322
>>
>>
--
Fred Wolfinger
Security Research Engineer
G2, Inc.

302 Sentinel Drive, Suite 300
Annapolis Junction, MD 20701
Office: 301-575-5142
Cell: 443-655-3322

Odd. Check /logs/tracer*

If you're in a distributed setup ( sounds like you are ), you will want to
use the host name of the outward interface ( fqdn ) instead of the loopback
adapter.

Try setting the IP in the tracer files to the master IP and removing the
slave IPs, and it should probably start fine.

On Wed, Sep 5, 2012 at 1:38 PM, Fred Wolfinger <[EMAIL PROTECTED]>wrote:

> Interesting...
>
> So, by that advice, I should have never had a problem since the tracers
> file on the master node had "localhost" inside. While waiting for your
> response, I went ahead and just put the three slaves in the file and
> restarted accumulo. Tracers were then started on each of the three slaves.
> My ingest just completed with zero errors.
>
> Fred
>
>
> On Wed, Sep 5, 2012 at 1:31 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> Typically I've only used 1 tracer ( and usually ) on my master; however,
>> I'm not sure of any suggested guidance, especially in regards to
>> scalability.
>>
>> Be aware of the following if you have a different password for the tracer
>>
>>
>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201202.mbox/%3CCADxc9BmhA4vRqT5Nn6TUAWNSRxtJMA3+K+[EMAIL PROTECTED]%3E
>>
>>
>> On Wed, Sep 5, 2012 at 1:21 PM, Fred Wolfinger <[EMAIL PROTECTED]
>> > wrote:
>>
>>> Hey Marc,
>>>
>>> I can't tell you how much I appreciate the quick response. I never
>>> touched the tracers fie, so it still just says localhost. If I have a
>>> master and 3 slaves, which IP's go into that file? All, or just the slaves?
>>>
>>> Thanks!
>>>
>>> Fred
>>>
>>>
>>> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>>
>>>> have you verified that the tracers are running?
>>>>
>>>> look at /conf/tracers to make sure you have them configured and that
>>>> they are running on those servers.
>>>>
>>>>
>>>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> I am trying to get data into a single table. I am getting very high
>>>>> error counts from all tablet servers and loggers of which there are 3
>>>>> each.....all with the same error:
>>>>>
>>>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>>>> java.net.ConnectException: Connection refused
>>>>> java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>>>  at
>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>  at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>>>> at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>>>  at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>>>> at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>>
>>>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>>>
>>>>> Thoughts?  Thanks a million,
>>>>>
>>>>> Fred
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Fred Wolfinger
>>> Security Research Engineer
>>> G2, Inc.
>>>
>>> 302 Sentinel Drive, Suite 300
>>> Annapolis Junction, MD 20701
>>> Office: 301-575-5142
>>> Cell: 443-655-3322
>>>
>>>
>>
>
>
> --
> Fred Wolfinger
> Security Research Engineer
> G2, Inc.
>
> 302 Sentinel Drive, Suite 300
> Annapolis Junction, MD 20701
> Office: 301-575-5142
> Cell: 443-655-3322
>
>

My guess is that your tracer was listening on 127.0.0.1 and registered that
ip address in zookeeper. Other nodes would have had trouble contacting that
ip address. Switching to the hostname of the master node should fix it too
if that was the problem.

Maybe we should put in a ticket to have services register all of their
addresses when they start up?

Adam
On Sep 5, 2012 1:44 PM, "Fred Wolfinger" <[EMAIL PROTECTED]> wrote:

> Interesting...
>
> So, by that advice, I should have never had a problem since the tracers
> file on the master node had "localhost" inside. While waiting for your
> response, I went ahead and just put the three slaves in the file and
> restarted accumulo. Tracers were then started on each of the three slaves.
> My ingest just completed with zero errors.
>
> Fred
>
> On Wed, Sep 5, 2012 at 1:31 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>
>> Typically I've only used 1 tracer ( and usually ) on my master; however,
>> I'm not sure of any suggested guidance, especially in regards to
>> scalability.
>>
>> Be aware of the following if you have a different password for the tracer
>>
>>
>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201202.mbox/%3CCADxc9BmhA4vRqT5Nn6TUAWNSRxtJMA3+K+[EMAIL PROTECTED]%3E
>>
>>
>> On Wed, Sep 5, 2012 at 1:21 PM, Fred Wolfinger <[EMAIL PROTECTED]
>> > wrote:
>>
>>> Hey Marc,
>>>
>>> I can't tell you how much I appreciate the quick response. I never
>>> touched the tracers fie, so it still just says localhost. If I have a
>>> master and 3 slaves, which IP's go into that file? All, or just the slaves?
>>>
>>> Thanks!
>>>
>>> Fred
>>>
>>>
>>> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>>
>>>> have you verified that the tracers are running?
>>>>
>>>> look at /conf/tracers to make sure you have them configured and that
>>>> they are running on those servers.
>>>>
>>>>
>>>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> I am trying to get data into a single table. I am getting very high
>>>>> error counts from all tablet servers and loggers of which there are 3
>>>>> each.....all with the same error:
>>>>>
>>>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>>>> java.net.ConnectException: Connection refused
>>>>> java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>>>  at
>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>  at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>>>> at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>>>  at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>>>> at
>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>>
>>>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>>>
>>>>> Thoughts?  Thanks a million,
>>>>>
>>>>> Fred
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Fred Wolfinger
>>> Security Research Engineer
>>> G2, Inc.
>>>
>>> 302 Sentinel Drive, Suite 300
>>> Annapolis Junction, MD 20701
>>> Office: 301-575-5142
>>> Cell: 443-655-3322
>>>
>>>
>>
>
>
> --
> Fred Wolfinger
> Security Research Engineer
> G2, Inc.
>
> 302 Sentinel Drive, Suite 300
> Annapolis Junction, MD 20701
> Office: 301-575-5142

I must be clairvoyant....that's exactly what my next move was - used the
raw IP of the master and deleted the slaves. Everything is operating
perfectly so far.

Thanks so much, Marc!

Fred

On Wed, Sep 5, 2012 at 1:50 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:

> Odd. Check /logs/tracer*
>
> If you're in a distributed setup ( sounds like you are ), you will want to
> use the host name of the outward interface ( fqdn ) instead of the loopback
> adapter.
>
> Try setting the IP in the tracer files to the master IP and removing the
> slave IPs, and it should probably start fine.
>
>
>
> On Wed, Sep 5, 2012 at 1:38 PM, Fred Wolfinger <[EMAIL PROTECTED]>wrote:
>
>> Interesting...
>>
>> So, by that advice, I should have never had a problem since the tracers
>> file on the master node had "localhost" inside. While waiting for your
>> response, I went ahead and just put the three slaves in the file and
>> restarted accumulo. Tracers were then started on each of the three slaves.
>> My ingest just completed with zero errors.
>>
>> Fred
>>
>>
>> On Wed, Sep 5, 2012 at 1:31 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> Typically I've only used 1 tracer ( and usually ) on my master; however,
>>> I'm not sure of any suggested guidance, especially in regards to
>>> scalability.
>>>
>>> Be aware of the following if you have a different password for the tracer
>>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201202.mbox/%3CCADxc9BmhA4vRqT5Nn6TUAWNSRxtJMA3+K+[EMAIL PROTECTED]%3E
>>>
>>>
>>> On Wed, Sep 5, 2012 at 1:21 PM, Fred Wolfinger <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hey Marc,
>>>>
>>>> I can't tell you how much I appreciate the quick response. I never
>>>> touched the tracers fie, so it still just says localhost. If I have a
>>>> master and 3 slaves, which IP's go into that file? All, or just the slaves?
>>>>
>>>> Thanks!
>>>>
>>>> Fred
>>>>
>>>>
>>>> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> have you verified that the tracers are running?
>>>>>
>>>>> look at /conf/tracers to make sure you have them configured and that
>>>>> they are running on those servers.
>>>>>
>>>>>
>>>>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> I am trying to get data into a single table. I am getting very high
>>>>>> error counts from all tablet servers and loggers of which there are 3
>>>>>> each.....all with the same error:
>>>>>>
>>>>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>>>>> java.net.ConnectException: Connection refused
>>>>>> java.net.ConnectException: Connection refused
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at
>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>>>>  at
>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>>>> at
>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>>  at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>>>>> at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>>>>  at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>>>>> at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>>>
>>>>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>>>>
>>>>>> Thoughts?  Thanks a million,
>>>>>>
>>>>>> Fred
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Fred Wolfinger
Fred Wolfinger
Security Research Engineer
G2, Inc.

302 Sentinel Drive, Suite 300
Annapolis Junction, MD 20701
Office: 301-575-5142
Cell: 443-655-3322

I was just saying that to Marc....I used the raw IP of the master and
everything seems to be working perfectly. I think that ticket would be a
great addition, for what it's worth.

Thanks so much for the help!

Fred

On Wed, Sep 5, 2012 at 1:52 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:

> My guess is that your tracer was listening on 127.0.0.1 and registered
> that ip address in zookeeper. Other nodes would have had trouble contacting
> that ip address. Switching to the hostname of the master node should fix it
> too if that was the problem.
>
> Maybe we should put in a ticket to have services register all of their
> addresses when they start up?
>
> Adam
> On Sep 5, 2012 1:44 PM, "Fred Wolfinger" <[EMAIL PROTECTED]>
> wrote:
>
>> Interesting...
>>
>> So, by that advice, I should have never had a problem since the tracers
>> file on the master node had "localhost" inside. While waiting for your
>> response, I went ahead and just put the three slaves in the file and
>> restarted accumulo. Tracers were then started on each of the three slaves.
>> My ingest just completed with zero errors.
>>
>> Fred
>>
>> On Wed, Sep 5, 2012 at 1:31 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>
>>> Typically I've only used 1 tracer ( and usually ) on my master; however,
>>> I'm not sure of any suggested guidance, especially in regards to
>>> scalability.
>>>
>>> Be aware of the following if you have a different password for the tracer
>>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201202.mbox/%3CCADxc9BmhA4vRqT5Nn6TUAWNSRxtJMA3+K+[EMAIL PROTECTED]%3E
>>>
>>>
>>> On Wed, Sep 5, 2012 at 1:21 PM, Fred Wolfinger <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hey Marc,
>>>>
>>>> I can't tell you how much I appreciate the quick response. I never
>>>> touched the tracers fie, so it still just says localhost. If I have a
>>>> master and 3 slaves, which IP's go into that file? All, or just the slaves?
>>>>
>>>> Thanks!
>>>>
>>>> Fred
>>>>
>>>>
>>>> On Wed, Sep 5, 2012 at 1:15 PM, Marc Parisi <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> have you verified that the tracers are running?
>>>>>
>>>>> look at /conf/tracers to make sure you have them configured and that
>>>>> they are running on those servers.
>>>>>
>>>>>
>>>>> On Wed, Sep 5, 2012 at 1:06 PM, Fred Wolfinger <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> I am trying to get data into a single table. I am getting very high
>>>>>> error counts from all tablet servers and loggers of which there are 3
>>>>>> each.....all with the same error:
>>>>>>
>>>>>> 05 12:58:02,224 [receivers.SendSpansViaThrift] ERROR:
>>>>>> java.net.ConnectException: Connection refused
>>>>>> java.net.ConnectException: Connection refused
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at
>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>>>>  at
>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>>>> at
>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>>>>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>>  at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:53)
>>>>>> at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.SendSpansViaThrift.createDestination(SendSpansViaThrift.java:34)
>>>>>>  at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver.sendSpans(AsyncSpanReceiver.java:87)
>>>>>> at
>>>>>> org.apache.accumulo.cloudtrace.instrument.receivers.AsyncSpanReceiver$1.run(AsyncSpanReceiver.java:63)
>>>>>>  at java.util.TimerThread.mainLoop(Timer.java:534)
>>>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>>>
>>>>>> I can ping/ssh/etc. freely across all slaves and master nodes.
>>>>>>
>>>>>> Thoughts?  Thanks a million,
>

Fred Wolfinger
Security Research Engineer
G2, Inc.

302 Sentinel Drive, Suite 300
Annapolis Junction, MD 20701
Office: 301-575-5142
Cell: 443-655-3322