Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # user >> Distributed mode troubles: ZK/Curator connection time out


Copy link to this message
-
Re: Distributed mode troubles: ZK/Curator connection time out
As for the submit_plan tool, unfortunately at the time of the M1 release,
the tool was not fully baked. So it does not have very good parsing
capabilities. But as you wrote in your diagram, it takes three parameters,
a file, a type, and a zk connect string. The file is a path on the local
filesystem to a file that contains either a logical or a physical plan, in
json format. The type should be either physical or logical, and this
corresponds to the type of plan that the file parameter points to. And the
zk connect string is the zk connect string for the cluster you want to
submit to.

for example:

submit_plan /plans/plan1.json physical 1.1.1.1:5181,2.2.2.2:5181 will use
the zk quorum of servers running on 1.1.1.1 and 2.2.2.2 to find and connect
to a drillbit, and then submit the physical plan contained in
/plans/plan1.json to the drillbit. It will then print results to the screen
as they come in.

If you build drill off of master, the submit_plan tool has been fixed
somewhat:

$ bin/submit_plan -h

Usage: ./submit_plan [options]
  Options:
    -h, -help, --help
       show usage
       Default: false
    -bits
       number of drillbits to run. local mode only
       Default: 1
  * -f
       file containing plan
    -local
       run query in local mode
       Default: false
  * -t
       type of plan, logical/physical
    -zk
       zookeeper connect string.
       Default: localhost:2181

On Sun, Oct 27, 2013 at 3:17 PM, Steven Phillips <[EMAIL PROTECTED]>wrote:

> You need to replace localhost with the hostname of the node running
> zookeeper. If that zookeeper is configured to use a port different than
> 2181, then that needs to be set as well. If you have multiple zookeepers in
> the quorum, you then zk.connect should be a comma separated list of the
> host:port of each node.
>
> The default, localhost setting will only work when a drillbit is running
> on the same node as the zookeeper.
>
>
> On Sun, Oct 27, 2013 at 2:57 PM, Michael Hausenblas <
> [EMAIL PROTECTED]> wrote:
>
>>
>> > One thing to add to the diagram is that all of the drill java processes
>> will look at what is in drill-override.conf.
>>
>> Thanks, done.
>>
>>
>> > You must set zk.connect to the correct zk host:port.
>>
>>
>> Can you be a tad more explicit, please? In drill-override.conf I have
>>
>> [[
>> …
>> zk: {
>>         connect: "localhost:2181”,
>> …
>> ]]
>>
>>
>> What am I overlooking?
>>
>> Also, any directions re the rest of my questions (re bin/submit_plan
>> etc.)?
>>
>>
>> With a little help from here,  I’m happy to put together the description
>> how to set this up in the Wiki, also to address a query we’ve now lying
>> around for more than three weeks, by Steve McPherson – see
>> http://mail-archives.apache.org/mod_mbox/incubator-drill-user/201310.mbox/%3CCE71A20F.14F5B%25stevemp%40amazon.com%3E– the fact that it attracted 0 responses I find slightly embarrassing, and
>> if I were Steve, I’d prolly not touch Drill anymore, but let’s hope for the
>> best …
>>
>>
>> Cheers,
>>                 Michael
>>
>> --
>> Michael Hausenblas
>> Ireland, Europe
>> http://mhausenblas.info/
>>
>> On 27 Oct 2013, at 21:32, Steven Phillips <[EMAIL PROTECTED]> wrote:
>>
>> > One thing to add to the diagram is that all of the drill java processes
>> > will look at what is in drill-override.conf. You must set zk.connect to
>> the
>> > correct zk host:port.
>> >
>> >
>> > On Sun, Oct 27, 2013 at 2:00 PM, Michael Hausenblas <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >>
>> >> Folks,
>> >>
>> >> I’m trying to set up Drill in distributed mode. Here’s what I have so
>> far:
>> >> when I launch the first Drillbit with bin/drillbit.sh I get the
>> following
>> >> in log/drillbit.out:
>> >>
>> >> [[
>> >> 20:47:20.963 [main] ERROR com.netflix.curator.ConnectionState -
>> Connection
>> >> timed out for connection string (localhost:2181) and timeout (5000) /
>> >> elapsed (5045)
>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB