Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill >> mail # dev >> Setting up a cluster


+
Alexandre BECHE 2013-07-10, 13:22
+
Jacques Nadeau 2013-07-10, 21:21
+
Ted Dunning 2013-07-11, 00:36
+
Alexandre BECHE 2013-07-16, 21:26
+
Michael Hausenblas 2013-07-16, 21:33
+
Alexandre BECHE 2013-07-17, 07:47
Copy link to this message
-
Re: Setting up a cluster
Hi everybody,

I started to dump a few data into my cluster and play a bit with the
reference interpreter.
Today my dump is just a json file stored on my local FS, could I put it in
HDFS? Is the HDFS Scanner already available?

I understood that the reference interpreter is used for testing purpose
only, what would be the next step for me to get a full drill query on my
dataset?

Thanks for your help,
Cheers,
Alex
On Wed, Jul 17, 2013 at 9:47 AM, Alexandre BECHE
<[EMAIL PROTECTED]>wrote:

> Thanks for your answer.
>
> But in the meantime, does it make sense to start deploying drillbit and
> get the first steps working?
> Which deployment model would you recommend (single machine or cluster
> mode).
>
> Cheers,
> Alex
>
>
> On Tue, Jul 16, 2013 at 11:33 PM, Michael Hausenblas <
> [EMAIL PROTECTED]> wrote:
>
>>
>> > As far as I understood, this webUI
>> http://srvgal85.deri.ie/apache-drill/ is
>> > currently working with elasticsearch (does it mean it will be supported
>> in
>> > the future?).
>>
>>
>> Note that the above URL was just a demo deployment. As soon as DRILL-77
>> [1], the REST API, is resolved (or close to) I'll port the UI over to it.
>> ES was just necessary back then (Oct last year IIRC) as no backend existed.
>>
>> You can track the progress re the WebUI by subscribing to DRILL-58 [2]. I
>> would think that Hari soon has a breakthrough with the REST API and this
>> means I should get around to this also rather soonish.
>>
>> Hari, any thoughts re timing?
>>
>>
>> Cheers,
>>                 Michael
>>
>> [1] https://issues.apache.org/jira/browse/DRILL-77
>> [2] https://issues.apache.org/jira/browse/DRILL-58
>>
>> --
>> Michael Hausenblas
>> Ireland, Europe
>> http://mhausenblas.info/
>>
>> On 16 Jul 2013, at 22:26, Alexandre BECHE <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi everybody,
>> >
>> > I finally got the cluster ready and almost working properly. Now, I
>> hope to
>> > start the data acquisition by the end of the week. As I understood that
>> the
>> > JSON scanner is much more advanced than the HBase one, I will start
>> using
>> > it.
>> >
>> > Now I have a few more questions, maybe dedicated to Michael.
>> > As far as I understood, this webUI
>> http://srvgal85.deri.ie/apache-drill/ is
>> > currently working with elasticsearch (does it mean it will be supported
>> in
>> > the future?). My main question is: how far am I from using it with my
>> own
>> > json data stored in HDFS (ie. using the full stack) and what would be
>> the
>> > first step?
>> >
>> > Thanks for your help,
>> > Cheers,
>> > Alex
>> >
>> >
>> >
>> > On Thu, Jul 11, 2013 at 2:36 AM, Ted Dunning <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Even that dependency change is often not necessary.  If your original
>> >> dependency looks like this:
>> >>
>> >>    <dependency>
>> >>      <groupId>org.apache.hadoop</groupId>
>> >>      <artifactId>hadoop-core</artifactId>
>> >>      <version>1.1.0</version>
>> >>      <scope>provided</scope>
>> >>    </dependency>
>> >>
>> >> then your code should work with either Apache Hadoop or MapR.  All you
>> have
>> >> to do is make sure the jars for the distro you have are in the
>> classpath
>> >> correctly.  This does not package the Hadoop jars into your executable
>> >> which can either be a virtue or a vice depending on your requirements.
>> >>
>> >>
>> >>
>> >> On Wed, Jul 10, 2013 at 2:21 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>> >> wrote:
>> >>
>> >>> Ted gave a good long answer.  The short answer is that you'll need to
>> >>> change the dependency on Hadoop to use the MapR distribution which
>> >>> means changing:
>> >>>
>> >>> the following entry in two files files:
>> >>> sandbox/prototype/exec/java-exec/pom.xml and
>> >>> sandbox/prototype/exec/ref/pom.xml
>> >>>
>> >>>      <groupId>org.apache.hadoop</groupId>
>> >>>      <artifactId>hadoop-core</artifactId>
>> >>>      <version>1.1.0</version>
>> >>>
>> >>> to: (note the change in version number)
>> >>>
>> >>>  <groupId>org.apache.hadoop</groupId>
+
Ted Dunning 2013-07-10, 21:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB