Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill >> mail # dev >> Setting up a cluster


+
Alexandre BECHE 2013-07-10, 13:22
+
Jacques Nadeau 2013-07-10, 21:21
+
Ted Dunning 2013-07-11, 00:36
+
Alexandre BECHE 2013-07-16, 21:26
+
Michael Hausenblas 2013-07-16, 21:33
+
Alexandre BECHE 2013-07-17, 07:47
Copy link to this message
-
Re: Setting up a cluster
Hi everybody,

I started to dump a few data into my cluster and play a bit with the
reference interpreter.
Today my dump is just a json file stored on my local FS, could I put it in
HDFS? Is the HDFS Scanner already available?

I understood that the reference interpreter is used for testing purpose
only, what would be the next step for me to get a full drill query on my
dataset?

Thanks for your help,
Cheers,
Alex
On Wed, Jul 17, 2013 at 9:47 AM, Alexandre BECHE
<[EMAIL PROTECTED]>wrote:

> Thanks for your answer.
>
> But in the meantime, does it make sense to start deploying drillbit and
> get the first steps working?
> Which deployment model would you recommend (single machine or cluster
> mode).
>
> Cheers,
> Alex
>
>
> On Tue, Jul 16, 2013 at 11:33 PM, Michael Hausenblas <
> [EMAIL PROTECTED]> wrote:
>
>>
>> > As far as I understood, this webUI
>> http://srvgal85.deri.ie/apache-drill/ is
>> > currently working with elasticsearch (does it mean it will be supported
>> in
>> > the future?).
>>
>>
>> Note that the above URL was just a demo deployment. As soon as DRILL-77
>> [1], the REST API, is resolved (or close to) I'll port the UI over to it.
>> ES was just necessary back then (Oct last year IIRC) as no backend existed.
>>
>> You can track the progress re the WebUI by subscribing to DRILL-58 [2]. I
>> would think that Hari soon has a breakthrough with the REST API and this
>> means I should get around to this also rather soonish.
>>
>> Hari, any thoughts re timing?
>>
>>
>> Cheers,
>>                 Michael
>>
>> [1] https://issues.apache.org/jira/browse/DRILL-77
>> [2] https://issues.apache.org/jira/browse/DRILL-58
>>
>> --
>> Michael Hausenblas
>> Ireland, Europe
>> http://mhausenblas.info/
>>
>> On 16 Jul 2013, at 22:26, Alexandre BECHE <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi everybody,
>> >
>> > I finally got the cluster ready and almost working properly. Now, I
>> hope to
>> > start the data acquisition by the end of the week. As I understood that
>> the
>> > JSON scanner is much more advanced than the HBase one, I will start
>> using
>> > it.
>> >
>> > Now I have a few more questions, maybe dedicated to Michael.
>> > As far as I understood, this webUI
>> http://srvgal85.deri.ie/apache-drill/ is
>> > currently working with elasticsearch (does it mean it will be supported
>> in
>> > the future?). My main question is: how far am I from using it with my
>> own
>> > json data stored in HDFS (ie. using the full stack) and what would be
>> the
>> > first step?
>> >
>> > Thanks for your help,
>> > Cheers,
>> > Alex
>> >
>> >
>> >
>> > On Thu, Jul 11, 2013 at 2:36 AM, Ted Dunning <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Even that dependency change is often not necessary.  If your original
>> >> dependency looks like this:
>> >>
>> >>    <dependency>
>> >>      <groupId>org.apache.hadoop</groupId>
>> >>      <artifactId>hadoop-core</artifactId>
>> >>      <version>1.1.0</version>
>> >>      <scope>provided</scope>
>> >>    </dependency>
>> >>
>> >> then your code should work with either Apache Hadoop or MapR.  All you
>> have
>> >> to do is make sure the jars for the distro you have are in the
>> classpath
>> >> correctly.  This does not package the Hadoop jars into your executable
>> >> which can either be a virtue or a vice depending on your requirements.
>> >>
>> >>
>> >>
>> >> On Wed, Jul 10, 2013 at 2:21 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>> >> wrote:
>> >>
>> >>> Ted gave a good long answer.  The short answer is that you'll need to
>> >>> change the dependency on Hadoop to use the MapR distribution which
>> >>> means changing:
>> >>>
>> >>> the following entry in two files files:
>> >>> sandbox/prototype/exec/java-exec/pom.xml and
>> >>> sandbox/prototype/exec/ref/pom.xml
>> >>>
>> >>>      <groupId>org.apache.hadoop</groupId>
>> >>>      <artifactId>hadoop-core</artifactId>
>> >>>      <version>1.1.0</version>
>> >>>
>> >>> to: (note the change in version number)
>> >>>
>> >>>  <groupId>org.apache.hadoop</groupId>
+
Ted Dunning 2013-07-10, 21:02