Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Bigtop >> mail # dev >> [DISCUSS] Bigtop 0.8.0 BOM proposal


+
Roman Shaposhnik 2014-01-21, 13:22
+
Andrew Purtell 2014-01-21, 18:27
+
Mark Grover 2014-01-22, 01:52
+
Giridharan Kesavan 2014-01-22, 01:57
+
Andrew Purtell 2014-01-22, 18:03
+
Giridharan Kesavan 2014-01-22, 19:10
+
Andrew Purtell 2014-01-22, 21:15
+
Konstantin Boudnik 2014-01-22, 22:34
Copy link to this message
-
Re: [DISCUSS] Bigtop 0.8.0 BOM proposal
The work I mentioned earlier was related to Hive and Pig switching
to use Tez natively as compared to using Tez via the MR compat
layer built into Tez. I don't have an answer on when hive or pig would
get released with native Tez support, but I see hive getting tez support
in the next release.
Pig Tez integration work is still in progress and it might
get fixed in the next pig release. And there is also
a possibility this may not happen in the 0.8 timeline.

Tez is an independent project which already supports mapreduce
via a compat. layer. AFAIK, it is not targeted just for pig or hive.
Furthermore, users of MR, hive and pig can already make use of
Tez by switching MR to run in tez-execution mode.

To answer Cos's question on testing efforts, we could run simple
mapreduce like jobs (wordcount/ordered wordcount) to validate tez working.

When hive and pig is ready we can enhance hive and pig integration test
suites
to make use of Tez api's and certify those application's.

P.S: FWIW, the api is called doJobClientMagic() for the Tez-MR compat.
layer. :)

-Giri
On Wed, Jan 22, 2014 at 2:34 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> I was about asking the same question essentially. As of right now, there's
> no
> downstream components are using Tez. Hence, sure: we'll get it into the
> stack,
> but there's nothing to use it with not test it on. So, how we can make sure
> that the code doesn't bit-rot, etc.?
>
> And it isn't to give Giri any hard time, but just because there's no one
> else
> (as far as I know) in this community with any knowledge about that project.
>
> Cos
>
> P.S. I guess I wasn't totally accurate in my last statement. I know that
> Tez
> API has a method doClientMagic(). But that's all I know. Swear!
>
> On Wed, Jan 22, 2014 at 01:15PM, Andrew Purtell wrote:
> > My question is really asking for a guesstimate on when Tez, Hive, and Pig
> > all converge to a released state. I guess for Hive that would be release
> > 0.13. For Pig? Pardon me if I am mistaken but I thought Tez targets those
> > like Hive and Pig developers, not end users of MapReduce, Hive, or Pig.
> Can
> > we assume Hive and Pig when released with support for Tez will depend on
> > the same version of Tez? All of this is unlikely to happen within the
> > timeframe of Bigtop 0.8? Should we be talking about this for 0.9?
> >
> >
> > On Wed, Jan 22, 2014 at 11:10 AM, Giridharan Kesavan <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Andrew,
> > >
> > > Hive on tez is fixed in the hive trunk and should be part of the next
> > > release, 0.13
> > > Here is the link to the jira.
> > > https://issues.apache.org/jira/browse/HIVE-4660
> > >
> > > Pig on tez integration is tracked here
> > > https://issues.apache.org/jira/browse/PIG-3446
> > >
> > >
> > > -Giri
> > >
> > >
> > > On Wed, Jan 22, 2014 at 10:03 AM, Andrew Purtell <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > At some point other components like Hive and Pig will want to pull in
> > > Tez,
> > > > but as far as I know no released versions use it yet. Can you shed
> some
> > > > insight on when that might be Giri?
> > > >
> > > >
> > > > On Tue, Jan 21, 2014 at 5:57 PM, Giridharan Kesavan <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > > Can we include Tez-0.2.0  to the 0.8 BOM?
> > > > >
> > > > > I have working patch uploaded to BIGTOP-1179
> > > > >
> > > > > -Giri
> > > > >
> > > > >
> > > > > On Tue, Jan 21, 2014 at 5:52 PM, Mark Grover <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > > > Thanks for starting this email thread, Roman! I posted my
> opinion on
> > > > the
> > > > > > JIRA.
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 21, 2014 at 10:27 AM, Andrew Purtell <
> > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > Welcome back Roman.
> > > > > > >
> > > > > > > Likewise, I've been tied up with the day job but next month I
> can
> > > > > > > do BIGTOP-1105/BIGTOP-1148, and BIGTOP-1029 for HBase 0.98.
> > > > > > >
> > > > > > >
> > > > > >
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
+
Konstantin Boudnik 2014-01-23, 01:07
+
Konstantin Boudnik 2014-05-13, 01:16