Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Bigtop, mail # user - New jiras (thanks) :)


+
Jay Vyas 2013-09-25, 13:47
Copy link to this message
-
Re: New jiras (thanks) :)
Roman Shaposhnik 2013-10-05, 00:20
Hi Jay,

thanks for the kind words. Here's one thing that I wanted to comment on:

On Wed, Sep 25, 2013 at 6:47 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Thanks for all these new jiras focused on end users roman / cons !!(documentation, simplifying hive, jenkins)...
>
> The hive one is sooo true: it's annoying that I need a hive server running to do smoke tests.
> Would like to just smoke test hive ETL without the extra stuff for simpler clusters.
>
> I think in general there should be different types of smokes so that people don't require a whole
> installed infrastructure to run smokes.  Although the vendor distros expect all services running,
> individual users or small startups don't necessarily run all the ecosystem services ,
> they just run the bare Apis, and smoke testing those is still of value on a cluster.

I think this boils down to a few fundamental improvements that we'd have to have
in our test framework to make it flexible:
    * let tests declare (via annotations or somehow else) what
environment they expect
    * have more topologies defined via puppet or similar provisioning frameworks
    * embrace containers Docker.io or similar to not require a full fledged VMs

The above 3 would get us a lot of mileage.

> We do that instead of running the complex hive ETL. For pig we do something's similar ,
> since we want to test basic pig API and not only the new .11 Apis, which are the only one
> supported currently in bigtop .
>
> Anyways.. Thanks for keeping an eye out in this! Jiras like this will make bigtop more
> accessible to the masses !:)

I think its all about more and more folks feeling compelled to contribute small
incremental improvements. Lets hope this will continue!

Thanks,
Roman.