Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> [VOTE] Release hadoop-2.0.0-alpha


Copy link to this message
-
Re: [VOTE] Release hadoop-2.0.0-alpha
Hi Andrew,

Have you seen the new MiniMRClientCluster class? It's meant to be what
you describe - a minicluster which only exposes "external" APIs --
most importantly a way of getting at a JobClient to submit jobs. We
have it implemented in both 1.x and 2.x at this point, though I don't
recall if it's in the 1.0.x releases or if it's only slated for 1.1+

-Todd

On Wed, May 9, 2012 at 6:05 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> Hi Suresh,
>
> The unstable designation makes sense.  As would one for MiniMRCluster.
>
> I was over the top initially to surprise. I'm sure the MR minicluster seems a minor detail.
>
> Maybe it's worth thinking about the miniclusters differently? Please pardon if I am rehashing an old discussion.
>
> Things like MRUnit for applications and BigTop for full cluster tests can help, but for as mentioned in the below annotation Pig, Hive, HBase, and other parts of the stack use miniclusters for local end to end testing in unit tests. As the complexity of the stack increases and we consider cross version support, unit tests on miniclusters I think will have no substitute.
>
> As Hadoop 2 has been evolving there has been some difficulty keeping up with minicluster changes. This makes sense. The attention to stability to client APIs and such, and the lack thereof to the minicluster, I think is self evident. But the need to fix up tests unpredictably introduces some friction that perhaps need not be there.
>
> Would a JIRA to discuss defining a subset of the minicluster interfaces as more stable be worthwhile?
>
> Best regards,
>
>    - Andy
>
>
> On May 9, 2012, at 1:45 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
>
>> For this reason, in HDFS, we change MiniDFSCluster to LimitedPrivate and
>> not treat it as such:
>>
>> @InterfaceAudience.LimitedPrivate({"HBase", "HDFS", "Hive", "MapReduce",
>> "Pig"})
>> @InterfaceStability.Unstable
>> public class MiniDFSCluster { ...}
>>
>> On Wed, May 9, 2012 at 11:33 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>>
>>> Sounds good Arun.
>>>
>>> How should we consider the suitability and stability of MiniMRCluster
>>> for downstream projects?
>>>
>>> On Wed, May 9, 2012 at 11:30 AM, Arun C Murthy <[EMAIL PROTECTED]>
>>> wrote:
>>>> No worries Andy. I can spin an rc1 once we can pin-point the bug.
>>>>
>>>> thanks,
>>>> Arun
>>>>
>>>> On May 9, 2012, at 10:17 AM, Andrew Purtell wrote:
>>>>
>>>>> -1 (nonbinding), we are currently facing a minicluster semantic change
>>>>> of some kind, or more than one:
>>>>>
>>>>>   https://issues.apache.org/jira/browse/HBASE-5966
>>>>>
>>>>> There are other HBase JIRAs related to 2.0.0-alpha that we are working
>>>>> on, but I'd claim those are all our fault for breaking abstractions to
>>>>> solve issues. In one case there's a new helpful 2.x API
>>>>> (ShutdownHookManager, thank you!) that we can eventually move to.
>>>>>
>>>>> However, the minicluster changes are causing us some repeated
>>>>> discomfort. It will break, we'll get some help fixing up our tests for
>>>>> that, then some time later it will break again, repeat. Perhaps we
>>>>> have no right to complain, the minicluster isn't meant to be used by
>>>>> downstream projects. If so then please disregard the complaint, but
>>>>> your assistance in helping to fix the breakage again would be much
>>>>> appreciated. And, if so, perhaps we can discuss what makes sense in
>>>>> terms of a stable minicluster consumable for downstream projects?
>>>>>
>>>>> Best regards,
>>>>>
>>>>>   - Andy
>>>>>
>>>>> On Wed, May 9, 2012 at 9:58 AM, Arun C Murthy <[EMAIL PROTECTED]>
>>> wrote:
>>>>>> I've created a release candidate for hadoop-2.0.0-alpha that I would
>>> like to release.
>>>>>>
>>>>>> It is available at:
>>> http://people.apache.org/~acmurthy/hadoop-2.0.0-alpha-rc0/
>>>>>>
>>>>>> The maven artifacts are available via repository.apache.org.
>>>>>>
>>>>>> Please try the release and vote; the vote will run for the usual 7
>>> days.
>>>>>>
>>>>>> This is a big milestone for the Apache Hadoop community -

Todd Lipcon
Software Engineer, Cloudera