Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Hadoop 1.0 and WebHDFS


Copy link to this message
-
Re: Hadoop 1.0 and WebHDFS
I would differ on the terminology.  The 0.20/1.0 code line is STABLE.  0.21
is not, and probably never will be.  0.22 is said to be approaching
stability, but has not been adjudged to be so yet.  1.0 is indeed derived
from the 0.20 (security+append) branches, so it doesn't have some features
(stable and unstable) that are in 0.21.  It does, however, have a lot of
stable features originally in 0.21 and even 0.23, back-ported to the 0.20
code line.

At any rate, the good news for you is, MultipleInputs and MultipleOutputs
are indeed included in the MAPREDUCE-3607 patch, which is part of 1.0.1,
due out for vote tomorrow.

Cheers,
--Matt

On Thu, Feb 2, 2012 at 3:06 PM, Geoffry Roberts
<[EMAIL PROTECTED]>wrote:

> Yeesh! This is not good.  The whole reason I was using 0.21.0 was
> MultipleOutputs.  It disappeared for a while then came back in 21.
>
> Can you comfort me with any hope that this 1.0.1 will be soon? I gotta
> have MultipleOutputs and the WebHDFS, if it works as billed, will be most
> helpful.
>
>
> On 2 February 2012 13:40, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Yeah I was afraid the API availability would be your next question.
>> New API MultipleOutputs should be in the 1.0.1 micro update but there
>> could still be lots of things missing.
>>
>> On Fri, Feb 3, 2012 at 3:00 AM, Geoffry Roberts
>> <[EMAIL PROTECTED]> wrote:
>> > A down grade! I wouldn't have guessed. Thanks
>> >
>> > Do you know if anything happened to the class MultipleOutputs?
>> >
>> > I just tried running some of my old MR code against 1.0 and it seems
>> > MultipleOutputs cannot be found in the new hadoop-core-1.0.0.jar.
>> >
>> >
>> > On 2 February 2012 10:45, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Note that 0.21 to 1.0 is "sort-of" a downgrade in some ways,
>> >> considering 1.0 is a rename of the 0.20-series. You probably want to
>> >> review a lot of config params since those may not be present in 1.0.
>> >>
>> >> On Thu, Feb 2, 2012 at 11:47 PM, Geoffry Roberts
>> >> <[EMAIL PROTECTED]> wrote:
>> >> > All,
>> >> >
>> >> > I seem to have solved my problem.
>> >> >
>> >> > In my hdfs.site.xml I had the following:
>> >> >
>> >> > <property>
>> >> >   <name>dfs.name.dir</name>
>> >> >   <value>file:///hdfs/name</
>> >> >>
>> >> >> value>
>> >> >> </property>
>> >> >> <property>
>> >> >>   <name>dfs.data.dir</name>
>> >> >>   <value>file:///hdfs/data</value>
>> >> >> </property>
>> >> >
>> >> > The above worked on version 0.21.0, apparently not in 1.0.
>> >> >
>> >> > I changed them to
>> >> > /hdfs/name and /hdfs/data respecively and, well, at least my name
>> node
>> >> > is
>> >> > running.
>> >> >
>> >> >
>> >> > On 2 February 2012 09:48, Geoffry Roberts <[EMAIL PROTECTED]
>> >
>> >> > wrote:
>> >> >>
>> >> >> Thanks for the quick response.
>> >> >>
>> >> >> Here's a snippet from my hdfs.site.xml file.
>> >> >>
>> >> >>     <name>dfs.http.address</name>
>> >> >>     <value>qq000:50070</value>
>> >> >>
>> >> >> qq000 is my name node. Is this correct?
>> >> >>
>> >> >> I have also noticed that my name node is crashing.  It says my hdfs
>> is
>> >> >> in
>> >> >> a inconsistent state. I guess I'll have to (shudder) rebuild it.
>> >> >>
>> >> >> The complete contents of hdfs.site.xml is below.
>> >> >>
>> >> >> <configuration>
>> >> >> <property>
>> >> >>   <name>dfs.replication</name>
>> >> >>   <value>3</value>
>> >> >>   <description>Default block replication.
>> >> >>   The actual number of replications can be specified when the file
>> is
>> >> >> created.
>> >> >>   The default is used if replication is not specified in create
>> time.
>> >> >>   </description>
>> >> >> </property>
>> >> >> <property>
>> >> >>   <name>dfs.name.dir</name>
>> >> >>   <value>file:///hdfs/name</value>
>> >> >> </property>
>> >> >> <property>
>> >> >>   <name>dfs.data.dir</name>
>> >> >>   <value>file:///hdfs/data</value>
>> >> >> </property>
>> >> >> <property>
>> >> >>   <name>dfs.hosts</name>
>> >> >>   <value>includes</value>