madhu phatak
2012-03-20, 05:52
Owen O'Malley
2012-03-20, 05:58
madhu phatak
2012-03-20, 06:05
Owen O'Malley
2012-03-20, 06:20
madhu phatak
2012-03-20, 07:42
Milind.Bhandarkar@...
2012-03-21, 22:14
Jay Vyas
2012-03-23, 06:38
Jagat
2012-03-23, 06:42
-
Very strange Java Collection behavior in Hadoopmadhu phatak 2012-03-20, 05:52
Hi All,
I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java
Collection's . I have following code in reducer
public void reduce(Text text, Iterator<Text> values,
OutputCollector<Text, Text> collector, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
List<Text> list = new ArrayList<Text>();
while(values.hasNext())
{
Text value = values.next();
list.add(value);
System.out.println(value.toString()); }
for(Text value : list)
{
System.out.println(value.toString());
}
}
The first sysout prints following
4 5 6
1 2 3
But when I print from the List, it prints following
1 2 3
1 2 3
*
*The all List values are getting replaced by last added value.
I am not able to understand this behavior. Did anyone seen this behavior ?
Regards,
Madhukara Phatak
--
https://github.com/zinnia-phatak-dev/Nectar
I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java
Collection's . I have following code in reducer
public void reduce(Text text, Iterator<Text> values,
OutputCollector<Text, Text> collector, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
List<Text> list = new ArrayList<Text>();
while(values.hasNext())
{
Text value = values.next();
list.add(value);
System.out.println(value.toString()); }
for(Text value : list)
{
System.out.println(value.toString());
}
}
The first sysout prints following
4 5 6
1 2 3
But when I print from the List, it prints following
1 2 3
1 2 3
*
*The all List values are getting replaced by last added value.
I am not able to understand this behavior. Did anyone seen this behavior ?
Regards,
Madhukara Phatak
--
https://github.com/zinnia-phatak-dev/Nectar
-
Re: Very strange Java Collection behavior in HadoopOwen O'Malley 2012-03-20, 05:58
On Mon, Mar 19, 2012 at 10:52 PM, madhu phatak <[EMAIL PROTECTED]> wrote:
> Hi All,
> I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java
> Collection's . I have following code in reducer
That is my fault. *sigh* The input to the reducer is reused. Replace:
list.add(value);
with:
list.add(new Text(value));
and the problem will go away.
-- Owen
> Hi All,
> I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java
> Collection's . I have following code in reducer
That is my fault. *sigh* The input to the reducer is reused. Replace:
list.add(value);
with:
list.add(new Text(value));
and the problem will go away.
-- Owen
-
Re: Very strange Java Collection behavior in Hadoopmadhu phatak 2012-03-20, 06:05
Hi Owen O'Malley,
Thank you for that Instant reply. It's working now. Can you explain me
what you mean by "input to reducer is reused" in little detail?
On Tue, Mar 20, 2012 at 11:28 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> On Mon, Mar 19, 2012 at 10:52 PM, madhu phatak <[EMAIL PROTECTED]>
> wrote:
>
> > Hi All,
> > I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java
> > Collection's . I have following code in reducer
>
>
> That is my fault. *sigh* The input to the reducer is reused. Replace:
>
> list.add(value);
>
> with:
>
> list.add(new Text(value));
>
> and the problem will go away.
>
> -- Owen
>
--
https://github.com/zinnia-phatak-dev/Nectar
Thank you for that Instant reply. It's working now. Can you explain me
what you mean by "input to reducer is reused" in little detail?
On Tue, Mar 20, 2012 at 11:28 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> On Mon, Mar 19, 2012 at 10:52 PM, madhu phatak <[EMAIL PROTECTED]>
> wrote:
>
> > Hi All,
> > I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java
> > Collection's . I have following code in reducer
>
>
> That is my fault. *sigh* The input to the reducer is reused. Replace:
>
> list.add(value);
>
> with:
>
> list.add(new Text(value));
>
> and the problem will go away.
>
> -- Owen
>
--
https://github.com/zinnia-phatak-dev/Nectar
-
Re: Very strange Java Collection behavior in HadoopOwen O'Malley 2012-03-20, 06:20
On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]> wrote:
> Hi Owen O'Malley,
> Thank you for that Instant reply. It's working now. Can you explain me
> what you mean by "input to reducer is reused" in little detail?
Each time the statement "Text value = values.next();" is executed it always
returns the same Text object with the contents of that object changed. When
you add the Text to the list, you are adding a pointer to the same Text
object. At the end you have 6 copies of the same pointer instead of 6
different Text objects.
The reason that I said it is my fault, is because I added the optimization
that causes it. If you are interested in Hadoop archeology, it was
HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
documentation in the area.
-- Owen
> Hi Owen O'Malley,
> Thank you for that Instant reply. It's working now. Can you explain me
> what you mean by "input to reducer is reused" in little detail?
Each time the statement "Text value = values.next();" is executed it always
returns the same Text object with the contents of that object changed. When
you add the Text to the list, you are adding a pointer to the same Text
object. At the end you have 6 copies of the same pointer instead of 6
different Text objects.
The reason that I said it is my fault, is because I added the optimization
that causes it. If you are interested in Hadoop archeology, it was
HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
documentation in the area.
-- Owen
-
Re: Very strange Java Collection behavior in Hadoopmadhu phatak 2012-03-20, 07:42
Thanks a lot :)
On Tue, Mar 20, 2012 at 11:50 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
> wrote:
>
> > Hi Owen O'Malley,
> > Thank you for that Instant reply. It's working now. Can you explain me
> > what you mean by "input to reducer is reused" in little detail?
>
>
> Each time the statement "Text value = values.next();" is executed it always
> returns the same Text object with the contents of that object changed. When
> you add the Text to the list, you are adding a pointer to the same Text
> object. At the end you have 6 copies of the same pointer instead of 6
> different Text objects.
>
> The reason that I said it is my fault, is because I added the optimization
> that causes it. If you are interested in Hadoop archeology, it was
> HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> documentation in the area.
>
> -- Owen
>
--
https://github.com/zinnia-phatak-dev/Nectar
On Tue, Mar 20, 2012 at 11:50 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
> wrote:
>
> > Hi Owen O'Malley,
> > Thank you for that Instant reply. It's working now. Can you explain me
> > what you mean by "input to reducer is reused" in little detail?
>
>
> Each time the statement "Text value = values.next();" is executed it always
> returns the same Text object with the contents of that object changed. When
> you add the Text to the list, you are adding a pointer to the same Text
> object. At the end you have 6 copies of the same pointer instead of 6
> different Text objects.
>
> The reason that I said it is my fault, is because I added the optimization
> that causes it. If you are interested in Hadoop archeology, it was
> HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> documentation in the area.
>
> -- Owen
>
--
https://github.com/zinnia-phatak-dev/Nectar
-
Re: Very strange Java Collection behavior in HadoopMilind.Bhandarkar@... 2012-03-21, 22:14
Owen,
Is there interest in reverting hadoop-2399 in 0.23.x ?
- Milind
---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)
On 3/19/12 11:20 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
>On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
>wrote:
>
>> Hi Owen O'Malley,
>> Thank you for that Instant reply. It's working now. Can you explain me
>> what you mean by "input to reducer is reused" in little detail?
>
>
>Each time the statement "Text value = values.next();" is executed it
>always
>returns the same Text object with the contents of that object changed.
>When
>you add the Text to the list, you are adding a pointer to the same Text
>object. At the end you have 6 copies of the same pointer instead of 6
>different Text objects.
>
>The reason that I said it is my fault, is because I added the optimization
>that causes it. If you are interested in Hadoop archeology, it was
>HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
>documentation in the area.
>
>-- Owen
Is there interest in reverting hadoop-2399 in 0.23.x ?
- Milind
---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)
On 3/19/12 11:20 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
>On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
>wrote:
>
>> Hi Owen O'Malley,
>> Thank you for that Instant reply. It's working now. Can you explain me
>> what you mean by "input to reducer is reused" in little detail?
>
>
>Each time the statement "Text value = values.next();" is executed it
>always
>returns the same Text object with the contents of that object changed.
>When
>you add the Text to the list, you are adding a pointer to the same Text
>object. At the end you have 6 copies of the same pointer instead of 6
>different Text objects.
>
>The reason that I said it is my fault, is because I added the optimization
>that causes it. If you are interested in Hadoop archeology, it was
>HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
>documentation in the area.
>
>-- Owen
-
Re: Very strange Java Collection behavior in HadoopJay Vyas 2012-03-23, 06:38
Hi guys : What the latest, simplest, best directions to get a tiny,
psuedodistributed hadoop setup running on my ubuntu machine ?
On Wed, Mar 21, 2012 at 5:14 PM, <[EMAIL PROTECTED]> wrote:
> Owen,
>
> Is there interest in reverting hadoop-2399 in 0.23.x ?
>
> - Milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
>
> On 3/19/12 11:20 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
>
> >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
> >wrote:
> >
> >> Hi Owen O'Malley,
> >> Thank you for that Instant reply. It's working now. Can you explain me
> >> what you mean by "input to reducer is reused" in little detail?
> >
> >
> >Each time the statement "Text value = values.next();" is executed it
> >always
> >returns the same Text object with the contents of that object changed.
> >When
> >you add the Text to the list, you are adding a pointer to the same Text
> >object. At the end you have 6 copies of the same pointer instead of 6
> >different Text objects.
> >
> >The reason that I said it is my fault, is because I added the optimization
> >that causes it. If you are interested in Hadoop archeology, it was
> >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> >documentation in the area.
> >
> >-- Owen
>
>
--
Jay Vyas
MMSB/UCHC
psuedodistributed hadoop setup running on my ubuntu machine ?
On Wed, Mar 21, 2012 at 5:14 PM, <[EMAIL PROTECTED]> wrote:
> Owen,
>
> Is there interest in reverting hadoop-2399 in 0.23.x ?
>
> - Milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
>
> On 3/19/12 11:20 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
>
> >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
> >wrote:
> >
> >> Hi Owen O'Malley,
> >> Thank you for that Instant reply. It's working now. Can you explain me
> >> what you mean by "input to reducer is reused" in little detail?
> >
> >
> >Each time the statement "Text value = values.next();" is executed it
> >always
> >returns the same Text object with the contents of that object changed.
> >When
> >you add the Text to the list, you are adding a pointer to the same Text
> >object. At the end you have 6 copies of the same pointer instead of 6
> >different Text objects.
> >
> >The reason that I said it is my fault, is because I added the optimization
> >that causes it. If you are interested in Hadoop archeology, it was
> >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> >documentation in the area.
> >
> >-- Owen
>
>
--
Jay Vyas
MMSB/UCHC
-
Re: Very strange Java Collection behavior in HadoopJagat 2012-03-23, 06:42
Hi Jay
Just follow this to install
http://jugnu-life.blogspot.in/2012/03/hadoop-installation-tutorial.html
The official tutorial at link below is also useful
http://hadoop.apache.org/common/docs/r1.0.1/single_node_setup.html
Thanks
Jagat
On Fri, Mar 23, 2012 at 12:08 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Hi guys : What the latest, simplest, best directions to get a tiny,
> psuedodistributed hadoop setup running on my ubuntu machine ?
>
> On Wed, Mar 21, 2012 at 5:14 PM, <[EMAIL PROTECTED]> wrote:
>
> > Owen,
> >
> > Is there interest in reverting hadoop-2399 in 0.23.x ?
> >
> > - Milind
> >
> > ---
> > Milind Bhandarkar
> > Greenplum Labs, EMC
> > (Disclaimer: Opinions expressed in this email are those of the author,
> and
> > do not necessarily represent the views of any organization, past or
> > present, the author might be affiliated with.)
> >
> >
> >
> > On 3/19/12 11:20 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
> >
> > >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
> > >wrote:
> > >
> > >> Hi Owen O'Malley,
> > >> Thank you for that Instant reply. It's working now. Can you explain
> me
> > >> what you mean by "input to reducer is reused" in little detail?
> > >
> > >
> > >Each time the statement "Text value = values.next();" is executed it
> > >always
> > >returns the same Text object with the contents of that object changed.
> > >When
> > >you add the Text to the list, you are adding a pointer to the same Text
> > >object. At the end you have 6 copies of the same pointer instead of 6
> > >different Text objects.
> > >
> > >The reason that I said it is my fault, is because I added the
> optimization
> > >that causes it. If you are interested in Hadoop archeology, it was
> > >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> > >documentation in the area.
> > >
> > >-- Owen
> >
> >
>
>
> --
> Jay Vyas
> MMSB/UCHC
>
Just follow this to install
http://jugnu-life.blogspot.in/2012/03/hadoop-installation-tutorial.html
The official tutorial at link below is also useful
http://hadoop.apache.org/common/docs/r1.0.1/single_node_setup.html
Thanks
Jagat
On Fri, Mar 23, 2012 at 12:08 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Hi guys : What the latest, simplest, best directions to get a tiny,
> psuedodistributed hadoop setup running on my ubuntu machine ?
>
> On Wed, Mar 21, 2012 at 5:14 PM, <[EMAIL PROTECTED]> wrote:
>
> > Owen,
> >
> > Is there interest in reverting hadoop-2399 in 0.23.x ?
> >
> > - Milind
> >
> > ---
> > Milind Bhandarkar
> > Greenplum Labs, EMC
> > (Disclaimer: Opinions expressed in this email are those of the author,
> and
> > do not necessarily represent the views of any organization, past or
> > present, the author might be affiliated with.)
> >
> >
> >
> > On 3/19/12 11:20 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
> >
> > >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak <[EMAIL PROTECTED]>
> > >wrote:
> > >
> > >> Hi Owen O'Malley,
> > >> Thank you for that Instant reply. It's working now. Can you explain
> me
> > >> what you mean by "input to reducer is reused" in little detail?
> > >
> > >
> > >Each time the statement "Text value = values.next();" is executed it
> > >always
> > >returns the same Text object with the contents of that object changed.
> > >When
> > >you add the Text to the list, you are adding a pointer to the same Text
> > >object. At the end you have 6 copies of the same pointer instead of 6
> > >different Text objects.
> > >
> > >The reason that I said it is my fault, is because I added the
> optimization
> > >that causes it. If you are interested in Hadoop archeology, it was
> > >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> > >documentation in the area.
> > >
> > >-- Owen
> >
> >
>
>
> --
> Jay Vyas
> MMSB/UCHC
>
