Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Pig tutorial samples hang when run in local mode


Copy link to this message
-
RE: Pig tutorial samples hang when run in local mode
Olga Natkovich 2009-05-06, 01:20
I traced the issue to the DISTINCT operator that for 0.5 MB of data uses
over 1GB of memory and so I think it swaps forever. Will dig more
tomorrow.

Olga

> -----Original Message-----
> From: Olga Natkovich [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 05, 2009 2:31 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Pig tutorial samples hang when run in local mode
>
> Hi Rama,
>
> I was able to reproduce the behavior you described. Will
> investigate and let you know.
>
> Olga
>
> > -----Original Message-----
> > From: N . Ramakrishna [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, May 04, 2009 11:06 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Pig tutorial samples hang when run in local mode
> >
> >
> >
> > I waited for like 4-5 hours, which seemed long enough for
> processing a
> > file which it works on (only around 4.5 K entries).
> >
> > And is there any log in which I can monitor the progress??
> I found a
> > log4j.properties file in the conf directory and turned the level to
> > debug.
> > Still no other messages on the console.
> >
> >  
> >
> >
> >
> > ----- Original Message ----
> > From: Olga Natkovich <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Monday, May 4, 2009 10:00:03 PM
> > Subject: RE: Pig tutorial samples hang when run in local mode
> >
> > Hi,
> >
> > The warnings are benign. How long were you waiting for the job to
> > complete?
> >
> > Olga
> >
> > > -----Original Message-----
> > > From: N . Ramakrishna [mailto:[EMAIL PROTECTED]]
> > > Sent: Monday, May 04, 2009 4:00 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: Pig tutorial samples hang when run in local mode
> > >
> > >
> > > Hi,
> > >
> > > All that I did is checked the pig project from the trunk
> > > (http://svn.apache.org/repos/asf/hadoop/pig/trunk), built
> it, built
> > > the tutorial, and untarred the pigtutorial file, and then ran the
> > > examples. Both the sample tutorials hang. There is some
> > output for one
> > > of the examples:
> > >
> > > $ java -cp $PIGDIR/pig.jar org.apache.pig.Main -x local
> > > script1-local.pig
> > > 2009-05-04 14:38:15,432 [main] INFO  org.apache.pig.PigServer
> > > - Create a new graph.
> > >
> > > 2009-05-04 14:38:15,893 [main] WARN  org.apache.pig.PigServer
> > > - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 3 time(s).
> > > 2009-05-04 14:38:15,893 [main] WARN  org.apache.pig.PigServer
> > > - Encountered Warning USING_OVERLOADED_FUNCTION 3 time(s).
> > >
> > >
> > > But after that, the program simply hangs..
> > > Could someone help me with what's happening here?
> > >
> > >
> > > Regards,
> > > Rama.
> > >
> > >
> > >
> > >  History is the version of past events that people have
> decided to
> > > agree upon.
> > > -- Napoleon Bonaparte
> > >
> > >
> > >
> > >      
> > >
> >
> >
> >
> >      
> >
>