Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Re: [jira] [Commented] (PIG-2699) Reduce the number of instances of Load and Store Funcs down to 2+1. It should be 1 in the front-end and 1 in the backend


Copy link to this message
-
Re: [jira] [Commented] (PIG-2699) Reduce the number of instances of Load and Store Funcs down to 2+1. It should be 1 in the front-end and 1 in the backend
I'm looking into it.
Julien

On Fri, Jul 6, 2012 at 10:11 AM, Daniel Dai (JIRA) <[EMAIL PROTECTED]> wrote:

>
>     [
> https://issues.apache.org/jira/browse/PIG-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408159#comment-13408159]
>
> Daniel Dai commented on PIG-2699:
> ---------------------------------
>
> Seems TestLOLoadDeterminedSchema is broken with the patch. Julien, Do you
> have time to take a look?
>
> > Reduce the number of instances of Load and Store Funcs down to 2+1. It
> should be 1 in the front-end and 1 in the backend
> >
> ------------------------------------------------------------------------------------------------------------------------
> >
> >                 Key: PIG-2699
> >                 URL: https://issues.apache.org/jira/browse/PIG-2699
> >             Project: Pig
> >          Issue Type: Bug
> >          Components: internal-udfs
> >    Affects Versions: 0.10.0
> >            Reporter: Julien Le Dem
> >            Assignee: Julien Le Dem
> >             Fix For: 0.11
> >
> >         Attachments: PIG-2699.patch, PIG-2699_a.patch, PIG-2699_b.patch,
> PIG-2699_c.patch, PIG-2699_d.patch, PIG-2699_e.patch, PIG-2699_f.patch
> >
> >
> > Attached: a patch to get it down to 3
> > Here is the report of the remaining calls.
> > some methods are unnecessarily called multiple times, this should be
> improved as well.
> > {noformat}
> > A = LOAD 'foo' USING TestLoadStoreFuncLifeCycle$Loader();
> > STORE A INTO 'bar' USING TestLoadStoreFuncLifeCycle$Storer();
> > report:
> > 3 instances of Loader
> > 20 calls to Loader
> > 3 instances of Storer
> > 24 calls to Storer
> > all calls:
> > Loader[1].<init>()
> > Loader[1].relativeToAbsolutePath(foo,
> file:/Users/julien/svn/pig/trunk-LoadStoreFunc-lifecycle)
> > Loader[1].setUDFContextSignature(A_1-0)
> > Loader[1].getSchema(foo, org.apache.hadoop.mapreduce.Job@7ee49dcd)
> > Storer[1].<init>()
> > Storer[1].setStoreFuncUDFContextSignature(A_1-1)
> > Storer[1].relToAbsPathForStoreLocation(bar,
> file:/Users/julien/svn/pig/trunk-LoadStoreFunc-lifecycle)
> > Storer[1].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@776be68f
> )
> > Storer[1].getOutputFormat()
> > Loader[1].getStatistics(foo, org.apache.hadoop.mapreduce.Job@11e9c82e)
> > Loader[1].setLocation(foo, org.apache.hadoop.mapreduce.Job@11e9c82e)
> > Storer[1].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@57d840cd
> )
> > Storer[2].<init>()
> > Storer[2].setStoreFuncUDFContextSignature(A_1-1)
> > Storer[2].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@76996cca
> )
> > Storer[2].getOutputFormat()
> > Loader[2].<init>()
> > Loader[2].setUDFContextSignature(A_1-0)
> > Loader[2].setLocation(foo, org.apache.hadoop.mapreduce.Job@317cfd38)
> > Loader[2].getInputFormat()
> > Storer[3].<init>()
> > Storer[3].setStoreFuncUDFContextSignature(A_1-1)
> > Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@459d3b3a
> )
> > Storer[3].getOutputFormat()
> > Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@225f1ae9
> )
> > Loader[3].<init>()
> > Loader[3].setUDFContextSignature(A_1-0)
> > Loader[3].setLocation(foo, org.apache.hadoop.mapreduce.Job@6b98e8b4)
> > Loader[3].getInputFormat()
> > Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@5fb11b79
> )
> > Storer[3].getOutputFormat()
> >
> Storer[3].prepareToWrite(org.apache.pig.builtin.mock.Storage$MockRecordWriter@49b09282
> )
> > Loader[3].setUDFContextSignature(A_1-0)
> >
> Loader[3].prepareToRead(org.apache.pig.builtin.mock.Storage$MockRecordReader@2c8c7d6,
> Number of splits :1...)
> > Loader[3].getNext()
> > Storer[3].putNext((a))
> > Loader[3].getNext()
> > Storer[3].putNext((b))
> > Loader[3].getNext()
> > Storer[3].putNext((c))
> > Loader[3].getNext()
> > Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@3ebfbbe3
> )
> > Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@14d964af
> )
> > Storer[1].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job@644ca6b6
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB