Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Is it possible to implement transpose with PigLatin/any other MR language?


+
Subir S 2012-06-21, 08:16
Copy link to this message
-
Re: Is it possible to implement transpose with PigLatin/any other MR language?
Norbert Burger 2012-06-21, 18:15
While it may be fine for many cases, If I'm reading the Nectar code
correctly, that transpose doesn't guarantee anything about the order of
rows within each column.  In other words, transposing:

a - b -c
d - e - f
g - h - i

may give you different permutations of "a - d - g" as the first row,
depending on shuffle order.  You can trivially avoid this with one
mapper/reducer, but then you're not exploiting the framework.  Note that
you can accomplish same with a higher-level language like PIg by using a
UDF like LinkedIn's Enumerate [1] to tag each column, and then simply
GROUPing BY column number.

[1]
https://raw.github.com/linkedin/datafu/master/src/java/datafu/pig/bags/Enumerate.java

Norbert

On Thu, Jun 21, 2012 at 5:00 AM, madhu phatak <[EMAIL PROTECTED]> wrote:

> Hi,
>  Its possible in Map/Reduce. Look into the code here
>
> https://github.com/zinnia-phatak-dev/Nectar/tree/master/Nectar-regression/src/main/java/com/zinnia/nectar/regression/hadoop/primitive/mapreduce
>
>
>
> 2012/6/21 Subir S <[EMAIL PROTECTED]>
>
> > Hi,
> >
> > Is it possible to implement transpose operation of rows into columns and
> > vice versa...
> >
> >
> > i.e.
> >
> > col1 col2 col3
> > col4 col5 col6
> > col7 col8 col9
> > col10 col11 col12
> >
> > can this be converted to
> >
> > col1 col4 col7 col10
> > col2 col5 col8 col11
> > col3 col6 col9 col12
> >
> > Is this even possible with map reduce? If yes, which language helps to
> > achieve this faster?
> >
> > Thanks
> >
>
>
>
> --
> https://github.com/zinnia-phatak-dev/Nectar
>
+
Subir S 2012-06-22, 10:35