Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Re: [SQLWindowing] Windowing function output path syntax (#26)


+
neelesh gadhia 2013-02-22, 22:05
+
Butani, Harish 2013-02-22, 22:44
+
neelesh gadhia 2013-02-22, 22:55
+
Butani, Harish 2013-02-23, 06:40
+
neelesh gadhia 2013-02-23, 07:00
+
neelesh gadhia 2013-02-25, 04:26
+
Butani, Harish 2013-02-25, 05:22
+
neelesh gadhia 2013-02-25, 06:01
+
Butani, Harish 2013-02-25, 17:53
+
neelesh gadhia 2013-02-25, 18:55
Copy link to this message
-
Re: [SQLWindowing] Windowing function output path syntax (#26)
Seen this error before. It was pointed out by Ashutosh in Hive 896.
Haven't gotten around to changing the default…
To answer your question, the stack trace has:
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$ByteBasedList.<init>(PTFPersistence.java:95)
...
Regards,
Harish
From: neelesh gadhia <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, neelesh gadhia <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Monday, February 25, 2013 10:55 AM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Cc: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

my bad. After setting the value to 16MB in bytes (16777216) it worked fine.

Set hive.ptf.partition.persistence.memsize=16777216;

So does this mean default value of 64MB was causing issue? if yes, can you please elaborate a little how were you able to troubleshoot/debut the issue to this parameter setting?

Thanks,
Neelesh

________________________________
From: "Butani, Harish" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: neelesh gadhia <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Cc: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>; Ashutosh Chauhan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Monday, February 25, 2013 9:53 AM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

How are you setting the value? It needs to be set in bytes.

From: neelesh gadhia <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: neelesh gadhia <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Sunday, February 24, 2013 10:01 PM
To: SAP SAP <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Cc: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

treid 16 and 32 mb and get a different error.

hive> Set hive.ptf.partition.persistence.memsize=16;
hive> select mid, tdate, tamt,sum(tamt) as com_sum over (rows between unbounded preceding and current row)
    > from t_enc
    > distribute by mid
    > sort by mid, tdate;

1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.SEL :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

4.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
    ]

5.PTF :
RowResolver::
    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> com_sum
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
    ]
1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.SEL :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

4.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
    ]

5.PTF :
RowResolver::
    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB