Hi,
Currently I am facing an issue with Hive/HBase integration.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;

StackTrace:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;

    at org.apache.hadoop.security.UnixUserGroupInformation.getUnixGroups(UnixUserGroupInformation.java:320)

    at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:243)

    at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)

    at org.apache.hadoop.hive.ql.Driver.<init>(Driver.java:273)

    at org.apache.hadoop.hive.ql.processors.CommandProcessorFactory.get(CommandProcessorFactory.java:49)

    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:131)

    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:302)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

I believe, It is because  of jdk6 backward compatibility. I tried to set -Dsun.lang.ClassLoader.allowArraySyntax=true. But unfortunately didn't work.

Any help will be greatly appreciated.

Thanks and Regards,

Vivek Mishra
________________________________
Impetus is a proud sponsor for ASCI Tour 2010 (Agile Software Community of India) on Oct 30 in Noida, India.

Meet Impetus at the Cloud Computing Expo from Nov 1-4 in Santa Clara. Our Sr. Director of Engineering, Vineet Tyagi will be speaking about 'Using Hadoop for Deriving Intelligence from Large Data'.

Click http://www.impetus.com/ to know more. Follow us on www.twitter.com/impetuscalling

NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Hi,

Does  the INSERT clause have to include the OVERWRITE, which means that the
new data will overwrite the previous data?  How to implement the indeed
INSERT operation, instead of OVERWRITE?
BTW: How to implement the DELETE operator? thanks

afancy
-------------------------------------------------------
hive> insert OVERWRITE table  pagedim select 0, url, strToint('2'),
'domain', 'serversion' from downloadlog;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201011121525_0006, Tracking URL http://localhost:50030/jobdetails.jsp?jobid=job_201011121525_0006
Kill Command = /home/xiliu/hadoop-0.20.2/bin/../bin/hadoop job
 -Dmapred.job.tracker=localhost:54311 -kill job_201011121525_0006
2010-11-18 09:55:52,155 Stage-1 map = 0%,  reduce = 0%
2010-11-18 09:55:55,169 Stage-1 map = 100%,  reduce = 0%
2010-11-18 09:55:58,200 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201011121525_0006
Ended Job = 487027960, job is filtered out (removed at runtime).
Launching Job 2 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201011121525_0007, Tracking URL http://localhost:50030/jobdetails.jsp?jobid=job_201011121525_0007
Kill Command = /home/xiliu/hadoop-0.20.2/bin/../bin/hadoop job
 -Dmapred.job.tracker=localhost:54311 -kill job_201011121525_0007
2010-11-18 09:56:04,701 Stage-2 map = 0%,  reduce = 0%
2010-11-18 09:56:07,723 Stage-2 map = 100%,  reduce = 0%
2010-11-18 09:56:10,751 Stage-2 map = 100%,  reduce = 100%
Ended Job = job_201011121525_0007
Loading data to table pagedim
1000 Rows loaded to pagedim
OK
Time taken: 23.194 seconds
hive> insert table  pagedim select 0, url, strToint('2'), 'domain',
'serversion' from downloadlog;
FAILED: Parse Error: line 1:7 mismatched input 'table' expecting OVERWRITE
in insert clause

As noted here, when writing to HBase, existing rows are overwritten, but old rows are not deleted.

http://wiki.apache.org/hadoop/Hive/HBaseIntegration#Overwrite

There is not yet any deletion support.

JVS

On Nov 18, 2010, at 1:00 AM, afancy wrote:

> Hi,
>
> Does  the INSERT clause have to include the OVERWRITE, which means that the new data will overwrite the previous data?  How to implement the indeed INSERT operation, instead of OVERWRITE?
> BTW: How to implement the DELETE operator? thanks
>
> afancy
>
>
> -------------------------------------------------------
> hive> insert OVERWRITE table  pagedim select 0, url, strToint('2'), 'domain', 'serversion' from downloadlog;
> Total MapReduce jobs = 2
> Launching Job 1 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201011121525_0006, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201011121525_0006
> Kill Command = /home/xiliu/hadoop-0.20.2/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:54311 -kill job_201011121525_0006
> 2010-11-18 09:55:52,155 Stage-1 map = 0%,  reduce = 0%
> 2010-11-18 09:55:55,169 Stage-1 map = 100%,  reduce = 0%
> 2010-11-18 09:55:58,200 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201011121525_0006
> Ended Job = 487027960, job is filtered out (removed at runtime).
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201011121525_0007, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201011121525_0007
> Kill Command = /home/xiliu/hadoop-0.20.2/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:54311 -kill job_201011121525_0007
> 2010-11-18 09:56:04,701 Stage-2 map = 0%,  reduce = 0%
> 2010-11-18 09:56:07,723 Stage-2 map = 100%,  reduce = 0%
> 2010-11-18 09:56:10,751 Stage-2 map = 100%,  reduce = 100%
> Ended Job = job_201011121525_0007
> Loading data to table pagedim
> 1000 Rows loaded to pagedim
> OK
> Time taken: 23.194 seconds
> hive> insert table  pagedim select 0, url, strToint('2'), 'domain', 'serversion' from downloadlog;          
> FAILED: Parse Error: line 1:7 mismatched input 'table' expecting OVERWRITE in insert clause
>

This is unrelated to Hive/HBase integration; it looks like a Hadoop version issue.

JVS

On Nov 17, 2010, at 9:56 PM, Vivek Mishra wrote:

> Hi,
> Currently I am facing an issue with Hive/HBase integration.
>  
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;
>  
> StackTrace:
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;
>     at org.apache.hadoop.security.UnixUserGroupInformation.getUnixGroups(UnixUserGroupInformation.java:320)
>     at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:243)
>     at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>     at org.apache.hadoop.hive.ql.Driver.<init>(Driver.java:273)
>     at org.apache.hadoop.hive.ql.processors.CommandProcessorFactory.get(CommandProcessorFactory.java:49)
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:131)
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:302)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>  
>  
> I believe, It is because  of jdk6 backward compatibility. I tried to set -Dsun.lang.ClassLoader.allowArraySyntax=true. But unfortunately didn't work.
>  
>  
> Any help will be greatly appreciated.
>  
>  
> Thanks and Regards,
> Vivek Mishra
>  
>
> Impetus is a proud sponsor for ASCI Tour 2010 (Agile Software Community of India) on Oct 30 in Noida, India.
>
> Meet Impetus at the Cloud Computing Expo from Nov 1-4 in Santa Clara. Our Sr. Director of Engineering, Vineet Tyagi will be speaking about ‘Using Hadoop for Deriving Intelligence from Large Data’.
>
> Click http://www.impetus.com/ to know more. Follow us on www.twitter.com/impetuscalling
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Hi,
Just found that, It is related to HIVE-1264 JIRA. Thanks for all help.
Vivek

-----Original Message-----
From: John Sichi [mailto:[EMAIL PROTECTED]]
Sent: Friday, November 19, 2010 1:02 AM
To: <[EMAIL PROTECTED]>
Subject: Re: Hive/HBase integration issue.

This is unrelated to Hive/HBase integration; it looks like a Hadoop version issue.

JVS

On Nov 17, 2010, at 9:56 PM, Vivek Mishra wrote:

> Hi,
> Currently I am facing an issue with Hive/HBase integration.
>
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;
>
> StackTrace:
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;
>     at org.apache.hadoop.security.UnixUserGroupInformation.getUnixGroups(UnixUserGroupInformation.java:320)
>     at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:243)
>     at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>     at org.apache.hadoop.hive.ql.Driver.<init>(Driver.java:273)
>     at org.apache.hadoop.hive.ql.processors.CommandProcessorFactory.get(CommandProcessorFactory.java:49)
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:131)
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:302)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
>
> I believe, It is because  of jdk6 backward compatibility. I tried to set -Dsun.lang.ClassLoader.allowArraySyntax=true. But unfortunately didn't work.
>
>
> Any help will be greatly appreciated.
>
>
> Thanks and Regards,
> Vivek Mishra
>
>
> Impetus is a proud sponsor for ASCI Tour 2010 (Agile Software Community of India) on Oct 30 in Noida, India.
>
> Meet Impetus at the Cloud Computing Expo from Nov 1-4 in Santa Clara. Our Sr. Director of Engineering, Vineet Tyagi will be speaking about ‘Using Hadoop for Deriving Intelligence from Large Data’.
>
> Click http://www.impetus.com/ to know more. Follow us on www.twitter.com/impetuscalling
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Impetus is a proud sponsor for ASCI Tour 2010 (Agile Software Community of India) on Oct 30 in Noida, India.

Meet Impetus at the Cloud Computing Expo from Nov 1-4 in Santa Clara. Our Sr. Director of Engineering, Vineet Tyagi will be speaking about ‘Using Hadoop for Deriving Intelligence from Large Data’.

Click http://www.impetus.com/ to know more. Follow us on www.twitter.com/impetuscalling

NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Added 1 post at:
http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/

Sharing it if is useful.

Vivek

-----Original Message-----
From: Vivek Mishra
Sent: Friday, November 19, 2010 10:36 AM
To: <[EMAIL PROTECTED]>
Subject: RE: Hive/HBase integration issue.

Hi,
Just found that, It is related to HIVE-1264 JIRA. Thanks for all help.
Vivek

-----Original Message-----
From: John Sichi [mailto:[EMAIL PROTECTED]]
Sent: Friday, November 19, 2010 1:02 AM
To: <[EMAIL PROTECTED]>
Subject: Re: Hive/HBase integration issue.

This is unrelated to Hive/HBase integration; it looks like a Hadoop version issue.

JVS

On Nov 17, 2010, at 9:56 PM, Vivek Mishra wrote:

> Hi,
> Currently I am facing an issue with Hive/HBase integration.
>
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;
>
> StackTrace:
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Shell.getGROUPS_COMMAND()[Ljava/lang/String;
>     at org.apache.hadoop.security.UnixUserGroupInformation.getUnixGroups(UnixUserGroupInformation.java:320)
>     at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:243)
>     at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>     at org.apache.hadoop.hive.ql.Driver.<init>(Driver.java:273)
>     at org.apache.hadoop.hive.ql.processors.CommandProcessorFactory.get(CommandProcessorFactory.java:49)
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:131)
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:302)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
>
> I believe, It is because  of jdk6 backward compatibility. I tried to set -Dsun.lang.ClassLoader.allowArraySyntax=true. But unfortunately didn't work.
>
>
> Any help will be greatly appreciated.
>
>
> Thanks and Regards,
> Vivek Mishra
>
>
> Impetus is a proud sponsor for ASCI Tour 2010 (Agile Software Community of India) on Oct 30 in Noida, India.
>
> Meet Impetus at the Cloud Computing Expo from Nov 1-4 in Santa Clara. Our Sr. Director of Engineering, Vineet Tyagi will be speaking about ‘Using Hadoop for Deriving Intelligence from Large Data’.
>
> Click http://www.impetus.com/ to know more. Follow us on www.twitter.com/impetuscalling
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Watch Vineet Tyagi (Impetus – Sr. Director of Engineering)  explaining ‘The Hadoop, Cloud, and Mafia Connection ‘ at http://tinyurl.com/376gl62

Follow Impetus’ updates on www.twitter.com/impetuscalling.

NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.