Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Re: Review Request 8559: Add Accumulo support to Sqoop


Copy link to this message
-
Re: Review Request 8559: Add Accumulo support to Sqoop
Jarek Cecho 2013-11-19, 21:02

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8559/#review29129
-----------------------------------------------------------
Thank you for following up Philip!

I run tests on all supported Hadoop profiles and it seems to be working! I'm however having difficulties with real cluster tests, where I'm getting:

Error: java.lang.ClassNotFoundException: org.apache.accumulo.core.client.MutationsRejectedException
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1526)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1491)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
        at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.<init>(DelegatingOutputFormat.java:104)
        at org.apache.sqoop.mapreduce.DelegatingOutputFormat.getRecordWriter(DelegatingOutputFormat.java:82)
        at org

Sqoop do not propagate it's entire classpath to the mapreduce job, instead it's the connector/tool/job that specifies which jars needs to be propagated. For example HCatalog integration is having a util method [1], HBase integration is using special HBase API [2]. I think that the accumulo job have to do something similar.

Links:
1: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L727
2: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java#L228

src/docs/user/accumulo.txt
<https://reviews.apache.org/r/8559/#comment56257>

    the accumulo-max-latency parameter is not closed with '+' sign.
Jarcec

- Jarek Cecho
On Nov. 18, 2013, 10:16 p.m., Philip Grim wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8559/
> -----------------------------------------------------------
>
> (Updated Nov. 18, 2013, 10:16 p.m.)
>
>
> Review request for accumulo, Sqoop and Jarek Cecho.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Adds the ability to import to an Accumulo table in much the same manner as the current HBase import capability.  Reported in JIRA as ACCUMULO-141 and SQOOP-767.
>
>
> Diffs
> -----
>
>   bin/configure-sqoop e604197
>   bin/configure-sqoop.cmd ec57e37
>   ivy.xml c5130ae
>   src/docs/user/accumulo-args.txt PRE-CREATION
>   src/docs/user/accumulo.txt PRE-CREATION
>   src/docs/user/import.txt dfc9b39
>   src/docs/user/validation.txt 282cfd6
>   src/java/org/apache/sqoop/SqoopOptions.java 5c7a56a
>   src/java/org/apache/sqoop/accumulo/AccumuloConstants.java PRE-CREATION
>   src/java/org/apache/sqoop/accumulo/AccumuloMutationProcessor.java PRE-CREATION
>   src/java/org/apache/sqoop/accumulo/AccumuloUtil.java PRE-CREATION
>   src/java/org/apache/sqoop/accumulo/MutationTransformer.java PRE-CREATION
>   src/java/org/apache/sqoop/accumulo/ToStringMutationTransformer.java PRE-CREATION
>   src/java/org/apache/sqoop/manager/SqlManager.java 1ffa40f
>   src/java/org/apache/sqoop/mapreduce/AccumuloImportJob.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/AccumuloImportMapper.java PRE-CREATION
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 018d11f
>   src/java/org/apache/sqoop/tool/ImportTool.java fbbde1d
>   src/test/org/apache/sqoop/accumulo/AccumuloTestCase.java PRE-CREATION