Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request 14240: SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function


Copy link to this message
-
Re: Review Request 14240: SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function


> On Oct. 1, 2013, 7:49 p.m., Jarek Cecho wrote:
> > Thank you Alexandre for taking up this ticket!
> >
> > Would you mind adding automated tests to ensure that the functionality is indeed working?

I'd like to be able to integration test bulk load but this is still not a possibility in non-distributed mode. We currently have the same problems in some of our map/reduce jobs that use bulk loading. The way we've approached it with those is that we have two modes for each job: bulk load enabled/bulk load disabled. The integration tests always run with bulk load disabled. The "bulk load mode" is tested on a real cluster with automated tests that run from jenkins.

For what it's worth, I've done the manual testing on our cluster for this patch.
- Alexandre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14240/#review26576
-----------------------------------------------------------
On Oct. 2, 2013, 9:21 p.m., Alexandre Normand wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14240/
> -----------------------------------------------------------
>
> (Updated Oct. 2, 2013, 9:21 p.m.)
>
>
> Review request for Sqoop, Jarek Cecho, Jean-Marc Spaggiari, and vasanthkumar.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is the patch to address some of the comments on Zhancheng Deng's review request (https://reviews.apache.org/r/13052/) which was basically the change to add the --bulk-load-dir option to support the HBase doBulkLoad function.
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/SqoopOptions.java 01805f9
>   src/java/org/apache/sqoop/hbase/HBasePutProcessor.java 9ceb5bd
>   src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java 5ccf311
>   src/java/org/apache/sqoop/manager/SqlManager.java 2a4992d
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java ab7f21e
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 0eca991
>
> Diff: https://reviews.apache.org/r/14240/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Alexandre Normand
>
>