Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 16533: Add StoreFunc and LoadFunc classes to Pig for Accumulo


Copy link to this message
-
Re: Review Request 16533: Add StoreFunc and LoadFunc classes to Pig for Accumulo

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16533/
-----------------------------------------------------------

(Updated Jan. 10, 2014, 7:18 p.m.)
Review request for pig.
Changes
-------

Incorporates suggestions from Daniel's review, removal of some "less desirable" Storage options, and more tests.

* Class level and constructor-level javadoc for AccumuloStorage
* Removal of AccumuloWholeRowStorage and AccumuloKeyValueStorage
* Remove accumulo15 profile from Ivy
* Added JUnit @Test timeout
* Promote configuration of AccumuloStorage through CLI-esque option parsing in constructor, but leave URI capabilities present for those currently using this code.
* Copied the TableMapReduceUtil code from HBase to localize the Accumulo dependencies automatically to alleviate the necessity for REGISTER statements.
* Implement proper LoadStoreCaster
Bugs: PIG-3573
    https://issues.apache.org/jira/browse/PIG-3573
Repository: pig-git
Description
-------

Provides basic StoreFunc and LoadFunc implementations. Based off of code that was in an Accumulo contrib project.
Diffs (updated)
-----

  build.xml 575c9ae
  ivy.xml 180eb2c
  ivy/libraries.properties 14abdf8
  src/org/apache/pig/backend/hadoop/accumulo/AbstractAccumuloStorage.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/accumulo/AccumuloBinaryConverter.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/accumulo/AccumuloStorage.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/accumulo/AccumuloStorageOptions.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/accumulo/FixedByteArrayOutputStream.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/accumulo/Utils.java PRE-CREATION
  test/org/apache/pig/backend/hadoop/accumulo/AbstractAccumuloStorageTest.java PRE-CREATION
  test/org/apache/pig/backend/hadoop/accumulo/AccumuloPigClusterTest.java PRE-CREATION
  test/org/apache/pig/backend/hadoop/accumulo/AccumuloStorageConfigurationTest.java PRE-CREATION
  test/org/apache/pig/backend/hadoop/accumulo/AccumuloStorageOptionsTest.java PRE-CREATION
  test/org/apache/pig/backend/hadoop/accumulo/AccumuloStorageTest.java PRE-CREATION

Diff: https://reviews.apache.org/r/16533/diff/
Testing
-------

Local tests reading, writing and JOIN'ing Accumulo tables. Tested against Hadoop-1.0.4 and 2.2.0, with Accumulo 1.5.0
Thanks,

Josh Elser