-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/
-----------------------------------------------------------

Review request for hive.
Description
-------

Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:

CREATE EXTERNAL TABLE hive_hbase_test
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
TBLPROPERTIES ("hbase.table.name" = "TEST_NEW_KEPLER_TABLE");

This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
This addresses bug HIVE-3725.
    https://issues.apache.org/jira/browse/HIVE-3725
Diffs
-----

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282

Diff: https://reviews.apache.org/r/9276/diff/
Testing
-------

Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
Thanks,

Swarnim Kulkarni

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/
-----------------------------------------------------------

(Updated Feb. 3, 2013, 1:04 a.m.)
Review request for hive.
Changes
-------

Updated description.
Description (updated)
-------

Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:

CREATE EXTERNAL TABLE hive_hbase_test
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");

This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
This addresses bug HIVE-3725.
    https://issues.apache.org/jira/browse/HIVE-3725
Diffs
-----

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282

Diff: https://reviews.apache.org/r/9276/diff/
Testing
-------

Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
Thanks,

Swarnim Kulkarni

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/#review16080
-----------------------------------------------------------

hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
<https://reviews.apache.org/r/9276/#comment34401>

    This seems like a limited case of pattern matching. Swarnim, any way we can support generic regex matching instead?
- Mark Grover
On Feb. 3, 2013, 1:04 a.m., Swarnim Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9276/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2013, 1:04 a.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:
>
> CREATE EXTERNAL TABLE hive_hbase_test
> ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
> TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");
>
> This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
>
>
> This addresses bug HIVE-3725.
>     https://issues.apache.org/jira/browse/HIVE-3725
>
>
> Diffs
> -----
>
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282
>
> Diff: https://reviews.apache.org/r/9276/diff/
>
>
> Testing
> -------
>
> Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
>
>
> Thanks,
>
> Swarnim Kulkarni
>
>

> On Feb. 5, 2013, 3:43 a.m., Mark Grover wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, line 192
> > <https://reviews.apache.org/r/9276/diff/1/?file=254957#file254957line192>
> >
> >     This seems like a limited case of pattern matching. Swarnim, any way we can support generic regex matching instead?

Mark, in this case I specifically wanted to only allow strings that end with exactly the character "*" and using String#endsWith seemed more simpler and readable than a regex. Do you still want me to replace this with a regex matching?
- Swarnim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/#review16080
-----------------------------------------------------------
On Feb. 3, 2013, 1:04 a.m., Swarnim Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9276/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2013, 1:04 a.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:
>
> CREATE EXTERNAL TABLE hive_hbase_test
> ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
> TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");
>
> This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
>
>
> This addresses bug HIVE-3725.
>     https://issues.apache.org/jira/browse/HIVE-3725
>
>
> Diffs
> -----
>
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282
>
> Diff: https://reviews.apache.org/r/9276/diff/
>
>
> Testing
> -------
>
> Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
>
>
> Thanks,
>
> Swarnim Kulkarni
>
>

> On Feb. 5, 2013, 3:43 a.m., Mark Grover wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, line 192
> > <https://reviews.apache.org/r/9276/diff/1/?file=254957#file254957line192>
> >
> >     This seems like a limited case of pattern matching. Swarnim, any way we can support generic regex matching instead?
>
> Swarnim Kulkarni wrote:
>     Mark, in this case I specifically wanted to only allow strings that end with exactly the character "*" and using String#endsWith seemed more simpler and readable than a regex. Do you still want me to replace this with a regex matching?
>
> Brock Noland wrote:
>     I think the issue is that this would make it difficult to implement enhanced pattern matching later. Implementing it now, you'd only need to specify:
>    
>     col.*
>    
>     in the table configuration. Now the issue would be detecting if the particular column was a regex pattern. Because #, comma, and : are used as separators that would exclude those characters from being used.
>
> Swarnim Kulkarni wrote:
>     Thanks Brock. Makes sense. To be sure I am understanding you right, the change now would be just to replace the "parts[1].endsWith(*)" with something more regexy that would still imply that the string ends with "*". Correct?
>
> Mark Grover wrote:
>     I think that should be do it.
>    
>     Personally, I think having limited regex matching is just going to confuse people, so if you could implement (and test) full Nava style regex matching (like we do for RegexSerDe for example), that would be fantastic. Of course, let me know if you have questions!
>    
>     Thanks for doing this, BTW!

Thanks for the suggestions. I incorporated them and updated the review. If you get a chance, please let me know if they look any better.
- Swarnim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/#review16080
-----------------------------------------------------------
On Feb. 9, 2013, 9:56 p.m., Swarnim Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9276/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2013, 9:56 p.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:
>
> CREATE EXTERNAL TABLE hive_hbase_test
> ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
> TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");
>
> This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
>
>
> This addresses bug HIVE-3725.
>     https://issues.apache.org/jira/browse/HIVE-3725
>
>
> Diffs
> -----
>
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282
>
> Diff: https://reviews.apache.org/r/9276/diff/
>
>
> Testing
> -------
>
> Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
>
>
> Thanks,
>
> Swarnim Kulkarni
>
>

> On Feb. 5, 2013, 3:43 a.m., Mark Grover wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, line 192
> > <https://reviews.apache.org/r/9276/diff/1/?file=254957#file254957line192>
> >
> >     This seems like a limited case of pattern matching. Swarnim, any way we can support generic regex matching instead?
>
> Swarnim Kulkarni wrote:
>     Mark, in this case I specifically wanted to only allow strings that end with exactly the character "*" and using String#endsWith seemed more simpler and readable than a regex. Do you still want me to replace this with a regex matching?

I think the issue is that this would make it difficult to implement enhanced pattern matching later. Implementing it now, you'd only need to specify:

col.*

in the table configuration. Now the issue would be detecting if the particular column was a regex pattern. Because #, comma, and : are used as separators that would exclude those characters from being used.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/#review16080
-----------------------------------------------------------
On Feb. 3, 2013, 1:04 a.m., Swarnim Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9276/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2013, 1:04 a.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:
>
> CREATE EXTERNAL TABLE hive_hbase_test
> ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
> TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");
>
> This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
>
>
> This addresses bug HIVE-3725.
>     https://issues.apache.org/jira/browse/HIVE-3725
>
>
> Diffs
> -----
>
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282
>
> Diff: https://reviews.apache.org/r/9276/diff/
>
>
> Testing
> -------
>
> Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
>
>
> Thanks,
>
> Swarnim Kulkarni
>
>

> On Feb. 5, 2013, 3:43 a.m., Mark Grover wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, line 192
> > <https://reviews.apache.org/r/9276/diff/1/?file=254957#file254957line192>
> >
> >     This seems like a limited case of pattern matching. Swarnim, any way we can support generic regex matching instead?
>
> Swarnim Kulkarni wrote:
>     Mark, in this case I specifically wanted to only allow strings that end with exactly the character "*" and using String#endsWith seemed more simpler and readable than a regex. Do you still want me to replace this with a regex matching?
>
> Brock Noland wrote:
>     I think the issue is that this would make it difficult to implement enhanced pattern matching later. Implementing it now, you'd only need to specify:
>    
>     col.*
>    
>     in the table configuration. Now the issue would be detecting if the particular column was a regex pattern. Because #, comma, and : are used as separators that would exclude those characters from being used.

Thanks Brock. Makes sense. To be sure I am understanding you right, the change now would be just to replace the "parts[1].endsWith(*)" with something more regexy that would still imply that the string ends with "*". Correct?
- Swarnim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/#review16080
-----------------------------------------------------------
On Feb. 3, 2013, 1:04 a.m., Swarnim Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9276/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2013, 1:04 a.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:
>
> CREATE EXTERNAL TABLE hive_hbase_test
> ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
> TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");
>
> This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
>
>
> This addresses bug HIVE-3725.
>     https://issues.apache.org/jira/browse/HIVE-3725
>
>
> Diffs
> -----
>
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282
>
> Diff: https://reviews.apache.org/r/9276/diff/
>
>
> Testing
> -------
>
> Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
>
>
> Thanks,
>
> Swarnim Kulkarni
>
>

> On Feb. 5, 2013, 3:43 a.m., Mark Grover wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, line 192
> > <https://reviews.apache.org/r/9276/diff/1/?file=254957#file254957line192>
> >
> >     This seems like a limited case of pattern matching. Swarnim, any way we can support generic regex matching instead?
>
> Swarnim Kulkarni wrote:
>     Mark, in this case I specifically wanted to only allow strings that end with exactly the character "*" and using String#endsWith seemed more simpler and readable than a regex. Do you still want me to replace this with a regex matching?
>
> Brock Noland wrote:
>     I think the issue is that this would make it difficult to implement enhanced pattern matching later. Implementing it now, you'd only need to specify:
>    
>     col.*
>    
>     in the table configuration. Now the issue would be detecting if the particular column was a regex pattern. Because #, comma, and : are used as separators that would exclude those characters from being used.
>
> Swarnim Kulkarni wrote:
>     Thanks Brock. Makes sense. To be sure I am understanding you right, the change now would be just to replace the "parts[1].endsWith(*)" with something more regexy that would still imply that the string ends with "*". Correct?

I think that should be do it.

Personally, I think having limited regex matching is just going to confuse people, so if you could implement (and test) full Nava style regex matching (like we do for RegexSerDe for example), that would be fantastic. Of course, let me know if you have questions!

Thanks for doing this, BTW!
- Mark
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/#review16080
-----------------------------------------------------------
On Feb. 3, 2013, 1:04 a.m., Swarnim Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9276/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2013, 1:04 a.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:
>
> CREATE EXTERNAL TABLE hive_hbase_test
> ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
> TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");
>
> This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
>
>
> This addresses bug HIVE-3725.
>     https://issues.apache.org/jira/browse/HIVE-3725
>
>
> Diffs
> -----
>
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282
>
> Diff: https://reviews.apache.org/r/9276/diff/
>
>
> Testing
> -------
>
> Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
>
>
> Thanks,
>
> Swarnim Kulkarni
>
>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9276/
-----------------------------------------------------------

(Updated Feb. 9, 2013, 9:56 p.m.)
Review request for hive.
Changes
-------

Updated diff with the proposed changes.
Description
-------

Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this:

CREATE EXTERNAL TABLE hive_hbase_test
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,fam1:col*")
TBLPROPERTIES ("hbase.table.name" = "TEST_HBASE_TABLE");

This would pull in all columns under column family "fam1" which start with "col". This gives a little more flexibility over pull all columns format.
This addresses bug HIVE-3725.
    https://issues.apache.org/jira/browse/HIVE-3725
Diffs (updated)
-----

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 7f37ba5
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java d35bb52
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282

Diff: https://reviews.apache.org/r/9276/diff/
Testing
-------

Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed.
Thanks,

Swarnim Kulkarni

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10868/
-----------------------------------------------------------

Review request for hive.
Summary (updated)
-----------------

Add support for vectorized string concatenation
Description (updated)
-------

Add support for vectorized string concatenation
This addresses bug HIVE-4431.
    https://issues.apache.org/jira/browse/HIVE-4431
Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java 246170d
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringConcatColCol.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringConcatColScalar.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringConcatScalarCol.java PRE-CREATION
  ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorStringExpressions.java c0e71e5

Diff: https://reviews.apache.org/r/10868/diff/
Testing
-------
Thanks,

Eric Hanson