Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Review Request: HIVE-4548 Speed up vectorized LIKE filter for special cases abc%, %abc and %abc%


Copy link to this message
-
Re: Review Request: HIVE-4548 Speed up vectorized LIKE filter for special cases abc%, %abc and %abc%


> On May 22, 2013, midnight, Eric Hanson wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java, line 227
> > <https://reviews.apache.org/r/11222/diff/2/?file=295212#file295212line227>
> >
> >     I think this code will fail for multi-byte characters beyond the standard Unicode code points. char is 16 bits. String in java represents supplementary characters by surrogate pairs (2 char values). If the 2nd char of a surrogate pair is the same as, say, '_' then this will fail.
> >    
> >     the Text class has a charAt() that returns an int (32 bit Unicode value). Maybe you could use that. Otherwise you may have to look for start byte characters in UTF-8.
> >    
> >     See http://docs.oracle.com/javase/6/docs/api/java/lang/String.html
> >     http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/Text.html
> >     http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html
> >

Thank you for your review. I never knew this problem. I will try the Text class first to solve it.

This method is copied from the UDFLike class, so there is same problem. I'll consider a reusable method for those classes.
- Teddy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11222/#review20877
-----------------------------------------------------------
On May 21, 2013, 12:14 p.m., Teddy Choi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11222/
> -----------------------------------------------------------
>
> (Updated May 21, 2013, 12:14 p.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> I edited FilterStringColLikeStringScala.java as Eric Hanson wrote.
>
> For none-complex patterns, it calls a static method that doesn't call others and uses its given byte arrays only. For complex patterns, it reuses a ByteBuffer and a CharBuffer for decoding UTF-8 to avoid object constructions.
>
> There is 30%~170% performance improvement for all cases. Its benchmark result is on https://issues.apache.org/jira/browse/HIVE-4548#comment-13660750.
>
> It still can be more efficient by using a template-driven approach. I'll apply it soon.
>
>
> This addresses bug HIVE-4548.
>     https://issues.apache.org/jira/browse/HIVE-4548
>
>
> Diffs
> -----
>
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java 24ba861
>   ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorStringExpressions.java 6e26412
>
> Diff: https://reviews.apache.org/r/11222/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Teddy Choi
>
>