Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Filtering


Hi,

I'm using Hive 0.10.0 and Hadoop 1.0.4.

I would like to create a normal table but have some of my code run so that I can remove filtering
parts of the query and limit the output in the splits of the InputFormat. I believe that this is
"Filtering Pushdown" as described in https://cwiki.apache.org/Hive/filterpushdowndev.html
I have tried various approaches and run into problems and I was wondering
if anyone had any suggestions as to how I might proceed.

Firstly although that page mentions InputFormat there doesn't seem to be any way (that I can find)
to perform filter passing to InputFormats and so I gave up on that approach.

I see that in method pushFilterToStorageHandler around line 776 of file OpProcFactory.java
there is a call to the decomposePredicate method of the storage handler if it's an instance of HiveStoragePredicateHandler.
This looks exactly like what I want. So my first approach is to provide my own StorageHandler.

However when I create a table with my own custom storage handler I get this behaviour:

                > DROP TABLE ordinals2;
                OK
                Time taken: 0.136 seconds
                hive> CREATE TABLE ordinals2 (english STRING, number INT, italian STRING)
                >  ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
                >  STORED BY 'StorageWindow';
                getInputFormatClass
                getOutputFormatClass
                OK
                Time taken: 0.075 seconds
                hive> LOAD DATA LOCAL INPATH 'ordinals.txt' OVERWRITE INTO TABLE ordinals2;
                FAILED: SemanticException [Error 10101]: A non-native table cannot be used as target for LOAD
                hive>

However I don't care if the table is "non-native" or not, I just want to use it in the same
way that I use a normal table. Is there some way that I can get the normal behaviour
and still use a custom storage handler?

OK, given that the StorageHandler approach doesn't seem to working out for me I tried
the next obvious approach - Indexes. So I created my own index class called "DummyIndex'.
Because I wanted to keep this index as simple as possible I had it return 'false' in the
method usesIndexTable. However when I then try and create an index of this type I get
this error.

                > CREATE INDEX dummy_ordinals ON TABLE ordinals (italian)
                >  AS 'DummyIndex' WITH DEFERRED REBUILD;
                FAILED: Error in metadata: java.lang.NullPointerException
                FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
                hive>

A little investigation suggests that this is failing in method add_Index
lines 2796-2822 of file HiveMetaStore.java. Here we can see that the
argument indexTable is de-referenced on line 2798 for what looks like a tracing
statement. This gives the NPE if the argument is null.

If we look at method createIndex lines 614-759 of file Hive.java
we can see that the variable tt is initialised to null on line 722 and passed
to the CreateIndex method on line 754. But it can only change to a non-null
value if the indexHandler returns true in the usesIndexTable method on line 725.

The bottom line seems to be that the code can't work if you have an index
that returns false for usesIndexTable. Isn't this a bug? Do I need to raise a JIRA?

If I change my index so that it returns true for usesIndexTable but still try and keep
all the other methods minimal I get this error when I try to create an index.

                > CREATE INDEX dummy_ordinals ON TABLE ordinals (italian)
                >  AS 'DummyIndex' WITH DEFERRED REBUILD;
                FAILED: Error in metadata: at least one column must be specified for the table
                FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

So, at the moment, it looks like I'm going to have my index class derive from one of the reference
implementations (org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler) in order to get it to work at all.

Is it worth trying to build Hive from source so that I can hack out the tracing that causes the NPE?
Or is it likely to start failing somewhere else?

Any comments welcome.

Peter Marron
Trillium Software UK Limited

Tel : +44 (0) 118 940 7609
Fax : +44 (0) 118 940 7699
E: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>