Hi,
We have many data in complex formats, similar to a json file, with nested collections and I would like to know what is the best approach to implement an adapter to read it efficiently.Users will probably end up joining the top level struct with the nested collections of struct when they need the nested data, so this is something I want to assure that works.But I would like the adapter to be able to select the used columns to be read from the nested collections, because they can be very large.It seems the function UNNEST is the way to make the nested collection available as a table in Calcite, but the whole collection is read with all the columns, so it is not efficient.Probably I'll need rules to push the projects used from the nested collections to the TableScan.It would be even greater if the query didn't need to use a function like UNNEST, but simply allow a join between the table and the nested collection field.Is there anything available for this?Any suggestion?
Best regards,
Luis Fernando
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB