Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> Query plans that write to in-memory queue rather than file


Copy link to this message
-
Query plans that write to in-memory queue rather than file
Hi drillers,

I made a small architectural change in my first iteration of the SQL parser/validator/optimizer. I changed the LogicalPlan specification to allow query output to go to a java.util.Queue data structure rather than a file.

It was convenient for my purposes because I didn't need to allocate a temporary file, wait til the process had finished, and then start returning results. I know that in the big data world, many queries will produce large results and the results will not pass through just one processing node. But still, many queries will produce small results.

A logical plan is a graph, and there is an argument that it should be symmetric. A logical plan has sources, so why shouldn't it declare sinks?

I hacked the change in [ see my changes to ROPConverter in https://github.com/apache/incubator-drill/commit/7f294adb649064e26dd2f28864260b17b07315ef ] , creating a "write" operator with a special file name "socket:0", and I modified ROPConverter to create a QueueSinkROP rather than the usual JSONWriter.

Let's discuss what is the correct architecture for this.

Julian
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB