Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> a file can be used as a queue?


Copy link to this message
-
Re: a file can be used as a queue?
This feature doesn't currently work. I don't remember the JIRA for it, but there's a ticket which will allow a reader to read from an HDFS file before it's closed. In that case, you implement a queue by having the producer write to the end of the file and the reader read from the beginning of the file.

I'm not sure if there will be a way to tell that a file is still being written, so you may need your own end of stream marker.

-Joey

On Jun 13, 2011, at 2:55, ltomuno <[EMAIL PROTECTED]> wrote:

> I heard a HDFS file as a producer - consumer queue, a file can be used as a queue? I am very confused
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB