Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Is there any way to use a hdfs file as a Circular buffer?


Copy link to this message
-
Re: Is there any way to use a hdfs file as a Circular buffer?
Niels Basjes 2013-07-24, 22:22
A circular file on hdfs is not possible.

Some of the ways around this limitation:
- Create a series of files and delete the oldest file when you have too
much.
- Put the data into an hbase table and do something similar.
- Use completely different technology like mongodb which has built in
support for a circular buffer (capped collection).

Niels

Hi all,
   Is there any way to use a hdfs file as a Circular buffer? I mean,
if I set a quotas to a directory on hdfs, and writting data to a file
in that directory continuously. Once the quotas exceeded, I can
redirect the writter and write the data from the beginning of the file
automatically .