Harsh J 2013-02-22, 06:03
abhishek 2013-02-22, 06:35
abhishek 2013-02-22, 16:03
Harsh J 2013-02-22, 16:09
-Re: Locks in HDFS
abhishek 2013-02-22, 16:41
As a part of my use case, my work was to read a file from HDFS and update a value in it , the value gets incremented for every source load ( load id )
Source load1 is started the value in the file could incremented by 1.
Source load 2 is started then value in the file could incremented by 1.
If two sources or multiple sources kicks off at same time, then I want other sources to wait till source1 updates the value in the file.So that other sources read the correct value and increment it by 1.
So I am looking for a best approach to lock the file till source1 (or) which ever source increments the value in the file.
So I thought it may be possible to load or put a file in HDFS, with single replication and use java synchronized method. So that only user or app can read access file at any given point of time.
On Feb 22, 2013, at 11:09 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hi Abhishek,
> I fail to understand what you mean by that; but HDFS generally has no
> client-exposed file locking on reads. There's leases for preventing
> multiple writers to a single file, but nothing on the read side.
> Replication of the blocks under a file is a different concept and is
> completely unrelated to this.
> This needs to be built at your application's/stack's access/control
> levels, since HDFS does not provide this.
> On Fri, Feb 22, 2013 at 9:33 PM, abhishek <[EMAIL PROTECTED]> wrote:
>> Can we load the file into HDFS with one replication and lock the file.
>> On Feb 22, 2013, at 1:03 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>> HDFS does not have such a client-side feature, but your applications
>>> can use Apache Zookeeper to coordinate and implement this on their own
>>> - it can be used to achieve distributed locking. While at ZooKeeper,
>>> also checkout https://github.com/Netflix/curator which makes using it
>>> for common needs very easy.
>>> On Fri, Feb 22, 2013 at 5:17 AM, abhishek <[EMAIL PROTECTED]> wrote:
>>>>> How can I impose read lock, for a file in HDFS
>>>>> So that only one user (or) one application , can access file in hdfs at any point of time.
>>> Harsh J
> Harsh J