Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> libhdfs process fork problem

Tareq Aljabban 2012-03-23, 04:38
Copy link to this message
Re: libhdfs process fork problem
Hi Tareq,

This is because libhdfs will keep a bit of state data (especially if the master was connected to HDFS).

Three suggestions:
1) [Likely to work] Fork the children first, then do any HDFS actions in the master.  Alternately, don't have the master do any HDFS actions; have it fork a child which does them for it.
2) [May work] Try disconnecting the master from HDFS, then forking the children.
3) [If 2 doesn't work] Have the master disconnect, then fork the children, then have the children exec.


On Mar 22, 2012, at 11:38 PM, Tareq Aljabban wrote:

> Hi,
> I'm using libhdfs C interface to access HDFS.
> The application accessing HDFS has a master process that forks many
> processes, and these processes try to connect to HDFS.
> Upon connection, one of the processes is exiting as soon as it reaches the
> hdfsConnect().
> I tried simulating this behavior by creating a separate application where
> the master process forks two processes that connect to HDFS and send
> requests.. this worked without any problem.. so I got confused.. what's
> really causing the connection problems then?
> Any insights on this is much appreciated.
Tareq Aljabban 2012-03-27, 15:07
Brian Bockelman 2012-03-27, 18:49