Tareq Aljabban 2012-03-23, 04:38
-Re: libhdfs process fork problem
Brian Bockelman 2012-03-23, 11:25
This is because libhdfs will keep a bit of state data (especially if the master was connected to HDFS).
1) [Likely to work] Fork the children first, then do any HDFS actions in the master. Alternately, don't have the master do any HDFS actions; have it fork a child which does them for it.
2) [May work] Try disconnecting the master from HDFS, then forking the children.
3) [If 2 doesn't work] Have the master disconnect, then fork the children, then have the children exec.
On Mar 22, 2012, at 11:38 PM, Tareq Aljabban wrote:
> I'm using libhdfs C interface to access HDFS.
> The application accessing HDFS has a master process that forks many
> processes, and these processes try to connect to HDFS.
> Upon connection, one of the processes is exiting as soon as it reaches the
> I tried simulating this behavior by creating a separate application where
> the master process forks two processes that connect to HDFS and send
> requests.. this worked without any problem.. so I got confused.. what's
> really causing the connection problems then?
> Any insights on this is much appreciated.
Tareq Aljabban 2012-03-27, 15:07
Brian Bockelman 2012-03-27, 18:49