Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> libhdfs process fork problem


Copy link to this message
-
Re: libhdfs process fork problem
Hi Tareq,

This is because libhdfs will keep a bit of state data (especially if the master was connected to HDFS).

Three suggestions:
1) [Likely to work] Fork the children first, then do any HDFS actions in the master.  Alternately, don't have the master do any HDFS actions; have it fork a child which does them for it.
2) [May work] Try disconnecting the master from HDFS, then forking the children.
3) [If 2 doesn't work] Have the master disconnect, then fork the children, then have the children exec.

Brian

On Mar 22, 2012, at 11:38 PM, Tareq Aljabban wrote:

> Hi,
> I'm using libhdfs C interface to access HDFS.
> The application accessing HDFS has a master process that forks many
> processes, and these processes try to connect to HDFS.
> Upon connection, one of the processes is exiting as soon as it reaches the
> hdfsConnect().
> I tried simulating this behavior by creating a separate application where
> the master process forks two processes that connect to HDFS and send
> requests.. this worked without any problem.. so I got confused.. what's
> really causing the connection problems then?
> Any insights on this is much appreciated.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB