Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Pig and Hadoop Integration Error
rahul
2010-08-27, 00:32
Jeff Zhang
2010-08-27, 01:10
rahul
2010-08-27, 01:22
Jeff Zhang
2010-08-27, 01:30
Jeff Zhang
2010-08-27, 01:33
rahul
2010-08-27, 01:49
rahul
2010-08-27, 01:51
Jeff Zhang
2010-08-27, 01:59
rahul
2010-08-27, 02:00
Jeff Zhang
2010-08-27, 02:07
rahul
2010-08-27, 02:12
Jeff Zhang
2010-08-27, 02:23
rahul
2010-08-27, 02:49
Jeff Zhang
2010-08-27, 03:17
rahul
2010-08-27, 16:17
Copy link to this message
-
RE: Pig and Hadoop Integration Error
Santhosh Srinivasan 2010-08-27, 01:25
Can you try replacing localhost with the fully qualified name of your host?

Santhosh
 

-----Original Message-----
From: rahul [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 26, 2010 6:22 PM
To: [EMAIL PROTECTED]
Subject: Re: Pig and Hadoop Integration Error

Hi Jeff,

I have set the hadoop conf in class path by setting $HADOOP_CONF_DIR variable.

But I have both Pig and hadoop running at the same machine, so localhost should not make a difference.

So I have used all the default config setting for the core-site.xml, hdfs-site.xml, mapred-site.xml, as per the hadoop tutorial.

Please let me know if my understanding is correct ?

I am attaching the conf files as well :
hdfs-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description> </property>

<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>

</configuration>

core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/Users/rahulmalviya/Documents/Hadoop/hadoop-0.21.0/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description> </property> </configuration>

mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>localhost:9001</value>
  <description>The host and port that the MapReduce job tracker runs
  at. If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>

<property>
<name>mapred.tasktracker.tasks.maximum</name>
<value>8</value>
<description>The maximum number of tasks that will be run simultaneously by a a task tracker </description> </property> </configuration>

Please let me know if there is a issue in my configurations ? Any input is valuable for me.

Thanks,
Rahul

On Aug 26, 2010, at 6:10 PM, Jeff Zhang wrote:

> Do you put the hadoop conf on classpath ? It seems you are still using
> local file system but conncect Hadoop's JobTracker.
> Make sure you set the correct configuration in core-site.xml
> hdfs-site.xml, mapred-site.xml, and put them on classpath.
>
>
>
> On Thu, Aug 26, 2010 at 5:32 PM, rahul <[EMAIL PROTECTED]> wrote:
>> Hi ,
>>
>> I am trying to integrate Pig with Hadoop for processing of jobs.
>>
>> I am able to run Pig in local mode and Hadoop with streaming api perfectly.
>>
>> But when I try to run Pig with Hadoop I get follwong Error:
>>
>> Pig Stack Trace
>> ---------------
>> ERROR 2116: Unexpected error. Could not validate the output
>> specification for:
>> file:///Users/rahulmalviya/Documents/Pig/dev/main_merged_hdp_out
>>
>> org.apache.pig.impl.plan.PlanValidationException: ERROR 0: An unexpected exception caused the validation to stop
>>        at org.apache.pig.impl.plan.PlanValidator.validate(PlanValidator.java:56)
>>        at org.apache.pig.impl.logicalLayer.validators.InputOutputFileValidator.validate(InputOutputFileValidator.java:49)
>>        at org.apache.pig.impl.logicalLayer.validators.InputOutputFileValidator.validate(InputOutputFileValidator.java:37)