Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - setting hadoop for pseudo distributed mode.


Copy link to this message
-
Re: setting hadoop for pseudo distributed mode.
Mohammad Tariq 2012-12-27, 23:10
what are those libraries and how are they reading data from HDFS? you were
trying with MR jobs if i'm not wrong? in order to perform read/write on
HDFS we need HDFS API with a Configuration object. how are you doing it
here?

Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 2:38 AM, jamal sasha <[EMAIL PROTECTED]> wrote:

> Hi,
>   Thanks for throwing insight.
>
> So the code snippet looks like this
>
> String interout = final_output + "/intermediate";
>
>     try {
>       new CreateInterOutput().main(new String[] { input, interout });
>     } catch (Exception e) {
>       e.printStackTrace();
>       return;
>     }
>
>     try {
>     new CreateFinalOutput().main(new String[] { interout, final_output });
>
>     } catch (Exception e) {
>       e.printStackTrace();
>       return;
>     }
>
> Any suggestions where it might be faltering?
>
>
> On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>
>> Hello Jamal,
>>
>>    Please find my commands embedded below :
>>
>> Q1) How did putting those two lines solved the issue??
>> >> By adding those two resources you make sure that your code looks for
>> the input path inside HDFS, which would otherwise look for it in the local
>> FS by default. The files core-site.xml and hdfs-site.xml tell your code
>> where to go for NN and DN.
>>
>> Q2)
>>   I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>>  ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>> >>If you are able to compile your code properly then there is no problem
>> with the third party libraries which you are using. It looks like to me
>> that your code doesn't have the proper info about the intermediate path.
>> Please make sure you have told your code the exact location of intermediate
>> output.
>>
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>>
>> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>   So I am still in process of learning hadoop.
>>> I tried to run wordcount.java (by writing my own mapper reducer..
>>> creating jar and then running it in a pseudo distributed mode).
>>>
>>> At that time I got an error, something like
>>>  ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>>> So I googled around.. and found that I should put the following two
>>> lines on my driver code:
>>>
>>>     conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>>     conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>>> file"));
>>>
>>> (path of where my core-site and hdfs-site are) and after that it ran
>>> just fine.
>>>
>>> Q1) How did putting those two lines solved the issue??
>>> Q2)
>>>   I am now using third party libraries which are taking input from hdfs
>>> and writing output to hdfs...
>>> But in an intermediatory step, it creates a raw output.. I am again
>>> getting the error:
>>>  ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>>> How do I resolve this?
>>>
>>> Any suggestions.
>>> THanks
>>> Jamal.
>>>
>>
>>
>