Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> setting hadoop for pseudo distributed mode.


Copy link to this message
-
Re: setting hadoop for pseudo distributed mode.
what are those libraries and how are they reading data from HDFS? you were
trying with MR jobs if i'm not wrong? in order to perform read/write on
HDFS we need HDFS API with a Configuration object. how are you doing it
here?

Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Fri, Dec 28, 2012 at 2:38 AM, jamal sasha <[EMAIL PROTECTED]> wrote:

> Hi,
>   Thanks for throwing insight.
>
> So the code snippet looks like this
>
> String interout = final_output + "/intermediate";
>
>     try {
>       new CreateInterOutput().main(new String[] { input, interout });
>     } catch (Exception e) {
>       e.printStackTrace();
>       return;
>     }
>
>     try {
>     new CreateFinalOutput().main(new String[] { interout, final_output });
>
>     } catch (Exception e) {
>       e.printStackTrace();
>       return;
>     }
>
> Any suggestions where it might be faltering?
>
>
> On Thu, Dec 27, 2012 at 12:49 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>
>> Hello Jamal,
>>
>>    Please find my commands embedded below :
>>
>> Q1) How did putting those two lines solved the issue??
>> >> By adding those two resources you make sure that your code looks for
>> the input path inside HDFS, which would otherwise look for it in the local
>> FS by default. The files core-site.xml and hdfs-site.xml tell your code
>> where to go for NN and DN.
>>
>> Q2)
>>   I am now using third party libraries which are taking input from hdfs
>> and writing output to hdfs...
>> But in an intermediatory step, it creates a raw output.. I am again
>> getting the error:
>>  ERROR security.UserGroupInformation: PriviledgedActionException
>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>> How do I resolve this?
>> >>If you are able to compile your code properly then there is no problem
>> with the third party libraries which you are using. It looks like to me
>> that your code doesn't have the proper info about the intermediate path.
>> Please make sure you have told your code the exact location of intermediate
>> output.
>>
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>>
>> On Fri, Dec 28, 2012 at 1:33 AM, jamal sasha <[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>   So I am still in process of learning hadoop.
>>> I tried to run wordcount.java (by writing my own mapper reducer..
>>> creating jar and then running it in a pseudo distributed mode).
>>>
>>> At that time I got an error, something like
>>>  ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki
>>> So I googled around.. and found that I should put the following two
>>> lines on my driver code:
>>>
>>>     conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
>>>     conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml
>>> file"));
>>>
>>> (path of where my core-site and hdfs-site are) and after that it ran
>>> just fine.
>>>
>>> Q1) How did putting those two lines solved the issue??
>>> Q2)
>>>   I am now using third party libraries which are taking input from hdfs
>>> and writing output to hdfs...
>>> But in an intermediatory step, it creates a raw output.. I am again
>>> getting the error:
>>>  ERROR security.UserGroupInformation: PriviledgedActionException
>>> as:mhduser cause:org.apache.hadoop.mapred.InvalidInputException: Input path
>>> does not exist: hdfs://localhost:54310/user/hduser/wiki-inter-output
>>> How do I resolve this?
>>>
>>> Any suggestions.
>>> THanks
>>> Jamal.
>>>
>>
>>
>