Hi, I am novice in Hadoop. Kindly suggest how do we load log files into hdfs.
Please suggest the command and steps.
Thanks in advance!!
--
View this message in context: http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Hi,

Please read http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html
to get started with using HDFS.

To answer your question specifically though, use the "copyFromLocal"
or the "put" commands with the "hadoop" (0.20/1.0+) or "hdfs"
(0.22/0.23/2.0+) executables, as detailed here:
http://hadoop.apache.org/common/docs/current/file_system_shell.html#copyFromLocal

hadoop fs -put <local log file path> <path it should go to on the HDFS>

On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]> wrote:
>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into hdfs.
> Please suggest the command and steps.
> Thanks in advance!!
> --
> View this message in context: http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>

--
Harsh J

Hi
To load any file from local
Command:
 syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
   Example hadoop fs -copyFromLocal input/logs
hdfs://localhost/user/dataset/

  More Commans: http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]> wrote:

>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> hdfs.
> Please suggest the command and steps.
> Thanks in advance!!
> --
> View this message in context:
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

U can even use put/copyFromLocal

both are similar and does the job via terminal.

Or u can write a simple client program to do the job :)

Amith
________________________________________
From: samir das mohapatra [[EMAIL PROTECTED]]
Sent: Sunday, May 13, 2012 9:13 PM
To: [EMAIL PROTECTED]
Subject: Re: How to load raw log file into HDFS?

Hi
To load any file from local
Command:
 syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
   Example hadoop fs -copyFromLocal input/logs
hdfs://localhost/user/dataset/

  More Commans: http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]> wrote:

>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> hdfs.
> Please suggest the command and steps.
> Thanks in advance!!
> --
> View this message in context:
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Hi,

the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
http://www.manning.com/lam/)
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:
https://github.com/cloudera/flume/wiki

--
BR

Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org
On Mon, May 14, 2012 at 7:24 AM, Amith D K <[EMAIL PROTECTED]> wrote:

> U can even use put/copyFromLocal
>
> both are similar and does the job via terminal.
>
> Or u can write a simple client program to do the job :)
>
> Amith
>
>
> ________________________________________
> From: samir das mohapatra [[EMAIL PROTECTED]]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: [EMAIL PROTECTED]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>
>
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]>
> wrote:
>
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>

If you are a novice I'd suggest using a visual design tool like Pentaho
Kettle   http://wiki.pentaho.com/display/BAD/Loading+Data+into+HDFS  How To
Load Data into HDFS

AnExplorer wrote:
>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> hdfs. Please suggest the command and steps.
> Thanks in advance!!
>

--
View this message in context: http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33832683.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>.
Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
How do I know what <HDFS_FILE_PATH> are available?

-----Original Message-----
From: Alexander Fahlke [mailto:[EMAIL PROTECTED]]
Sent: Monday, May 14, 2012 1:53 AM
To: [EMAIL PROTECTED]
Subject: Re: How to load raw log file into HDFS?

Hi,

the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
http://www.manning.com/lam/)
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:
https://github.com/cloudera/flume/wiki

--
BR

Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org
On Mon, May 14, 2012 at 7:24 AM, Amith D K <[EMAIL PROTECTED]> wrote:

> U can even use put/copyFromLocal
>
> both are similar and does the job via terminal.
>
> Or u can write a simple client program to do the job :)
>
> Amith
>
>
> ________________________________________
> From: samir das mohapatra [[EMAIL PROTECTED]]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: [EMAIL PROTECTED]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>
>
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]>
> wrote:
>
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>

This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.

You first need to copy data using copyFromLocal to your HDFS and then you can utilize PIG and Hive program for further analysis which run on map reduce. Yes warehouse directory is in HDFS. If you want to run(test) PIG in local then in that case you don't to copy data to HDFS
Sent from my BlackBerry, pls excuse typo

-----Original Message-----
From: Michael Wang <[EMAIL PROTECTED]>
Date: Mon, 14 May 2012 18:43:47
To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: RE: How to load raw log file into HDFS?

I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>.
Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
How do I know what <HDFS_FILE_PATH> are available?

-----Original Message-----
From: Alexander Fahlke [mailto:[EMAIL PROTECTED]]
Sent: Monday, May 14, 2012 1:53 AM
To: [EMAIL PROTECTED]
Subject: Re: How to load raw log file into HDFS?

Hi,

the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
http://www.manning.com/lam/)
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:
https://github.com/cloudera/flume/wiki

--
BR

Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org
On Mon, May 14, 2012 at 7:24 AM, Amith D K <[EMAIL PROTECTED]> wrote:

> U can even use put/copyFromLocal
>
> both are similar and does the job via terminal.
>
> Or u can write a simple client program to do the job :)
>
> Amith
>
>
> ________________________________________
> From: samir das mohapatra [[EMAIL PROTECTED]]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: [EMAIL PROTECTED]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>
>
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]>
> wrote:
>
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>

This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.

You can load data directly into a hive table(external and internal) directly from the local file system. The same stands for pig. To Manish's point you can do the same using hadoop fs commands. I have tried it both ways and have seen a difference in performance. I would be interested to hear from the rest of the community about this to see it is consistent with what they have seen.

Thanks,
Ranjith

On May 14, 2012, at 8:45 PM, "Manish Bhoge" <[EMAIL PROTECTED]> wrote:

> You first need to copy data using copyFromLocal to your HDFS and then you can utilize PIG and Hive program for further analysis which run on map reduce. Yes warehouse directory is in HDFS. If you want to run(test) PIG in local then in that case you don't to copy data to HDFS
> Sent from my BlackBerry, pls excuse typo
>
> -----Original Message-----
> From: Michael Wang <[EMAIL PROTECTED]>
> Date: Mon, 14 May 2012 18:43:47
> To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
> Reply-To: [EMAIL PROTECTED]
> Subject: RE: How to load raw log file into HDFS?
>
> I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
> I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>.
> Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
> How do I know what <HDFS_FILE_PATH> are available?
>
> -----Original Message-----
> From: Alexander Fahlke [mailto:[EMAIL PROTECTED]]
> Sent: Monday, May 14, 2012 1:53 AM
> To: [EMAIL PROTECTED]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi,
>
> the best would be to read the documentation and some books to get familar
> with Hadoop.
>
> One of my favourite books is "Hadoop in Action" from Manning (
> http://www.manning.com/lam/)
> This book has an exmple for putting (log)-files into HDFS. Check out the
> source "listing-3-1"
>
> Later you can also check out Cloudera's Flume:
> https://github.com/cloudera/flume/wiki
>
> --
> BR
>
> Alexander Fahlke
> Java Developer
> www.nurago.com | www.fahlke.org
>
>
> On Mon, May 14, 2012 at 7:24 AM, Amith D K <[EMAIL PROTECTED]> wrote:
>
>> U can even use put/copyFromLocal
>>
>> both are similar and does the job via terminal.
>>
>> Or u can write a simple client program to do the job :)
>>
>> Amith
>>
>>
>> ________________________________________
>> From: samir das mohapatra [[EMAIL PROTECTED]]
>> Sent: Sunday, May 13, 2012 9:13 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: How to load raw log file into HDFS?
>>
>> Hi
>> To load any file from local
>> Command:
>> syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>>  Example hadoop fs -copyFromLocal input/logs
>> hdfs://localhost/user/dataset/
>>
>> More Commans:
>> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>>
>>
>> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[EMAIL PROTECTED]>
>> wrote:
>>
>>>
>>> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
>>> hdfs.
>>> Please suggest the command and steps.
>>> Thanks in advance!!
>>> --
>>> View this message in context:
>>>
>> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
>>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>>
>>>
>>
>
> This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.