Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Number of records in an HDFS file


+
Mix Nin 2013-05-13, 18:07
Copy link to this message
-
Re: Number of records in an HDFS file
Rahul Bhattacharjee 2013-05-13, 18:13
How about the second approach , get the application/job id which the pig
creates and submits to cluster and then find the job output counter for
that job from the JT.

Thanks,
Rahul
On Mon, May 13, 2013 at 11:37 PM, Mix Nin <[EMAIL PROTECTED]> wrote:

> It is a text file.
>
> If we want to use wc, we need to copy file from HDFS and then use wc, and
> this may take time. Is there a way without copying file from HDFS to local
> directory?
>
> Thanks
>
>
> On Mon, May 13, 2013 at 11:04 AM, Rahul Bhattacharjee <
> [EMAIL PROTECTED]> wrote:
>
>> few pointers.
>>
>> what kind of files are we talking about. for text you can use wc , for
>> avro data files you can use avro-tools.
>>
>> or get the job that pig is generating , get the counters for that job
>> from the jt of your hadoop cluster.
>>
>> Thanks,
>>  Rahul
>>
>>
>> On Mon, May 13, 2013 at 11:21 PM, Mix Nin <[EMAIL PROTECTED]> wrote:
>>
>>> Hello,
>>>
>>> What is the bets way to get the count of records in an HDFS file
>>> generated by a PIG script.
>>>
>>> Thanks
>>>
>>>
>>
>
+
Mix Nin 2013-05-13, 18:16
+
Rahul Bhattacharjee 2013-05-13, 18:26
+
Shahab Yunus 2013-05-13, 18:27
+
Mohammad Tariq 2013-05-13, 18:36
+
Mix Nin 2013-05-13, 18:58
+
Mohammad Tariq 2013-05-13, 19:07