Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Is there a loader that loads a file as a line?


Copy link to this message
-
Re: Is there a loader that loads a file as a line?
Hello Jonathan,
        Have a look at Hadoop's WholeFileInputFormat..Might fit into
your requirements.
Regards,
    Mohammad Tariq
On Fri, Jun 22, 2012 at 3:39 AM, Prashant Kommireddi
<[EMAIL PROTECTED]> wrote:
> I think you will need to implement a RecordReader/InputFormat of your own
> for this and use it with a LoadFunc. Not sure if Hadoop has a Reader that
> you could re-use for this.
>
> How do you handle the case when a file exceeds block size?
>
> On Thu, Jun 21, 2012 at 2:34 PM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:
>
>> It can even be a bytearray. Basically I have a bunch of files, and I want
>> one file -> one row. Is there an easy way to do this? Or will I need to
>> provide a special fileinputformat etc?
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB