Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> How to load a web log file (text format) to Hive with compression


Copy link to this message
-
Re: How to load a web log file (text format) to Hive with compression
Hive has built-in support for gzipped text files so you don't need to do
anything special with them (meaning you just load them like regular files).
However, there is one important limitation with this approach these files
are not splittable. Depending on the size of your log files and how many of
them you have perday that may not be a problem. The other, slightly more
complicated approach is to use sequence files, which support splittable
compression. You can find more details here:
https://cwiki.apache.org/confluence/display/Hive/CompressedStorage

On Mon, Nov 11, 2013 at 5:09 PM, Raj Hadoop <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a web log files (text format). I want to load these files to a Hive
> table in compressed format. How do I do it ?
>
> Should I compress the text file (using any Linux utilities) and then
> create the Hive table?
>
> Can any one provide me the Hive syntax for loading the compressed file?
>
> Thanks,
> Raj
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB