Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> sqoop, hive and lzo and cdh3u3 - not creating in index automatically


+
Chalcy Raja 2012-06-18, 13:16
Copy link to this message
-
Re: sqoop, hive and lzo and cdh3u3 - not creating in index automatically
Have you considered switching to sequence files using snappy
compression (or lzo). IIRC the process of generating LZO files and
then generating an index on top of these is cumbersome. When sequence
files are directly splittable.

On Mon, Jun 18, 2012 at 9:16 AM, Chalcy Raja
<[EMAIL PROTECTED]> wrote:
> I am posting it here first and then may be on sqoop user group as well.
>
>
>
> I am trying to use lzo compression.
>
>
>
> Tested on a standalone by installing cdh3u3 and did sqoop to hive import
> with lzo compression and everything works great. The data is sqooped into
> hdfs and lzo index file got created and data is in hive table.
>
>
>
> Did all the lzo necessary steps on the main cluster where the server already
> has cdh3u3 upgraded previously from cdh3u0 to cdh3u1 to cdh3u2 to cdh3u3.
> Did the same sqoop to hive with lzo compression.  Sqoop to hive works but
> lzo index is not getting created.
>
>
>
> Need expert opinion. What could be the reason for this behavior.  Compared
> all the versions of hive, sqoop etc., and checked all the configuration.
> Looks like we are missing something.
>
>
>
> Thanks,
>
> Chalcy
>
>
>
>
+
Bejoy KS 2012-06-18, 14:03
+
Chalcy Raja 2012-06-18, 14:31
+
Bejoy KS 2012-06-18, 14:38
+
Chalcy Raja 2012-06-18, 17:46
+
Chalcy Raja 2012-06-18, 19:28
+
Chalcy Raja 2012-06-19, 01:32
+
Bejoy KS 2012-06-19, 06:59
+
Chalcy Raja 2012-06-19, 12:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB