Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> First import into new partition disappears


Copy link to this message
-
Re: First import into new partition disappears
You can try "svn info" in the directory to get detailed information.

On Tue, Dec 29, 2009 at 11:02 PM, Saurabh Nanda <[EMAIL PROTECTED]>wrote:

> I reverted back to the old build and the same query is working fine now.
> How do I find out the SVN revision of the old build?
>
> Saurabh.
>
>
> On Wed, Dec 30, 2009 at 12:24 PM, Saurabh Nanda <[EMAIL PROTECTED]>wrote:
>
>> The rate at which "Map input bytes" and "Map input records" is growing is
>> extremely slow. Is something wrong with the HDFS configuration? But, it was
>> working perfectly fine with the previous Hive version.
>>
>> Saurabh.
>>
>>
>> On Wed, Dec 30, 2009 at 12:20 PM, Saurabh Nanda <[EMAIL PROTECTED]>wrote:
>>
>>> Attached are the task logs of one of the tasks.
>>>
>>> Saurabh.
>>>
>>>
>>> On Wed, Dec 30, 2009 at 12:16 PM, Zheng Shao <[EMAIL PROTECTED]> wrote:
>>>
>>>> This should be compiled into a single map-only job.
>>>> Can you take a look at the progress and the task logs of the job?
>>>>
>>>> We are not aware of any changes that might cause this problem.
>>>>
>>>> Zheng
>>>>
>>>> On Tue, Dec 29, 2009 at 10:35 PM, Saurabh Nanda <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > Picking up data from the 'raw' table, filtering the unwanted lines and
>>>> > inserting into 'raw_compressed' table which is stored as sequencefile:
>>>> >
>>>> > insert overwrite table raw_compressed partition(dt='2009-04-01')
>>>> select line
>>>> > from raw where dt='2009-04-01' and lower(line) rlike '.*get
>>>> > .*/confirmation.*http.*' and not lower(line) rlike
>>>> >
>>>> '(/images.*?|/styles.*?|/javascripts.*?|/adserver.*?|.*?favicon.*?|/includes/thwarte-logo.html.*)';
>>>> >
>>>> > Saurabh.
>>>> >
>>>> > On Wed, Dec 30, 2009 at 11:59 AM, Zheng Shao <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>
>>>> >> What is the import query? Do you mean "load data"?
>>>> >> Can you give an example?
>>>> >>
>>>> >> Zheng
>>>> >>
>>>> >> On Tue, Dec 29, 2009 at 10:22 PM, Saurabh Nanda <
>>>> [EMAIL PROTECTED]>
>>>> >> wrote:
>>>> >> > Also has something changed drastically in Hive over the last 2-3
>>>> months?
>>>> >> > A
>>>> >> > simply import query seems to be taking forever now!
>>>> >> >
>>>> >> > Saurabh.
>>>> >> >
>>>> >> > On Wed, Dec 30, 2009 at 11:48 AM, Saurabh Nanda <
>>>> [EMAIL PROTECTED]>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> I'm taking a look at the HDFS directories through the web
>>>> interface and
>>>> >> >> I
>>>> >> >> can see only 5 files there, not 6. I tried creating the partition
>>>> using
>>>> >> >> the
>>>> >> >> ADD PARTITION command. After that all 6 files get imported
>>>> >> >> successfully.
>>>> >> >>
>>>> >> >> Saurabh.
>>>> >> >>
>>>> >> >> On Wed, Dec 30, 2009 at 11:39 AM, Zheng Shao <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >> >>>
>>>> >> >>> Can you list the HDFS directories? Are the files in the
>>>> corresponding
>>>> >> >>> directories yet?
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> Zheng
>>>> >> >>>
>>>> >> >>> On Tue, Dec 29, 2009 at 9:57 PM, Saurabh Nanda
>>>> >> >>> <[EMAIL PROTECTED]>
>>>> >> >>> wrote:
>>>> >> >>> > Hi,
>>>> >> >>> >
>>>> >> >>> > I'm revisiting Hive after a long hiatus, so I may not be aware
>>>> of
>>>> >> >>> > any
>>>> >> >>> > new
>>>> >> >>> > developments. I had written a script some time back to import
>>>> >> >>> > webserver
>>>> >> >>> > logs
>>>> >> >>> > for a day into a new partition. The same script now running on
>>>> the
>>>> >> >>> > latest
>>>> >> >>> > version of Hive (r894548 compiled off trunk) seems to be
>>>> >> >>> > misbehaving.
>>>> >> >>> >
>>>> >> >>> > I'm importing about 6 files into each partition. However, after
>>>> the
>>>> >> >>> > script
>>>> >> >>> > ends, only 5 files show up in each partition. Do I need to
>>>> >> >>> > explicitly
>>>> >> >>> > issue
>>>> >> >>> > the ADD PARTITION command before loading data? Isn't the
>>>> partition
>>>> >> >>> > implicitly created?
>>>> >> >>> >
>>>> >> >>> > Saurabh.
>>>> >> >>> > --
>>>> >> >>> > http://nandz.blogspot.com
>>>> >> >>> > http://foodieforlife.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB