Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> First import into new partition disappears


Copy link to this message
-
Re: First import into new partition disappears
You can try "svn info" in the directory to get detailed information.

On Tue, Dec 29, 2009 at 11:02 PM, Saurabh Nanda <[EMAIL PROTECTED]>wrote:

> I reverted back to the old build and the same query is working fine now.
> How do I find out the SVN revision of the old build?
>
> Saurabh.
>
>
> On Wed, Dec 30, 2009 at 12:24 PM, Saurabh Nanda <[EMAIL PROTECTED]>wrote:
>
>> The rate at which "Map input bytes" and "Map input records" is growing is
>> extremely slow. Is something wrong with the HDFS configuration? But, it was
>> working perfectly fine with the previous Hive version.
>>
>> Saurabh.
>>
>>
>> On Wed, Dec 30, 2009 at 12:20 PM, Saurabh Nanda <[EMAIL PROTECTED]>wrote:
>>
>>> Attached are the task logs of one of the tasks.
>>>
>>> Saurabh.
>>>
>>>
>>> On Wed, Dec 30, 2009 at 12:16 PM, Zheng Shao <[EMAIL PROTECTED]> wrote:
>>>
>>>> This should be compiled into a single map-only job.
>>>> Can you take a look at the progress and the task logs of the job?
>>>>
>>>> We are not aware of any changes that might cause this problem.
>>>>
>>>> Zheng
>>>>
>>>> On Tue, Dec 29, 2009 at 10:35 PM, Saurabh Nanda <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > Picking up data from the 'raw' table, filtering the unwanted lines and
>>>> > inserting into 'raw_compressed' table which is stored as sequencefile:
>>>> >
>>>> > insert overwrite table raw_compressed partition(dt='2009-04-01')
>>>> select line
>>>> > from raw where dt='2009-04-01' and lower(line) rlike '.*get
>>>> > .*/confirmation.*http.*' and not lower(line) rlike
>>>> >
>>>> '(/images.*?|/styles.*?|/javascripts.*?|/adserver.*?|.*?favicon.*?|/includes/thwarte-logo.html.*)';
>>>> >
>>>> > Saurabh.
>>>> >
>>>> > On Wed, Dec 30, 2009 at 11:59 AM, Zheng Shao <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>
>>>> >> What is the import query? Do you mean "load data"?
>>>> >> Can you give an example?
>>>> >>
>>>> >> Zheng
>>>> >>
>>>> >> On Tue, Dec 29, 2009 at 10:22 PM, Saurabh Nanda <
>>>> [EMAIL PROTECTED]>
>>>> >> wrote:
>>>> >> > Also has something changed drastically in Hive over the last 2-3
>>>> months?
>>>> >> > A
>>>> >> > simply import query seems to be taking forever now!
>>>> >> >
>>>> >> > Saurabh.
>>>> >> >
>>>> >> > On Wed, Dec 30, 2009 at 11:48 AM, Saurabh Nanda <
>>>> [EMAIL PROTECTED]>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> I'm taking a look at the HDFS directories through the web
>>>> interface and
>>>> >> >> I
>>>> >> >> can see only 5 files there, not 6. I tried creating the partition
>>>> using
>>>> >> >> the
>>>> >> >> ADD PARTITION command. After that all 6 files get imported
>>>> >> >> successfully.
>>>> >> >>
>>>> >> >> Saurabh.
>>>> >> >>
>>>> >> >> On Wed, Dec 30, 2009 at 11:39 AM, Zheng Shao <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >> >>>
>>>> >> >>> Can you list the HDFS directories? Are the files in the
>>>> corresponding
>>>> >> >>> directories yet?
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> Zheng
>>>> >> >>>
>>>> >> >>> On Tue, Dec 29, 2009 at 9:57 PM, Saurabh Nanda
>>>> >> >>> <[EMAIL PROTECTED]>
>>>> >> >>> wrote:
>>>> >> >>> > Hi,
>>>> >> >>> >
>>>> >> >>> > I'm revisiting Hive after a long hiatus, so I may not be aware
>>>> of
>>>> >> >>> > any
>>>> >> >>> > new
>>>> >> >>> > developments. I had written a script some time back to import
>>>> >> >>> > webserver
>>>> >> >>> > logs
>>>> >> >>> > for a day into a new partition. The same script now running on
>>>> the
>>>> >> >>> > latest
>>>> >> >>> > version of Hive (r894548 compiled off trunk) seems to be
>>>> >> >>> > misbehaving.
>>>> >> >>> >
>>>> >> >>> > I'm importing about 6 files into each partition. However, after
>>>> the
>>>> >> >>> > script
>>>> >> >>> > ends, only 5 files show up in each partition. Do I need to
>>>> >> >>> > explicitly
>>>> >> >>> > issue
>>>> >> >>> > the ADD PARTITION command before loading data? Isn't the
>>>> partition
>>>> >> >>> > implicitly created?
>>>> >> >>> >
>>>> >> >>> > Saurabh.
>>>> >> >>> > --
>>>> >> >>> > http://nandz.blogspot.com
>>>> >> >>> > http://foodieforlife.blogspot.com