Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> hive multiple inserts


Copy link to this message
-
Re: hive multiple inserts
hi,

I've tested branch 0.5 rev 903141. The sql runs as expected. Thx.
I can't compile trunk version, so haven't test it.

2010/1/12 Ning Zhang <[EMAIL PROTECTED]>

> HIVE-1039 is created for the bug when inserting to multiple local
> directories.
>
> Thanks,
> Ning
> On Jan 11, 2010, at 11:05 AM, Zheng Shao wrote:
>
> > For your second question, currently we can do it with a little extra
> work:
> > 1. Create an external table on the target directory with the field
> > delimiter you want;
> > 2. Run the query and insert overwrite the target external table.
> >
> > For the first question we can also do the similar thing (create a
> > bunch of external table and then insert), but I think we should fix
> > the problem.
> >
> > Zheng
> >
> > On Mon, Jan 11, 2010 at 8:31 AM, Anty <[EMAIL PROTECTED]> wrote:
> >> HI:
> >>    I came across the same problean, therein is no data.I have one
> >> more question,can i specify the field delimiter for the output
> >> file,not just the default ctrl-a field delimiter?
> >>
> >> On Fri, Jan 8, 2010 at 2:23 PM, wd <[EMAIL PROTECTED]> wrote:
> >>> hi,
> >>>
> >>> I'v tried use hive svn version, seems this bug still exists.
> >>>
> >>> svn st -v
> >>>
> >>>           896805   896744 namit        .
> >>>            896805   894292 namit        eclipse-templates
> >>>            896805   894292 namit        eclipse-templates/.classpath
> >>>            896805   765509 zshao
> >>> eclipse-templates/TestHive.launchtemplate
> >>>            896805   765509 zshao
>  eclipse-templates/TestMTQueries.l
> >>>            ..........
> >>>
> >>> svn reversion 896805 ?
> >>>
> >>> follows is the execute log.
> >>>
> >>> hive> from
> >>> test
> >>>     > INSERT OVERWRITE LOCAL DIRECTORY '/home/stefdong/tmp/0' select *
> where
> >>> a = 1
> >>>     > INSERT OVERWRITE LOCAL DIRECTORY '/home/stefdong/tmp/1' select *
> where
> >>> a = 3;
> >>> Total MapReduce jobs = 1
> >>> Launching Job 1 out of 1
> >>> Number of reduce tasks is set to 0 since there's no reduce operator
> >>> Starting Job = job_201001071716_4691, Tracking URL > >>> http://abc.com:50030/jobdetails.jsp?jobid=job_201001071716_4691
> >>> Kill Command = hadoop job  -Dmapred.job.tracker=abc.com:9001 -kill
> >>> job_201001071716_4691
> >>> 2010-01-08 14:14:55,442 Stage-2 map = 0%,  reduce = 0%
> >>> 2010-01-08 14:15:00,643 Stage-2 map = 100%,  reduce = 0%
> >>> Ended Job = job_201001071716_4691
> >>> Copying data to local directory /home/stefdong/tmp/0
> >>> Copying data to local directory /home/stefdong/tmp/0
> >>> 13 Rows loaded to /home/stefdong/tmp/0
> >>> 9 Rows loaded to /home/stefdong/tmp/1
> >>> OK
> >>> Time taken: 9.409 seconds
> >>>
> >>>
> >>> thx.
> >>>
> >>> 2010/1/6 wd <[EMAIL PROTECTED]>
> >>>>
> >>>> hi,
> >>>>
> >>>> Single insert can extract data into '/tmp/out/1'.I even can see "xxx
> rows
> >>>> loaded to '/tmp/out/0', xxx rows loaded to '/tmp/out/1'...etc in multi
> >>>> inserts, but there is no data in fact.
> >>>>
> >>>> Havn't try svn revision, will try it today.thx.
> >>>>
> >>>> 2010/1/5 Zheng Shao <[EMAIL PROTECTED]>
> >>>>>
> >>>>> Looks like a bug.
> >>>>> What is the svn revision of Hive?
> >>>>>
> >>>>> Did you verify that single insert into '/tmp/out/1' produces
> non-empty
> >>>>> files?
> >>>>>
> >>>>> Zheng
> >>>>>
> >>>>> On Tue, Jan 5, 2010 at 12:51 AM, wd <[EMAIL PROTECTED]> wrote:
> >>>>>> In hive wiki:
> >>>>>>
> >>>>>> Hive extension (multiple inserts):
> >>>>>> FROM from_statement
> >>>>>> INSERT OVERWRITE [LOCAL] DIRECTORY directory1 select_statement1
> >>>>>>
> >>>>>> [INSERT OVERWRITE [LOCAL] DIRECTORY directory2 select_statement2]
> ...
> >>>>>>
> >>>>>> I'm try to use hive multi inserts to extract data from hive to local
> >>>>>> disk.
> >>>>>> Follows is the hql
> >>>>>>
> >>>>>> from test_tbl
> >>>>>> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/out/0' select select * where
> >>>>>> id%10=0
> >>>>>> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/out/1' select select * where
> >>>>>> id%10=1
> >>>>>> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/out/2' select select * where