Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - run pig explain command over the entire script in java


+
Petar Jovanovic 2013-02-18, 16:18
+
Jonathan Coveney 2013-02-18, 18:04
+
Petar Jovanovic 2013-02-18, 21:18
+
Jonathan Coveney 2013-02-18, 21:21
+
Petar Jovanovic 2013-02-18, 22:24
+
Jonathan Coveney 2013-02-18, 22:51
Copy link to this message
-
Re: run pig explain command over the entire script in java
Petar Jovanovic 2013-02-18, 23:27
I would like to do it from Java automatically and also for the scripts
with multiple stores.

On 2/18/2013 11:51 PM, Jonathan Coveney wrote:
> so what is the issue with just doing "explain final_alias;" ? Or do you
> want it to do explain on the final alias of any given script?
>
>
> 2013/2/18 Petar Jovanovic <[EMAIL PROTECTED]>
>
>> Well no. I want to do explain over the entire script as I show you in the
>> command I wrote.
>> That would produce the compact execution plan for the whole script and not
>> the several separate ones (one for each alias).
>>
>>
>>
>> On 2/18/2013 10:21 PM, Jonathan Coveney wrote:
>>
>>> I guess I'm confused at what you want then.
>>>
>>> So we have a script:
>>>
>>> a = ..
>>> b = ...
>>> c = ..
>>>
>>> my impression was that you essentially wanted to print "explain a; explain
>>> b; explain c;" is this not what you mean/
>>>
>>>
>>> 2013/2/18 Petar Jovanovic <[EMAIL PROTECTED]>
>>>
>>>   But pig already has this, it's jut that the java API seems not to have
>>>> it.
>>>> The point would be to have the execution flow from the entire script.
>>>> Indeed, the output is not exactly the same if the explain is called over
>>>> the entire script than if it is called for each query (alias).
>>>>
>>>> If from the shell we execute command
>>>>
>>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>>> explain-out9.txt'
>>>> it will not give the same output as if we did it for each operation
>>>> separately.
>>>>
>>>>
>>>> On 2/18/2013 7:04 PM, Jonathan Coveney wrote:
>>>>
>>>>   Hacky way: grep for "^\S =", pull out the names, and then do the
>>>>> explains.
>>>>>
>>>>> Why is doing the progressive explains useful? it wouldn't be too hard to
>>>>> build this into pig but the results would be pretty unwieldy, it'd be
>>>>> really big, and pretty redundant.
>>>>>
>>>>>
>>>>> 2013/2/18 Petar Jovanovic <[EMAIL PROTECTED]>
>>>>>
>>>>>    Hi,
>>>>>
>>>>>> I am trying to find the way to run the explain command over the entire
>>>>>> pig
>>>>>> script in java. I was using PigServer but it offers only to do explain
>>>>>> over
>>>>>> the single query (alias) not the entire script. Is there a way to do
>>>>>> something like:
>>>>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>>>>> explain-out9.txt'
>>>>>> but from my Java code?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>