Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: Added new config 'shell' for Exec source


Copy link to this message
-
Re: Review Request: Added new config 'shell' for Exec source
Brock Noland 2013-03-06, 12:55


> On Jan. 28, 2013, 7:27 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java, line 248
> > <https://reviews.apache.org/r/7748/diff/2/?file=186015#file186015line248>
> >
> >     nit: I understand the other member variables are not final, but since we are adding a new one, can we make this one final?
>
> Roshan Naik wrote:
>     This cannot be final..  its value is set by the configure method.

Hmm I might have highlighted the wrong one, I mean the member variable of ExecRunnable.
> On Jan. 28, 2013, 7:27 p.m., Brock Noland wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java, line 58
> > <https://reviews.apache.org/r/7748/diff/2/?file=186017#file186017line58>
> >
> >     +1 for moving this to shared methods. However, I cannot think of any other tests except the HBase tests which use static stuff. I think it makes sense if the static stuff is heavy weight but these objects are light weight. Let's stick with Before and After here.
>
> Roshan Naik wrote:
>    
>     >  'it makes sense if the static stuff is heavy weight but these objects are light weight'
>     I am not clear what heavy/light weight has to do with this being static.
>    
>     > Let's stick with Before and After here.
>     I assume you are suggesting moving things from @BeforeClass to  @Before. I dont see why we would want to de-optimize things by doing things multiple times when there is no need for it.

If it's static, it will only be initialized once so if it's heavy-weight that would speed up the tests considerably. However, these objects not heavy-weight and can be created in milliseconds so I prefer to not share them amongst the tests.
@Before and instance variables have the nice property that side affects from tests will not cause the tests to be in-determinate when running many tests in the same JVM.  Unless there is a proven performance issue I'd prefer to keep everything as instance variables so we don't see things like "if test B runs after A or by itself it's fine, but if it runs after C it fails."
> On Jan. 28, 2013, 7:27 p.m., Brock Noland wrote:
> > flume-ng-core/src/test/resources/test_command.txt, line 3
> > <https://reviews.apache.org/r/7748/diff/2/?file=186018#file186018line3>
> >
> >     Let's add tests to test shell arithmetic and also the more modern $( ) proces substitution.
>
> Roshan Naik wrote:
>     Brock these are arithmetic and $( ) features are very specific to certain shells. Although i dont mind adding more tests where needed .. it seems futile to do these shell specific feature testing.
>    
>     The specified shell will process the whole command/script.  If either $( ) or  shell arithmetic does not work.. we have a case for filing a bug against that particular shell or maybe upgrading bash to a newer version.
>    
>     On a side note I have validated that this mechanism works on Windows with powershell too.

I previously said I was not comfortable with this change so I'd like see many tests showing this works as possible. I'd also argue that it's not futile. When starting scripts via the -c mechanism it's almost always quoting, escaping, or command substitution that cause issues. Note that these features have been present in bash since 2004.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7748/#review15760
-----------------------------------------------------------
On Nov. 6, 2012, 9:52 p.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7748/
> -----------------------------------------------------------
>
> (Updated Nov. 6, 2012, 9:52 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Added new optional config directive 'shell' for Exec Source. One can specify how to invoke a shell to run the command (e.g. /bin/sh -c) . This is only needed for commands that use features like wildcards, backticks, pipes, etc that are supported by the shell.