Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: Added new config 'shell' for Exec source


+
Roshan Naik 2012-10-26, 18:15
+
Mike Percy 2012-11-02, 10:02
+
Roshan Naik 2012-11-03, 00:01
+
Roshan Naik 2012-11-03, 02:31
+
Roshan Naik 2012-11-03, 00:02
+
Brock Noland 2012-11-03, 03:51
+
Roshan Naik 2012-11-06, 21:38
+
Roshan Naik 2012-11-06, 21:52
+
Alexander Alten-Lorenz 2013-01-24, 10:43
+
Brock Noland 2013-01-28, 19:27
Copy link to this message
-
Re: Review Request: Added new config 'shell' for Exec source
Roshan Naik 2013-03-06, 06:50


> On Jan. 28, 2013, 7:27 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java, line 248
> > <https://reviews.apache.org/r/7748/diff/2/?file=186015#file186015line248>
> >
> >     nit: I understand the other member variables are not final, but since we are adding a new one, can we make this one final?

This cannot be final..  its value is set by the configure method.
> On Jan. 28, 2013, 7:27 p.m., Brock Noland wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java, line 58
> > <https://reviews.apache.org/r/7748/diff/2/?file=186017#file186017line58>
> >
> >     +1 for moving this to shared methods. However, I cannot think of any other tests except the HBase tests which use static stuff. I think it makes sense if the static stuff is heavy weight but these objects are light weight. Let's stick with Before and After here.
>  'it makes sense if the static stuff is heavy weight but these objects are light weight'
I am not clear what heavy/light weight has to do with this being static.

> Let's stick with Before and After here.
I assume you are suggesting moving things from @BeforeClass to  @Before. I dont see why we would want to de-optimize things by doing things multiple times when there is no need for it.
> On Jan. 28, 2013, 7:27 p.m., Brock Noland wrote:
> > flume-ng-core/src/test/resources/test_command.txt, line 3
> > <https://reviews.apache.org/r/7748/diff/2/?file=186018#file186018line3>
> >
> >     Let's add tests to test shell arithmetic and also the more modern $( ) proces substitution.

Brock these are arithmetic and $( ) features are very specific to certain shells. Although i dont mind adding more tests where needed .. it seems futile to do these shell specific feature testing.

The specified shell will process the whole command/script.  If either $( ) or  shell arithmetic does not work.. we have a case for filing a bug against that particular shell or maybe upgrading bash to a newer version.

On a side note I have validated that this mechanism works on Windows with powershell too.
- Roshan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7748/#review15760
-----------------------------------------------------------
On Nov. 6, 2012, 9:52 p.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7748/
> -----------------------------------------------------------
>
> (Updated Nov. 6, 2012, 9:52 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Added new optional config directive 'shell' for Exec Source. One can specify how to invoke a shell to run the command (e.g. /bin/sh -c) . This is only needed for commands that use features like wildcards, backticks, pipes, etc that are supported by the shell.
>
>
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1661.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1661
>
>
> Diffs
> -----
>
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
>   flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
>   flume-ng-core/src/test/resources/test_command.txt PRE-CREATION
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 3af400e
>   pom.xml 363c0e4
>
> Diff: https://reviews.apache.org/r/7748/diff/
>
>
> Testing
> -------
>
> Have added a few test cases for both simple and more complex commands.
>
>
> Thanks,
>
> Roshan Naik
>
>

+
Brock Noland 2013-03-06, 12:55
+
Roshan Naik 2013-03-06, 21:08
+
Brock Noland 2013-03-06, 23:03
+
Roshan Naik 2013-03-07, 19:25
+
Brock Noland 2013-03-07, 20:45
+
Brock Noland 2013-03-11, 19:17
+
Jarek Cecho 2012-10-26, 18:58
+
Roshan Naik 2012-11-03, 00:03