Flume, mail # dev - Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Review Request: FLUME-1657 - Regex Extractor Interceptor

Cameron Gandevia 2012-10-22, 18:26

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/
-----------------------------------------------------------

Review request for Flume.
Description
-------

A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
Diffs
-----

  flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
  flume-ng-doc/sphinx/FlumeUserGuide.rst bd244d4

Diff: https://reviews.apache.org/r/7700/diff/
Testing
-------
Thanks,

Cameron Gandevia

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-06, 17:03

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13155
-----------------------------------------------------------

flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java
<https://reviews.apache.org/r/7700/#comment28310>

I think we should be passing an explicit Charset here like UTF8?

flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java
<https://reviews.apache.org/r/7700/#comment28311>

If serializerString had ,, then splitNameAndSerializer[0].trim(); would throw an index out of bounds correct?
- Brock Noland
On Oct. 22, 2012, 6:26 p.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2012, 6:26 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst bd244d4
>
> Diff: https://reviews.apache.org/r/7700/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cameron Gandevia
>
>

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-06, 17:05

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13156
-----------------------------------------------------------

flume-ng-doc/sphinx/FlumeUserGuide.rst
<https://reviews.apache.org/r/7700/#comment28312>

Generally in flume when we are going to have multiple sub property items we give them a name and then have sub-properties under that name. Thoughts on doing that hear as opposed to using the class name?
- Brock Noland
On Oct. 22, 2012, 6:26 p.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2012, 6:26 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst bd244d4
>
> Diff: https://reviews.apache.org/r/7700/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cameron Gandevia
>
>

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Cameron Gandevia 2012-11-14, 01:24

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-14, 01:57

> On Nov. 6, 2012, 5:05 p.m., Brock Noland wrote:
> > flume-ng-doc/sphinx/FlumeUserGuide.rst, line 1824
> > <https://reviews.apache.org/r/7700/diff/1/?file=178736#file178736line1824>
> >
> > Generally in flume when we are going to have multiple sub property items we give them a name and then have sub-properties under that name. Thoughts on doing that hear as opposed to using the class name?
>
> Cameron Gandevia wrote:
> The idea I saw was allowing anyone to provide their own serializer implementation outside of the ones provided by the flume project, maybe I am not following how this would be done using sub property names.

Yes I think that is great a great idea!

So what I was thinking is that this:

agent.sources.r1.channels = c1
agent.sources.r1.type = SEQ
agent.sources.r1.interceptors = i1
agent.sources.r1.interceptors.i1.type = REGEX_EXTRACTOR
agent.sources.r1.interceptors.i1.regex = (WARNING)|(ERROR)|(FATAL)
agent.sources.r1.interceptors.i1.serializer = warning:com.blah.SomeSerializer,error,fatal:org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
agent.sources.r1.interceptors.i1.org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer.dateFormat = yyyy-MM-dd

becomes something approximately like this:

agent.sources.r1.channels = c1
agent.sources.r1.type = SEQ
agent.sources.r1.interceptors = i1
agent.sources.r1.interceptors.i1.type = REGEX_EXTRACTOR
agent.sources.r1.interceptors.i1.regex = (WARNING)|(ERROR)|(FATAL)
agent.sources.r1.interceptors.i1.serializers = warning:s1,error,fatal:s1
agent.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
agent.sources.r1.interceptors.i1.serializers.s1.dateFormat = yyyy-MM-dd
agent.sources.r1.interceptors.i1.serializers.s2.type = com.blah.SomeSerializer

That is how other plugabble components are configured. Then s1.* can be passed to the a configure method in the serializer and the plugin can do it's own configuration.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13156
-----------------------------------------------------------
On Nov. 14, 2012, 1:24 a.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2012, 1:24 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0596473
> flume-ng-doc/sphinx/FlumeUserGuide.rst.orig PRE-CREATION
>
> Diff: https://reviews.apache.org/r/7700/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cameron Gandevia
>
>

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Hari Shreedharan 2012-11-14, 02:15

> On Nov. 6, 2012, 5:05 p.m., Brock Noland wrote:
> > flume-ng-doc/sphinx/FlumeUserGuide.rst, line 1824
> > <https://reviews.apache.org/r/7700/diff/1/?file=178736#file178736line1824>
> >
> > Generally in flume when we are going to have multiple sub property items we give them a name and then have sub-properties under that name. Thoughts on doing that hear as opposed to using the class name?
>
> Cameron Gandevia wrote:
> The idea I saw was allowing anyone to provide their own serializer implementation outside of the ones provided by the flume project, maybe I am not following how this would be done using sub property names.
>
> Brock Noland wrote:
> Yes I think that is great a great idea!
>
> So what I was thinking is that this:
>
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> agent.sources.r1.interceptors = i1
> agent.sources.r1.interceptors.i1.type = REGEX_EXTRACTOR
> agent.sources.r1.interceptors.i1.regex = (WARNING)|(ERROR)|(FATAL)
> agent.sources.r1.interceptors.i1.serializer = warning:com.blah.SomeSerializer,error,fatal:org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
> agent.sources.r1.interceptors.i1.org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer.dateFormat = yyyy-MM-dd
>
> becomes something approximately like this:
>
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> agent.sources.r1.interceptors = i1
> agent.sources.r1.interceptors.i1.type = REGEX_EXTRACTOR
> agent.sources.r1.interceptors.i1.regex = (WARNING)|(ERROR)|(FATAL)
> agent.sources.r1.interceptors.i1.serializers = warning:s1,error,fatal:s1
> agent.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
> agent.sources.r1.interceptors.i1.serializers.s1.dateFormat = yyyy-MM-dd
> agent.sources.r1.interceptors.i1.serializers.s2.type = com.blah.SomeSerializer
>
> That is how other plugabble components are configured. Then s1.* can be passed to the a configure method in the serializer and the plugin can do it's own configuration.

I like Brock's idea. I'd also change this to be more uniform with other Flume components to something like this:
agent.sources.r1.interceptors.i1.serializers = s1 s2
agent.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
agent.sources.r1.interceptors.i1.serializers.s1.dateFormat = yyyy-MM-dd
agent.sources.r1.interceptors.i1.serializers.s1.severity = warning
agent.sources.r1.interceptors.i1.serializers.s2.type = com.blah.SomeSerializer
agent.sources.r1.interceptors.i1.serializers.s2.severity = error fatal
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13156
-----------------------------------------------------------
On Nov. 14, 2012, 1:24 a.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2012, 1:24 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-14, 20:55

> On Nov. 6, 2012, 5:05 p.m., Brock Noland wrote:
> > flume-ng-doc/sphinx/FlumeUserGuide.rst, line 1824
> > <https://reviews.apache.org/r/7700/diff/1/?file=178736#file178736line1824>
> >
> > Generally in flume when we are going to have multiple sub property items we give them a name and then have sub-properties under that name. Thoughts on doing that hear as opposed to using the class name?
>
> Cameron Gandevia wrote:
> The idea I saw was allowing anyone to provide their own serializer implementation outside of the ones provided by the flume project, maybe I am not following how this would be done using sub property names.
>
> Brock Noland wrote:
> Yes I think that is great a great idea!
>
> So what I was thinking is that this:
>
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> agent.sources.r1.interceptors = i1
> agent.sources.r1.interceptors.i1.type = REGEX_EXTRACTOR
> agent.sources.r1.interceptors.i1.regex = (WARNING)|(ERROR)|(FATAL)
> agent.sources.r1.interceptors.i1.serializer = warning:com.blah.SomeSerializer,error,fatal:org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
> agent.sources.r1.interceptors.i1.org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer.dateFormat = yyyy-MM-dd
>
> becomes something approximately like this:
>
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> agent.sources.r1.interceptors = i1
> agent.sources.r1.interceptors.i1.type = REGEX_EXTRACTOR
> agent.sources.r1.interceptors.i1.regex = (WARNING)|(ERROR)|(FATAL)
> agent.sources.r1.interceptors.i1.serializers = warning:s1,error,fatal:s1
> agent.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
> agent.sources.r1.interceptors.i1.serializers.s1.dateFormat = yyyy-MM-dd
> agent.sources.r1.interceptors.i1.serializers.s2.type = com.blah.SomeSerializer
>
> That is how other plugabble components are configured. Then s1.* can be passed to the a configure method in the serializer and the plugin can do it's own configuration.
>
> Hari Shreedharan wrote:
> I like Brock's idea. I'd also change this to be more uniform with other Flume components to something like this:
> agent.sources.r1.interceptors.i1.serializers = s1 s2
> agent.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorTimestampSerializer
> agent.sources.r1.interceptors.i1.serializers.s1.dateFormat = yyyy-MM-dd
> agent.sources.r1.interceptors.i1.serializers.s1.severity = warning
> agent.sources.r1.interceptors.i1.serializers.s2.type = com.blah.SomeSerializer
> agent.sources.r1.interceptors.i1.serializers.s2.severity = error fatal

+1
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13156
-----------------------------------------------------------
On Nov. 14, 2012, 1:24 a.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2012, 1:24 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Cameron Gandevia 2012-11-14, 01:24

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/
-----------------------------------------------------------

(Updated Nov. 14, 2012, 1:24 a.m.)
Review request for Flume.
Changes
-------

Code review comment fixes
Description
-------

A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
Diffs (updated)
-----

  flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
  flume-ng-doc/sphinx/FlumeUserGuide.rst 0596473
  flume-ng-doc/sphinx/FlumeUserGuide.rst.orig PRE-CREATION

Diff: https://reviews.apache.org/r/7700/diff/
Testing
-------
Thanks,

Cameron Gandevia

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-14, 01:57

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13410
-----------------------------------------------------------

flume-ng-doc/sphinx/FlumeUserGuide.rst.orig
<https://reviews.apache.org/r/7700/#comment28744>

Looks like this .orig file can be removed.
- Brock Noland
On Nov. 14, 2012, 1:24 a.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2012, 1:24 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0596473
> flume-ng-doc/sphinx/FlumeUserGuide.rst.orig PRE-CREATION
>
> Diff: https://reviews.apache.org/r/7700/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cameron Gandevia
>
>

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Cameron Gandevia 2012-11-14, 22:03

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/
-----------------------------------------------------------

(Updated Nov. 14, 2012, 10:03 p.m.)
Review request for Flume.
Changes
-------

-Code review changes
-Updated configuration to use sub-properties
Description
-------

A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
Diffs (updated)
-----

  flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
  flume-ng-doc/sphinx/FlumeUserGuide.rst 0596473

Diff: https://reviews.apache.org/r/7700/diff/
Testing
-------
Thanks,

Cameron Gandevia

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-15, 13:01

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13464
-----------------------------------------------------------
Great timing! (I am building flume 1.3.0-rc3 this morning) I will review this in a couple hours and hope to include it.

- Brock Noland
On Nov. 14, 2012, 10:03 p.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2012, 10:03 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0596473
>
> Diff: https://reviews.apache.org/r/7700/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cameron Gandevia
>
>

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-15, 15:35

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7700/#review13465
-----------------------------------------------------------

Ship it!
Ship It!

- Brock Noland
On Nov. 14, 2012, 10:03 p.m., Cameron Gandevia wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7700/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2012, 10:03 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> A RegexExtractor interceptor that will allow users to extract regex matches and append them as header fields of the event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java c478337
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexExtractorInterceptorSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorMillisSerializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0596473
>
> Diff: https://reviews.apache.org/r/7700/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cameron Gandevia
>
>

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Brock Noland 2012-11-15, 19:36

I had earlier planned on pulling this in earlier, but when it came
time to build the last RC's it was not ready. Since it's available
now, I don't see any reason not to pull it in. It's not a complex
feature and is not required for flume to operate correctly.

Brock

On Thu, Nov 15, 2012 at 1:31 PM, Roshan Naik <[EMAIL PROTECTED]> wrote:
> I am a bit concerned that new features are still being pulled into the 1.3
> release at this stage. is this normal ?
> -roshan
>
>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Re: Review Request: FLUME-1657 - Regex Extractor Interceptor

Hari Shreedharan 2012-11-15, 19:40

Roshan,

Core framework features are not usually pulled in at the last moment - and this is not one of them, this is only additional functionality. Also, this issue had been reviewed quite a while ago - and the only feedback in the reviews was to make the configuration more flexible - which was done, so it was pulled in (The same kind of configuration code appears all over the place in Flume). I think that is perfectly ok.
Thanks,
Hari

--
Hari Shreedharan
On Thursday, November 15, 2012 at 11:36 AM, Brock Noland wrote:

> I had earlier planned on pulling this in earlier, but when it came
> time to build the last RC's it was not ready. Since it's available
> now, I don't see any reason not to pull it in. It's not a complex
> feature and is not required for flume to operate correctly.
>
> Brock
>
> On Thu, Nov 15, 2012 at 1:31 PM, Roshan Naik <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I am a bit concerned that new features are still being pulled into the 1.3
> > release at this stage. is this normal ?
> > -roshan
> >
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>
>