Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem


Copy link to this message
-
Re: Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem


> On Jan. 16, 2013, 12:21 a.m., Hari Shreedharan wrote:
> > Brock,
> >
> > Looks good in general. One question I have is if we could just do without a flume flag. Rather check the Configuration object to see if it is set to write to local file system. Does that make sense?

I thought about that as well. The reason I did not do that is because file:/// implements ChecksumFileSystem so data you read/write is checksummed. We could probably change it and no one would notice but it's possible someone is using this feature.... Thoughts?
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8964/#review15361
-----------------------------------------------------------
On Jan. 15, 2013, 10:33 p.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8964/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 10:33 p.m.)
>
>
> Review request for Flume, Juhani Connolly and Hari Shreedharan.
>
>
> Description
> -------
>
> Add's flag to HDFSEventSink Serializer's to use raw local file system.
>
>
> This addresses bug FLUME-1844.
>     https://issues.apache.org/jira/browse/FLUME-1844
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java f14f7cb
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 18fe6d4
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java bd40a88
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1e6d68f
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 1035ac3
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8964/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
>
>
> Thanks,
>
> Brock Noland
>
>