Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop Binary File

Copy link to this message
Re: Hadoop Binary File
On Jan 25, 2011, at 21:47 , F.Ozgur Catak wrote:

> Can you give me a simple example/source code for this project.
> On Tue, Jan 25, 2011 at 10:13 PM, Keith Wiley <[EMAIL PROTECTED]> wrote:
>> I'm also doing binary image processing on Hadoop.  Where relevant, my Key
>> and Value types are a WritableComparable class of my own creation which
>> contains as members a BytesWritable object, obviously read from the file
>> itself directly into memory.  I also keep the path in my class so I know
>> where the file came from later.
>> On Jan 25, 2011, at 11:46 , F.Ozgur Catak wrote:
>>> Hi,
>>> I'm trying to develop an image processing application with hadoop. All
>> image
>>> files are in HDFS.  But I don't know how to read this files with
>> binary/byte
>>> stream. What is correct decleration of Mapper<K,V,K,V> and
>> Reducer<K,V,K,V>
>>> Class.

Hmmm, "simple" you say...not even remotely.  Our system has grown into quite the behemoth.

Let's see...here's my Writable class that I use to pass images around Hadoop as keys and values:

public class FileWritable
implements WritableComparable<FileWritable> {

private Path filePath_ = null;
private BytesWritable fileContents_;

public FileWritable() {
set(null, new BytesWritable());

public FileWritable(Path filePath, BytesWritable fileContents) {
set(filePath, fileContents);

public void set(Path filePath, BytesWritable fileContents) {
filePath_ = filePath;
fileContents_ = fileContents;

public Path getPath() {
return filePath_;

* The key is the filename, i.e., the last component of the full path
* @return
public Text getKey() {
return new Text(filePath_.getName());

public BytesWritable getContents() {
return fileContents_;

public void write(DataOutput out) throws IOException {
new Text(filePath_.getName()).write(out);

public void readFields(DataInput in) throws IOException {
Text filePath = new Text();
filePath_ = new Path(filePath.toString());


// If we ever use this class as a key, might want to do this a little better.
public int hashCode() {
return fileContents_.hashCode();

public boolean equals(Object o) {
if (o instanceof FileWritable) {
FileWritable f = (FileWritable) o;
//Is the second half of this comparison *really* necessary?!
return filePath_.equals(f.filePath_)
&& fileContents_.equals(f.fileContents_);
return false;

public int compareTo(FileWritable f) {
//Is the second half of this comparison *really* necessary?!
int cmp = filePath_.compareTo(f.filePath_);
if (cmp != 0)
return cmp;
return fileContents_.compareTo(f.fileContents_);

Does that help?

Keith Wiley               [EMAIL PROTECTED]               www.keithwiley.com

"Luminous beings are we, not this crude matter."
  -- Yoda