Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: Wrapping around BitSet with the Writable interface


Copy link to this message
-
Re: Wrapping around BitSet with the Writable interface
In order to make the code more readable, you could start by using the
methods toByteArray() and valueOf(bytes)

http://docs.oracle.com/javase/7/docs/api/java/util/BitSet.html#toByteArray%28%29
http://docs.oracle.com/javase/7/docs/api/java/util/BitSet.html#valueOf%28byte[]%29

Regards

Bertrand
On Sun, May 12, 2013 at 8:24 PM, Jim Twensky <[EMAIL PROTECTED]> wrote:

> I have large java.util.BitSet objects that I want to bitwise-OR using a
> MapReduce job. I decided to wrap around each object using the Writable
> interface. Right now I convert each BitSet to a byte array and serialize
> the byte array on disk.
>
> Converting them to byte arrays is a bit inefficient but I could not find a
> work around to write them directly to the DataOutput. Is there a way to
> skip this and serialize the object directly? Here is what my current
> implementation looks like:
>
> public class BitSetWritable implements Writable {
>
>   private BitSet bs;
>
>   public BitSetWritable() {
>     this.bs = new BitSet();
>   }
>
>   @Override
>   public void write(DataOutput out) throws IOException {
>
>     ByteArrayOutputStream bos = new ByteArrayOutputStream(bs.size()/8);
>     ObjectOutputStream oos = new ObjectOutputStream(bos);
>     oos.writeObject(bs);
>     byte[] bytes = bos.toByteArray();
>     oos.close();
>     out.writeInt(bytes.length);
>     out.write(bytes);
>
>   }
>
>   @Override
>   public void readFields(DataInput in) throws IOException {
>
>     int len = in.readInt();
>     byte[] bytes = new byte[len];
>     in.readFully(bytes);
>
>     ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
>     ObjectInputStream ois = new ObjectInputStream(bis);
>     try {
>       bs = (BitSet) ois.readObject();
>     } catch (ClassNotFoundException e) {
>       throw new IOException(e);
>     }
>
>     ois.close();
>   }
>
> }
>

--
Bertrand Dechoux
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB