Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> accumulo for a bi-map?


Copy link to this message
-
RE: accumulo for a bi-map?
I'm not sure how familiar you are with Accumulo, but you do not need to
specify your columns when you create the table. You could create a table
that stores the feature vector for your source followed by columns for the
related objects. Sounds like you are already thinking down this path. For
example:

 

 
Row

Column Family

Column Qualifier

Value
abcd

 

 

Feature vector
abcd

efgh

88

 
abcd

ijkl

90

 
ijkl

 

 

Feature vector
ijkl

abcd

90

 

 

The RFile format will compress repeating row, colf, and colq values down to
1. Not sure how you are searching, but you could switch the colq and colf in
the example above to sort by relative score. Requirements change over time,
so the table format above would also allow you to store different versions
of the same relationship so that you could track the history over time if
that became important. It would also allow you to provide a different score
for each direction of the relationship if that matters later.

 

From: Marc Reichman [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, July 16, 2013 5:28 PM
To: [EMAIL PROTECTED]
Subject: accumulo for a bi-map?

 

We are using accumulo as a mechanism to store feature data (binary byte[])
for some simple keys which are used for a search algorithm. We currently
search by iterating over the feature space using AccumuloRowInputFormat.
Results come out of a reducer into HDFS, currently in a SequenceFile.

 

A customer has asked if we can store our results somewhere in our Hadoop
infrastructure, and also perform nightly searches of everything vs
everything to keep match results up to date.

 

To me, the storage of the results in alternate column families (from the
features) would be a way way to store the matches alongside the key rows:

(key: abcd, features:{...}, matches{ 'm0: efgh-88%, 'm1': ijkl-90%, ...,
'mN': etc }

(key: ijkl, features:{...}, matches{ 'm0: efgh-88%, 'm1': abcd-90%, ...,
'mN': etc }

 

Match scores are equal between two items regardless of perspective, so a->b
is 90% as b->a is 90%.

 

Is there a way to simply add columns to an existing family without having to
name them or keep track of how many there are? Am I better off making a
column family for each match key and then store score and other fields in
columns? Making one column with the key as the name and the score as the
value for each match under one family?

 

Ideally I would have some form of bidirectional map so I could look at any
key and find all the results as other keys, and find any results to get
other matches.

 

One approach is to simply add both sides of the relationship every time
anything matches anything else, which seems a bit wasteful, space-wise.

 

Curious if any pre-existing ideas are out there. Currently on hadoop
1.0.3/accumulo 1.4.1, not set in (hard) concrete.

 

Thanks,

Marc

 

 

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB