Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> issue with writing an array of records


Copy link to this message
-
issue with writing an array of records
Hi, I have a schema with an array of records (I'm open to other suggestions
too) field
called ifnet to store misc attribute name/values for a host's network
interfaces.
e.g.

{ "type": "record",
  "namespace": "com.company.avro.data",
  "name": "MyRecord",
  "doc": "My Data Record.",
  "fields": [
    // (required) fields
    {"name":          "time_stamp", "type": "long"
     },
    {"name":            "hostname", "type": "string"
     },
    // (optional) array of ifnet instances
    {"name": "ifnet",
     "type": ["null", {
                "type": "array",
                "items": { "type": "record", "name": "Ifnet",
                           "namespace": "com.company.avro.data",
                           "fields": [ {"name": "name",           "type":
"string"},
                                       {"name": "send_bps",       "type":
"long"  },
                                       {"name": "recv_bps",       "type":
"long"  }
                           ]
                }
              }
     ]
    }
 ]
}

I can write the records, (time_stamp and hostname are correct) but
my "array of records" field (ifnet) only contains the last element of my
java List.

Am I writing the field correctly?  I'm trying to write the ifnet field with
a
java.util.List<com.company.avro.data.Ifnet>

Here's the related code lines that write the ifnet field. (Yes, I'm
attempting to use reflection
because Ifnet is only 1 of approx 11 other array of record fields I'm
trying to implement.)

   Class[] paramObj = new Class[1];
   paramObj[0] = Ifnet.class;
   Method method = cls.getMethod(methodName, List.class);
   jsonObj = new Ifnet();
   listOfObj = new ArrayList<Ifnet>();
   .......

   // in a loop building the List<Ifnet>.......
    LOG.info(String.format("   [%s] %s %s(%s) as %s", name,
k,methNm,v,types[j].toString()));
   .......
    LOG.info(String.format("   [%s] setting name to %s", name, name));
   .......
   istOfObj.add(jsonObj);
   .......

  // then finally I call invoke with a List of Ifnet records
  if (method != null) { method.invoke(obj, listOfObj); }
  LOG.info(String.format("  invoking %s.%s",
method.getClass().getSimpleName(), method.getName()));
  LOG.info(String.format("  param: listObj<%s> with %d entries" ,
jsonObj.getClass().getName(), listOfObj.size()));

and the respective output
20130107T172303  INFO c.c.a.d.MyDriver - Set                ifnet     json
via             setIfnet(Ifnet object)
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] setting name to e0c
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] setting name to e0c
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] setting name to e0d
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] setting name to e0d
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] setting name to e0a
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] send_bps
setSendBps(170720) as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] setting name to e0a
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] recv_bps
setRecvBps(244480) as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] setting name to e0b
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] setting name to e0b
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] setting name to e0P
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] setting name to e0P
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] setting name to losk
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] send_bps setSendBps(0)
as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] setting name to losk
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] recv_bps setRecvBps(0)
as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -   invoking Method.setIfnet
20130107T172303  INFO c.c.a.d.MyDriver -   param:
listObj<com.synopsys.iims.be.storage.Ifnet> with 6 entries
20130107T172303  INFO c.c.a.d.MyDriver - Set           time_stamp  integer
via         setTimeStamp to 1357513251

When I dump the records I see an array of 6 entries but the values all
reflect the last last entry in my java.util.List.
The print statements suggest I'm passing a list of 6 items with different
values,

  [amiller@localhost $] hadoop fs -copyToLocal /data/test_2013-01-07.avro
  [amiller@localhost $] avro cat --fields=time_stamp,ifnet
 test_2013-01-07.avro|less
  {"time_stamp": 1357513251, "ifnet": [{"send_bps": 0, "recv_bps": 0,
"name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}]}
  {"time_stamp": 1357513550, "ifnet": [{"send_bps": 0, "recv_bps": 0,
"name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}]}
Any ideas.
Alan
+
Scott Carey 2013-01-08, 10:36
+
Alan Miller 2013-01-08, 11:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB