Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - issue with writing an array of records


Copy link to this message
-
issue with writing an array of records
Alan Miller 2013-01-07, 16:35
Hi, I have a schema with an array of records (I'm open to other suggestions
too) field
called ifnet to store misc attribute name/values for a host's network
interfaces.
e.g.

{ "type": "record",
  "namespace": "com.company.avro.data",
  "name": "MyRecord",
  "doc": "My Data Record.",
  "fields": [
    // (required) fields
    {"name":          "time_stamp", "type": "long"
     },
    {"name":            "hostname", "type": "string"
     },
    // (optional) array of ifnet instances
    {"name": "ifnet",
     "type": ["null", {
                "type": "array",
                "items": { "type": "record", "name": "Ifnet",
                           "namespace": "com.company.avro.data",
                           "fields": [ {"name": "name",           "type":
"string"},
                                       {"name": "send_bps",       "type":
"long"  },
                                       {"name": "recv_bps",       "type":
"long"  }
                           ]
                }
              }
     ]
    }
 ]
}

I can write the records, (time_stamp and hostname are correct) but
my "array of records" field (ifnet) only contains the last element of my
java List.

Am I writing the field correctly?  I'm trying to write the ifnet field with
a
java.util.List<com.company.avro.data.Ifnet>

Here's the related code lines that write the ifnet field. (Yes, I'm
attempting to use reflection
because Ifnet is only 1 of approx 11 other array of record fields I'm
trying to implement.)

   Class[] paramObj = new Class[1];
   paramObj[0] = Ifnet.class;
   Method method = cls.getMethod(methodName, List.class);
   jsonObj = new Ifnet();
   listOfObj = new ArrayList<Ifnet>();
   .......

   // in a loop building the List<Ifnet>.......
    LOG.info(String.format("   [%s] %s %s(%s) as %s", name,
k,methNm,v,types[j].toString()));
   .......
    LOG.info(String.format("   [%s] setting name to %s", name, name));
   .......
   istOfObj.add(jsonObj);
   .......

  // then finally I call invoke with a List of Ifnet records
  if (method != null) { method.invoke(obj, listOfObj); }
  LOG.info(String.format("  invoking %s.%s",
method.getClass().getSimpleName(), method.getName()));
  LOG.info(String.format("  param: listObj<%s> with %d entries" ,
jsonObj.getClass().getName(), listOfObj.size()));

and the respective output
20130107T172303  INFO c.c.a.d.MyDriver - Set                ifnet     json
via             setIfnet(Ifnet object)
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] setting name to e0c
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] setting name to e0c
20130107T172303  INFO c.c.a.d.MyDriver -    [e0c] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] setting name to e0d
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] setting name to e0d
20130107T172303  INFO c.c.a.d.MyDriver -    [e0d] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] setting name to e0a
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] send_bps
setSendBps(170720) as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] setting name to e0a
20130107T172303  INFO c.c.a.d.MyDriver -    [e0a] recv_bps
setRecvBps(244480) as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] setting name to e0b
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] setting name to e0b
20130107T172303  INFO c.c.a.d.MyDriver -    [e0b] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] setting name to e0P
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] send_bps setSendBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] setting name to e0P
20130107T172303  INFO c.c.a.d.MyDriver -    [e0P] recv_bps setRecvBps(0) as
class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] setting name to losk
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] send_bps setSendBps(0)
as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] setting name to losk
20130107T172303  INFO c.c.a.d.MyDriver -    [losk] recv_bps setRecvBps(0)
as class java.lang.Long
20130107T172303  INFO c.c.a.d.MyDriver -   invoking Method.setIfnet
20130107T172303  INFO c.c.a.d.MyDriver -   param:
listObj<com.synopsys.iims.be.storage.Ifnet> with 6 entries
20130107T172303  INFO c.c.a.d.MyDriver - Set           time_stamp  integer
via         setTimeStamp to 1357513251

When I dump the records I see an array of 6 entries but the values all
reflect the last last entry in my java.util.List.
The print statements suggest I'm passing a list of 6 items with different
values,

  [amiller@localhost $] hadoop fs -copyToLocal /data/test_2013-01-07.avro
  [amiller@localhost $] avro cat --fields=time_stamp,ifnet
 test_2013-01-07.avro|less
  {"time_stamp": 1357513251, "ifnet": [{"send_bps": 0, "recv_bps": 0,
"name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}]}
  {"time_stamp": 1357513550, "ifnet": [{"send_bps": 0, "recv_bps": 0,
"name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}, {"send_bps": 0, "recv_bps": 0, "name": "losk"},
{"send_bps": 0, "recv_bps": 0, "name": "losk"}]}
Any ideas.
Alan