Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> issue with writing an array of records


Copy link to this message
-
Re: issue with writing an array of records
Thanks Scott, I figured it out. The problem was simply an improper loop
design.
I corrected the loop code and now I get an array of Ifnet records.

[amiller@loclhost $] avro cat --fields hostname,ifnet data_2013-01-08.avro
{"hostname": "host08", "ifnet": [{"send_bps": 0, "recv_bps": 0, "name":
"e0P"}, {"send_bps": 171256, "recv_bps": 197512, "name": "vif1"},
{"send_bps": 0, "recv_bps": 0, "name": "vif2"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}]}
{"hostname": "host11", "ifnet": [{"send_bps": 9992353544, "recv_bps":
14049010776, "name": "e1a"}, {"send_bps": 0, "recv_bps": 0, "name": "e1b"},
{"send_bps": 2736, "recv_bps": 2960, "name": "e0P"}, {"send_bps": 9376,
"recv_bps": 19312, "name": "c0a"}, {"send_bps": 0, "recv_bps": 0, "name":
"vif1"}, {"send_bps": 9376, "recv_bps": 19312, "name": "c0b"}, {"send_bps":
0, "recv_bps": 0, "name": "e0M"}, {"send_bps": 0, "recv_bps": 0, "name":
"losk"}]}
I did not have to change my schema.
For anyone interested, I verified I could write my array of records with
the schema using this test program.

public class ArrayOfRecTest {
public static void main(String[] args) throws IOException {
String fileName = "/tmp/test.avro";
File avroFile = new File(fileName);
 // NOTE: MyRecord.java was generated from MyRecord.avsc
and avro.compiler.specific.SchemaTask
Schema schema = new MyRecord().getSchema();
 DatumWriter<MyRecord> writer = new SpecificDatumWriter<MyRecord>(schema);
DataFileWriter<MyRecord> dataFileWriter = new
DataFileWriter<MyRecord>(writer);
dataFileWriter.create(schema, avroFile);

MyRecord record = new MyRecord();
record.setHostname("localhost");
record.setTimeStamp(new Long(1357639723));
 List<Ifnet> ifnetList = new ArrayList<Ifnet>();
Ifnet ifc1 = new Ifnet("eth0", 1234L, 5678L); ifnetList.add(ifc1);
Ifnet ifc2 = new Ifnet("eth1", 100L, 200L);   ifnetList.add(ifc2);
Ifnet ifc3 = new Ifnet("eth2", 0L, 0L);       ifnetList.add(ifc3);
 record.setIfnet(ifnetList);
dataFileWriter.append(record);
dataFileWriter.flush();
dataFileWriter.close();
}
}
Alan

On Tue, Jan 8, 2013 at 11:36 AM, Scott Carey <[EMAIL PROTECTED]> wrote:

>
> On 1/7/13 8:35 AM, "Alan Miller" <[EMAIL PROTECTED]> wrote:
>
>
> >Hi, I have a schema with an array of records (I'm open to other
> >suggestions too) field
> >called ifnet to store misc attribute name/values for a host's network
> >interfaces.
> >e.g.
> >
> >
> >{ "type": "record",
> >  "namespace": "com.company.avro.data",
> >  "name": "MyRecord",
> >  "doc": "My Data Record.",
> >  "fields": [
> >    // (required) fields
> >    {"name":          "time_stamp", "type": "long"
> >      },
> >    {"name":            "hostname", "type": "string"
> >      },
> >
> >    // (optional) array of ifnet instances
> >    {"name": "ifnet",
> >     "type": ["null", {
> >                "type": "array",
> >                "items": { "type": "record", "name": "Ifnet",
> >                           "namespace": "com.company.avro.data",
> >                           "fields": [ {"name": "name",           "type":
> >"string"},
> >                                       {"name": "send_bps",       "type":
> >"long"  },
> >                                       {"name": "recv_bps",       "type":
> >"long"  }
> >                           ]
> >                }
> >              }
> >     ]
> >    }
> >
> > ]
> >}
>
> First thought:  Why the union of null and the array?  It may be easier to
> simply  have an empty list when there are no Ifnet data.
>
>
>
> >
> >
> >I can write the records, (time_stamp and hostname are correct) but
> >my "array of records" field (ifnet) only contains the last element of my
> >java List.
> >
> >Am I writing the field correctly?  I'm trying to write the ifnet field
> >with a
> >java.util.List<com.company.avro.data.Ifnet>
> >
> >Here's the related code lines that write the ifnet field. (Yes, I'm
> >attempting to use reflection
> >because Ifnet is only 1 of approx 11 other array of record fields I'm
> >trying to implement.)
> >
> >   Class[] paramObj = new Class[1];
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB