Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - issue with writing an array of records


+
Alan Miller 2013-01-07, 16:35
+
Scott Carey 2013-01-08, 10:36
Copy link to this message
-
Re: issue with writing an array of records
Alan Miller 2013-01-08, 11:48
Thanks Scott, I figured it out. The problem was simply an improper loop
design.
I corrected the loop code and now I get an array of Ifnet records.

[amiller@loclhost $] avro cat --fields hostname,ifnet data_2013-01-08.avro
{"hostname": "host08", "ifnet": [{"send_bps": 0, "recv_bps": 0, "name":
"e0P"}, {"send_bps": 171256, "recv_bps": 197512, "name": "vif1"},
{"send_bps": 0, "recv_bps": 0, "name": "vif2"}, {"send_bps": 0, "recv_bps":
0, "name": "losk"}]}
{"hostname": "host11", "ifnet": [{"send_bps": 9992353544, "recv_bps":
14049010776, "name": "e1a"}, {"send_bps": 0, "recv_bps": 0, "name": "e1b"},
{"send_bps": 2736, "recv_bps": 2960, "name": "e0P"}, {"send_bps": 9376,
"recv_bps": 19312, "name": "c0a"}, {"send_bps": 0, "recv_bps": 0, "name":
"vif1"}, {"send_bps": 9376, "recv_bps": 19312, "name": "c0b"}, {"send_bps":
0, "recv_bps": 0, "name": "e0M"}, {"send_bps": 0, "recv_bps": 0, "name":
"losk"}]}
I did not have to change my schema.
For anyone interested, I verified I could write my array of records with
the schema using this test program.

public class ArrayOfRecTest {
public static void main(String[] args) throws IOException {
String fileName = "/tmp/test.avro";
File avroFile = new File(fileName);
 // NOTE: MyRecord.java was generated from MyRecord.avsc
and avro.compiler.specific.SchemaTask
Schema schema = new MyRecord().getSchema();
 DatumWriter<MyRecord> writer = new SpecificDatumWriter<MyRecord>(schema);
DataFileWriter<MyRecord> dataFileWriter = new
DataFileWriter<MyRecord>(writer);
dataFileWriter.create(schema, avroFile);

MyRecord record = new MyRecord();
record.setHostname("localhost");
record.setTimeStamp(new Long(1357639723));
 List<Ifnet> ifnetList = new ArrayList<Ifnet>();
Ifnet ifc1 = new Ifnet("eth0", 1234L, 5678L); ifnetList.add(ifc1);
Ifnet ifc2 = new Ifnet("eth1", 100L, 200L);   ifnetList.add(ifc2);
Ifnet ifc3 = new Ifnet("eth2", 0L, 0L);       ifnetList.add(ifc3);
 record.setIfnet(ifnetList);
dataFileWriter.append(record);
dataFileWriter.flush();
dataFileWriter.close();
}
}
Alan

On Tue, Jan 8, 2013 at 11:36 AM, Scott Carey <[EMAIL PROTECTED]> wrote:

>
> On 1/7/13 8:35 AM, "Alan Miller" <[EMAIL PROTECTED]> wrote:
>
>
> >Hi, I have a schema with an array of records (I'm open to other
> >suggestions too) field
> >called ifnet to store misc attribute name/values for a host's network
> >interfaces.
> >e.g.
> >
> >
> >{ "type": "record",
> >  "namespace": "com.company.avro.data",
> >  "name": "MyRecord",
> >  "doc": "My Data Record.",
> >  "fields": [
> >    // (required) fields
> >    {"name":          "time_stamp", "type": "long"
> >      },
> >    {"name":            "hostname", "type": "string"
> >      },
> >
> >    // (optional) array of ifnet instances
> >    {"name": "ifnet",
> >     "type": ["null", {
> >                "type": "array",
> >                "items": { "type": "record", "name": "Ifnet",
> >                           "namespace": "com.company.avro.data",
> >                           "fields": [ {"name": "name",           "type":
> >"string"},
> >                                       {"name": "send_bps",       "type":
> >"long"  },
> >                                       {"name": "recv_bps",       "type":
> >"long"  }
> >                           ]
> >                }
> >              }
> >     ]
> >    }
> >
> > ]
> >}
>
> First thought:  Why the union of null and the array?  It may be easier to
> simply  have an empty list when there are no Ifnet data.
>
>
>
> >
> >
> >I can write the records, (time_stamp and hostname are correct) but
> >my "array of records" field (ifnet) only contains the last element of my
> >java List.
> >
> >Am I writing the field correctly?  I'm trying to write the ifnet field
> >with a
> >java.util.List<com.company.avro.data.Ifnet>
> >
> >Here's the related code lines that write the ifnet field. (Yes, I'm
> >attempting to use reflection
> >because Ifnet is only 1 of approx 11 other array of record fields I'm
> >trying to implement.)
> >
> >   Class[] paramObj = new Class[1];