-Re: Using AVRO C with a large schema
Douglas Creager 2013-08-18, 15:30
>> 3. AVRO C cannot write values that are bigger than the file writer's
>> specified block_size. I don't think there's enough memory to hold both the
>> original structures and a gigantic block_size.
> I don't know enough about the C implementation to verify this one and
> will leave it to others.
#3 is also true. AVRO-724  is the relevant issue. Matt Massie's
comment on that issue discusses a couple of proposed solutions.
If your data structures are large arrays, then one option is to have a
separate file for each array, and have the array elements correspond to
records in the file. Then it's only each individual array element that
needs to fit in the file's block_size.