Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Using AVRO C with a large schema

Dan.Davies@... 2013-08-16, 18:22
Doug Cutting 2013-08-16, 18:29
Copy link to this message
Re: Using AVRO C with a large schema
Douglas Creager 2013-08-18, 15:30
>> 3. AVRO C cannot write values that are bigger than the file writer's
>> specified block_size.  I don't think there's enough memory to hold both the
>> original structures and a gigantic block_size.
> I don't know enough about the C implementation to verify this one and
> will leave it to others.

#3 is also true.  AVRO-724 [1] is the relevant issue.  Matt Massie's
comment on that issue discusses a couple of proposed solutions.

If your data structures are large arrays, then one option is to have a
separate file for each array, and have the array elements correspond to
records in the file.  Then it's only each individual array element that
needs to fit in the file's block_size.