Avro files have special byte-string markers that can be used to find the
beginning of the next block but it will add some complexity.  My
understanding is the Python Avro libraries are very slow.  You may want to
try a prototype in Java and see if that meets your performance needs.

Alternatively, you can try writing several small files and process those in


On Thu, Jul 5, 2018 at 5:11 PM Troy X <[EMAIL PROTECTED]> wrote: