Interesting thread. Thanks for digging that up.
I would try the shuffle_mode=service experiment (forgot that wasn't
yet the default). If that doesn't do the trick, though avro as a
materialization format does not provide perfect parallelism, it should
be significantly better than what you have now (large gzip files) and
may be good enough.
On Wed, May 15, 2019 at 2:34 PM Michael Luckey <[EMAIL PROTECTED]> wrote: