This looks very cool, and will certainly make writing future APIs more
sane and maintainable.

Few questions/comments:

* There's one part of the current API (I forget where) that does not use
a length-prefixed repeating element, how would (or would you) represent
this using Schema? IMO, we should be consistent and always use length
prefixing for variable length things.
* I've been meaning to ask for a while, but will there be a way to
interrogate Kafka for the current API version(s)? Clients will
definitely need this
* Looks like Schema#read eagerly read values, think there is any reason
why you'd want to do this lazily?
* Now that Struct is in the mix, it seems we have 2x objects. We should
keep an eye on the GC impact of this
* Using Struct, which is more of a dynamic thing, the conversion to a
typed response class (like MetadataResponse) looks a little hairy and
probably error prone. Maybe it just needs to be decomposed a bit. Maybe
if Struct could return a "sub-Struct" for nested/repeating elements so
parsing logic could be organized better.

While we are looking at protocols, maybe we should think about what we
could do to make Kafka more amenable to large payloads. I'm not sure how
we could do that with a nice abstraction like you have, but it's worth
thinking about.

As for storing the protocol definitions, it would be really sweet if we
could keep them in a text form (something like BNF, if not just BNF
itself). This would allow other languages to write parsers for it and
automatically generate protocols. If not using a text form as the source
of truth, maybe Schema could simply generate a BNF text file.

Again, this is really cool and could definitely be it's own standalone
library :)

On 2/7/14 3:56 PM, Jay Kreps wrote: