I agree with the points that Kenneth has raised, mainly:
> In both of the above approaches, diamond dependency problems between IOs
are quite possible.
Option 1. With having more IO-s in Beam we would start hitting diamond
dependency problem more often. Relocating dependencies will help, but for
this, we should avoid exposing relocated classes to end-users of IO-s. I
can't speak about everything, but in the case of BigtableIO, it only
exposes proto-classes that aren't part of bigtable-client-core, and
shouldn't be relocated.
Option 2. Without relocation, every other IO can be potentially broken, and
we can solve this problem on a case-by-case basis. In maven world situation
becomes a little better with requireUpperBoundDeps  from
maven-enforcer-plugin. I don't know if there is a similar solution for
Option 3. There is a potential future solution for dependency conflicts
between IO-s with Java 9 JPMS , however, it could take a while before we
could use it due to compatibility issues.
As a short term solution, option 2 seems the best, we could go through
known conflicts and see if it's possible to resolve them, potentially
looking into option 1 that would take much more time.
On Mon, Mar 4, 2019 at 4:45 PM Ismaël Mejía <[EMAIL PROTECTED]> wrote:
> That looks interesting but I am not sure if I understand correctly,
> isn't the problem that the system API (Bigtable, Cassandra, etc)
> exposes guava related stuff? Or in other words, wouldn't the
> transitivie version of guava leak anyway?
> If it does not I am pretty interested on doing this to fix the
> Cassandra IO from leaking too.
> On Thu, Feb 28, 2019 at 5:17 PM Kenneth Knowles <[EMAIL PROTECTED]> wrote:
> > If someone is using BigTableIO with bigtable-client-core then having
> BigTableIO and bigtable-client-core both depend on Guava 26.0 is fine,
> right? Specifically, a user of BigTableIO after
will still have non-vendored
> Guava on the classpath due to the transitive deps of bigtable-client-core.
> > In any case it seems very wrong for the Beam root project to manage the
> version of Guava in BigTableIO since the whole point is to be compatible
> with bigtable-client-core. Would it work to delete our pinned Guava version
>  and chase down all the places it breaks, moving Guava dependencies
> local to places where an IO or extension must use it for interop? Then you
> don't need adapters.
> > In both of the above approaches, diamond dependency problems between IOs
> are quite possible.
> > I don't know if we can do better. For example, producing a
> bigtable-client-core where we have relocated Guava internally and using
> that could really be an interop nightmare as things that look like the same
> type would not be. Less likely to be broken would be bigtable-client-core
> entirely relocated and vendored, but generally IO connectors exchange
> objects with users and the users would have to use the relocated versions,
> so that's gross.
> > Kenn
> > 
> > On Thu, Feb 28, 2019 at 2:29 AM Gleb Kanterov <[EMAIL PROTECTED]> wrote:
> >> For the past week, two independent people have asked me if I can help
> with guava NoSuchMethodError in BigtableIO. It turns out we still have a
> potential problem with dependencies that don't vendor guava, in this case,
> it was bigtable-client-core that depends on guava-26.0. However, the root
> cause of the classpath problem was in the usage of a deprecated method from
> non-vendored guava in BigtableServiceClientImpl in the code path where we
> integrate with bigtable client.
> >> I created apache/beam#7957  to fix that. There few other IO-s where