Thanks for this discussion everyone. It has been very useful in getting an
overall understanding here.
I think in general, consensus is that this change doesn't introduce
behavioral changes, and it's definitely an advantage to reuse the
constructs that Spark provides to us.

Moving on to a different question here - of pushing this through to Spark.
The init containers have been tested over the past two Spark releases by
external users and integration testing - and this would be a fundamental
change to that behavior.
We should work on getting enough test coverage and confidence here.

We can start by getting a PR going perhaps, and start augmenting the
integration testing to ensure that there are no surprises - with/without
credentials, accessing GCS, S3 etc as well.
When we get enough confidence and test coverage, let's merge this in.
Does that sound like a reasonable path forward?

On Wed, Jan 10, 2018 at 2:53 PM, Marcelo Vanzin <[EMAIL PROTECTED]> wrote:

Anirudh Ramanathan