Subject: spark.submit.deployMode: cluster


;-)

Great idea. Can you suggest a project?

Apache PredictionIO uses spark-submit (very ugly) and Apache Mahout only
launches trivially in test apps since most uses are as a lib.
From: Felix Cheung <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Reply: Felix Cheung <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Date: March 28, 2019 at 9:42:31 AM
To: Pat Ferrel <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>, Marcelo
Vanzin <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Cc: user <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Subject:  Re: spark.submit.deployMode: cluster

If anyone wants to improve docs please create a PR.

lol
But seriously you might want to explore other projects that manage job
submission on top of spark instead of rolling your own with spark-submit.
------------------------------
*From:* Pat Ferrel <[EMAIL PROTECTED]>
*Sent:* Tuesday, March 26, 2019 2:38 PM
*To:* Marcelo Vanzin
*Cc:* user
*Subject:* Re: spark.submit.deployMode: cluster

Ahh, thank you indeed!

It would have saved us a lot of time if this had been documented. I know,
OSS so contributions are welcome… I can also imagine your next comment; “If
anyone wants to improve docs see the Apache contribution rules and create a
PR.” or something like that.

BTW the code where the context is known and can be used is what I’d call a
Driver and since all code is copied to nodes and is know in jars, it was
not obvious to us that this rule existed but it does make sense.

We will need to refactor our code to use spark-submit it appears.

Thanks again.
From: Marcelo Vanzin <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Reply: Marcelo Vanzin <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Date: March 26, 2019 at 1:59:36 PM
To: Pat Ferrel <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Cc: user <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Subject:  Re: spark.submit.deployMode: cluster

If you're not using spark-submit, then that option does nothing.

If by "context creation API" you mean "new SparkContext()" or an
equivalent, then you're explicitly creating the driver inside your
application.

On Tue, Mar 26, 2019 at 1:56 PM Pat Ferrel <[EMAIL PROTECTED]> wrote:
It DOES NOY use spark-submit.
application “name” goes back to my server, the machine that launched the
job.
the Driver to run on the cluster but it runs on the client, ignoring the
spark.submit.deployMode.
--
Marcelo