Subject: Unable to understand error “SparkListenerBus has already stopped! Dropping event …”

Hi there,

I'd like to know if anyone has a magic method to avoid such messages in
Spark logs:

2015-08-30 19:30:44 ERROR LiveListenerBus:75 - SparkListenerBus has already
stopped! Dropping event

After further investigations, I understand that LiveListenerBus extends
AsynchronousListenerBus. And thus, at some point, .stop() method is called.
Then, messages that might be sent/received will be dropped and remain
unprocessed. Basically, some SparkListenerExecutorMetricsUpdate messages
are unfortunately not received yet, and once they are, they become dropped
to nowhere.

This doesn't look critical since SparkListenerExecutorMetricsUpdate only
corresponds to Periodic updates from executors.

What is embarrassing is that I absolutely don't understand why this happens
and nothings refers to this issue. Note that this is totally
non-deterministic and I can't reproduce this, probably due to the
asynchronous nature and my lack of understand on how/when stop() is
supposed to be called.

Any idea?



*Adrien Mogenet*
Head of Backend/Infrastructure
50, avenue Montaigne - 75008 Paris