Subject: Spark structured streaming: periodically refresh static data frame


1. Just loop like this.
def startQuery(): Streaming Query = {
   // Define the dataframes and start the query
}

// call this on main thread
while (notShutdown) {
   val query = startQuery()
   query.awaitTermination(refreshIntervalMs)
   query.stop()
   // refresh static data
}
2. Yes, stream-stream joins in 2.3.0, soon to be released. RC3 is available
if you want to test it right now -
https://dist.apache.org/repos/dist/dev/spark/v2.3.0-rc3-bin/.

On Wed, Feb 14, 2018 at 3:34 AM, Appu K <[EMAIL PROTECTED]> wrote: