clear query| facets| time Search criteria: author:"Tathagata Das".   Results from 11 to 20 from 1176 (0.0s).
Loading phrases to help you
refine your search...
Apache Spark - Exception on adding column to Structured Streaming DataFrame - Spark - [mail # user]
...Could you give the full stack trace of the exception?Also, can you do `dataframe2.explain(true)` and show us the plan output?On Wed, Jan 31, 2018 at 3:35 PM, M Singh wrote:> Hi Folks:>...
   Author: Tathagata Das , 2018-01-31, 23:46
mapGroupsWithState in Python - Spark - [mail # user]
...Hello Ayan,From what I understand, mapGroupsWithState (probably the more generalflatMapGroupsWithState) is the best way forward (not available in python).However, you need to figure out your...
   Author: Tathagata Das , 2018-01-31, 23:39
[expand - 3 more] - what is the right syntax for self joins in Spark 2.3.0 ? - Spark - [mail # user]
...This doc is unrelated to the stream-stream join we added in StructuredStreaming. :)That said we added append mode first because it easier to reason about thesemantics of append mode especial...
   Author: Tathagata Das , 2018-03-09, 02:15
Upgrades of streaming jobs - Spark - [mail # user]
...Yes, all checkpoints are forward compatible.However, you do need to restart the query if you want to update the code ofthe query. This downtime can be in less than a second (if you just rest...
   Author: Tathagata Das , 2018-03-09, 22:24
Trigger.ProcessingTime("10 seconds") & Trigger.Continuous(10.seconds) - Spark - [mail # user]
...The continuous one is our new low latency continuous processing engine inStructured Streaming (to be released in 2.3).Here is the pre-release doc -https://dist.apache.org/repos/dist/dev/spar...
   Author: Tathagata Das , 2018-02-26, 06:11
[Structured Streaming] Avoiding multiple streaming queries - Spark - [mail # user]
...Of course, you can write to multiple Kafka topics from a single query. Ifyour dataframe that you want to write has a column named "topic" (alongwith "key", and "value" columns), it will writ...
   Author: Tathagata Das , 2018-02-15, 02:11
Apache Spark - Custom structured streaming data source - Spark - [mail # user]
...Hello Mans,The streaming DataSource APIs are still evolving and are not public yet.Hence there is no official documentation. In fact, there is a newDataSourceV2 API (in Spark 2.3) that we ar...
   Author: Tathagata Das , 2018-01-26, 07:33
Spark Streaming withWatermark - Spark - [mail # user]
...That may very well be possible. The watermark delay guarantees that anyrecord newer than or equal to watermark (that is, max event time seen - 20seconds), will be considered and never be ign...
   Author: Tathagata Das , 2018-02-07, 01:41
[expand - 1 more] - Multiple Kafka Spark Streaming Dataframe Join query - Spark - [mail # user]
...Relevant:https://databricks.com/blog/2018/03/13/introducing-stream-stream-joins-in-apache-spark-2-3.htmlThis is true stream-stream join which will automatically buffer delayeddata and approp...
   Author: Tathagata Das , 2018-03-14, 19:35
[expand - 1 more] - CachedKafkaConsumer: CachedKafkaConsumer is not running in UninterruptibleThread warning - Spark - [mail # user]
...Which version of Spark are you using? And can you give us the full stacktrace of the exception?On Tue, Mar 6, 2018 at 1:53 AM, Junfeng Chen  wrote:> I am trying to read kafka and sav...
   Author: Tathagata Das , 2018-03-06, 19:35