clear query| facets| time Search criteria: author:"Tathagata Das".   Results from 31 to 40 from 1165 (0.0s).
Loading phrases to help you
refine your search...
Reset the offsets, Kafka 0.10 and Spark - Spark - [mail # user]
...Structured Streaming really makes this easy. You can simply specify theoption of whether the start the query from earliest or latest.Check out-https://www.slideshare.net/databricks/a-deep-di...
   Author: Tathagata Das , 2018-06-08, 21:54
How to reduceByKeyAndWindow in Structured Streaming? - Spark - [mail # user]
...The fundamental conceptual difference between the windowing in DStream vsStructured Streaming is that DStream used the arrival time of the record inSpark (aka processing time) and Structured...
   Author: Tathagata Das , 2018-06-28, 20:38
[structured-streaming][parquet] readStream files order in Parquet - Spark - [mail # user]
...The files are processed in the order the file last modified timestamp. Thepath and partitioning scheme are not used for ordering.On Thu, Jun 14, 2018 at 6:59 AM, karthikjay  wrote:> ...
   Author: Tathagata Das , 2018-06-15, 21:02
Announcing Delta Lake 0.2.0 - Spark - [mail # user]
...@ayan guha  @Gourav SenguptaDelta Lake is OSS currently does not support defining tables in Hivemetastore using DDL commands. We are hoping to add the necessarycompatibility fixes in Ap...
   Author: Tathagata Das , 2019-06-21, 10:03
Iterator of KeyValueGroupedDataset.flatMapGroupsWithState function - Spark - [mail # user]
...It is okay to collect the iterator. That will not break Spark. However,collecting it requires memory in the executor, so you may cause OOMs if agroup has a LOT of new data.On Wed, Oct 31, 20...
   Author: Tathagata Das , 2018-10-31, 21:36
Can we pass the Calcite streaming sql queries to spark sql? - Spark - [mail # user]
...I dont think so. Calcite's SQL is an extension of standard SQL (keywordslike STREAM, etc.) which we dont support; we just support regular SQL, soqueries like "SELECT STREAM ...." will not wo...
   Author: Tathagata Das , 2017-11-09, 21:15
In structured streamin, multiple streaming aggregations are not yet supported. - Spark - [mail # dev]
...Hello,What do you mean by multiple streaming aggregations? Something like this isalready supported.*df.groupBy("key").agg(min("colA"), max("colB"), avg("colC"))*But the following is not supp...
   Author: Tathagata Das , 2017-11-29, 06:24
OOM: Structured Streaming aggregation state not cleaned up properly - Spark - [mail # user]
...Just to be clear, these screenshots are about the memory consumption of thedriver. So this is nothing to do with streaming aggregation state which arekept in the memory of the executors, not...
   Author: Tathagata Das , 2018-05-22, 23:06
[Structured Streaming] Two watermarks and StreamingQueryListener - Spark - [mail # user]
...Structured Streaming internally maintains one global watermark by taking amin of the two watermarks. Thats why one gets reported. In Spark 2.4, therewill be the option of choosing max instea...
   Author: Tathagata Das , 2018-08-10, 23:15
Does structured streaming support Spark Kafka Direct? - Spark - [mail # user]
...The parallelism is same for Structured Streaming. In fact, the KafkaStructured Streaming source is based on the same principle as DStream'sKafka Direct, hence it has very similar behavior.On...
   Author: Tathagata Das , 2018-04-12, 07:03