clear query| facets| time Search criteria: author:"Tathagata Das".   Results from 1 to 10 from 1165 (0.0s).
Loading phrases to help you
refine your search...
Spark structured streaming time series forecasting - Spark - [mail # user]
...Spark-ts has been under development for a while. So I doubt there is anyintegration with Structured Streaming. That said, Structured Streaming usesDataFrames and Datasets, and a lot of exist...
   Author: Tathagata Das , 2018-01-09, 22:24
Apache Spark - Question about Structured Streaming Sink addBatch dataframe size - Spark - [mail # user]
...1. It is all the result data in that trigger. Note that it takes aDataFrame which is a purely logical representation of data and has noassociation with partitions, etc. which are physical re...
   Author: Tathagata Das , 2018-01-03, 22:28
[expand - 1 more] - flatMapGroupsWithState not timing out (spark 2.2.1) - Spark - [mail # user]
...Hello Dan,From your code, it seems like you are setting the timeout timestamp basedon the current processing-time / wall-clock-time, while the watermark isbeing calculated on the event-time ...
   Author: Tathagata Das , 2018-01-12, 23:40
[Spark structured streaming] Use of (flat)mapgroupswithstate takes long time - Spark - [mail # user]
...For computing mapGroupsWithState, can you check the following.- How many tasks are being launched in the reduce stage (that is, the stageafter the shuffle, that is computing mapGroupsWithSta...
   Author: Tathagata Das , 2018-01-22, 23:04
[expand - 1 more] - Spark structured streaming: periodically refresh static data frame - Spark - [mail # user]
...Let me fix my mistake :)What I suggested in that earlier thread does not work. The streaming querythat joins a streaming dataset with a batch view, does not correctly pickup when the view is...
   Author: Tathagata Das , 2018-02-14, 11:11
[expand - 1 more] - [Beginner] Kafka 0.11 header support in Spark Structured Streaming - Spark - [mail # user]
...Unfortunately, exposing Kafka headers is not yet supported in StructuredStreaming. The community is more than welcome to add support for it :)On Tue, Feb 27, 2018 at 2:51 PM, Karthik Jayaram...
   Author: Tathagata Das , 2018-02-28, 01:12
How does Spark Structured Streaming determine an event has arrived late? - Spark - [mail # user]
...Let me answer the original question directly, that is, how do we determinethat an event is late. We simply track the maximum event time the enginehas seen in the data it has processed till n...
   Author: Tathagata Das , 2018-02-28, 01:46
[expand - 1 more] - [Beginner] How to save Kafka Dstream data to parquet ? - Spark - [mail # user]
...There is no good way to save to parquet without causing downstreamconsistency issues.You could use foreachRDD to get each RDD, convert it to DataFrame/Dataset,and write out as parquet files....
   Author: Tathagata Das , 2018-02-28, 22:00
[expand - 1 more] - Spark Structured Streaming for Twitter Streaming data - Spark - [mail # user]
...The code uses the format "socket" which is only for text sent over a simplesocket, which is completely different from how Twitter APIs works. So thiswont work at all.Fundamentally, for Struc...
   Author: Tathagata Das , 2018-02-01, 03:36
Max number of streams supported ? - Spark - [mail # user]
...Just to clarify a subtle difference between DStreams and StructuredStreaming. Multiple input streams in a DStreamGraph is likely to mean theyare all being processed/computed in the same way ...
   Author: Tathagata Das , 2018-01-31, 23:11