clear query| facets| time Search criteria: author:"Tathagata Das".   Results from 1 to 10 from 14 (0.0s).
Loading phrases to help you
refine your search...
Announcing Delta Lake 0.2.0 - Spark - [mail # user]
...@ayan guha  @Gourav SenguptaDelta Lake is OSS currently does not support defining tables in Hivemetastore using DDL commands. We are hoping to add the necessarycompatibility fixes in Ap...
   Author: Tathagata Das , 2019-06-21, 10:03
Announcing Delta Lake 0.3.0 - Spark - [mail # user]
...Hello everyone,We are excited to announce the availability of Delta Lake 0.3.0 whichintroduces new programmatic APIs for manipulating and managing data inDelta Lake tables.Here are the main ...
   Author: Tathagata Das , 2019-08-02, 01:45
Structured Streaming: How to add a listener for when a batch is complete - Spark - [mail # user]
...https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#reporting-metrics-programmatically-using-asynchronous-apisOn Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz wrot...
   Author: Tathagata Das , 2019-09-04, 01:28
Stateful Structured Spark Streaming: Timeout is not getting triggered - Spark - [mail # user]
...Make sure that you are continuously feeding data into the query to triggerthe batches. only then timeouts are processed.See the timeout behavior details here -https://spark.apache.org/docs/l...
   Author: Tathagata Das , 2020-03-05, 02:31
[expand - 1 more] - dropDuplicates and watermark in structured streaming - Spark - [mail # user]
...1. Yes. All times in event time, not processing time. So you may get 10AMevent time data at 11AM processing time, but it will still be comparedagain all data within 9-10AM event times.2. Sho...
   Author: Tathagata Das , 2020-02-28, 02:25
Spark Streaming: Aggregating values across batches - Spark - [mail # user]
...Use Structured Streaming. Its aggregation, by definition, is across batches.On Thu, Feb 27, 2020 at 3:17 PM Something Something <[EMAIL PROTECTED]> wrote:> We've a Spark Streaming j...
   Author: Tathagata Das , 2020-02-28, 01:44
[expand - 1 more] - Structured Streaming: mapGroupsWithState UDT serialization does not work - Spark - [mail # user]
...You are deserializing by explicitly specifying UTC timezone, but whenserializing you are not specifying it. Maybe that is reason?Also, if you can encode it using just long, then I recommend ...
   Author: Tathagata Das , 2020-02-28, 21:56
[expand - 1 more] - Structured Streaming Dataframe Size - Spark - [mail # user]
...https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#basic-concepts*Note that Structured Streaming does not materialize the entire table*. It> reads the latest...
   Author: Tathagata Das , 2019-08-27, 22:42
how can I dynamic parse json in kafka when using Structured Streaming - Spark - [mail # user]
...You can use *from_json* built-in SQL function to parse json.https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html#from_json-org.apache.spark.sql.Column-org.apache...
   Author: Tathagata Das , 2019-09-17, 08:13
What is the best way to consume parallely from multiple topics in Spark Stream with Kafka - Spark - [mail # user]
...Why are you not using Structured Streaming? Structured Streaming kafkasupport directly support multiple topics.val df = spark.readStream.format("kafka").option("subscribe","topic1,topic2").l...
   Author: Tathagata Das , 2020-03-18, 19:02