clear query| facets| time Search criteria: author:"Tathagata Das".   Results from 21 to 30 from 1165 (0.0s).
Loading phrases to help you
refine your search...
[Structured Streaming] Reading Checkpoint data - Spark - [mail # user]
...Only the stream metadata (e.g., streamid, offsets) are stored as json. Thestream state data is stored in an internal binary format.On Mon, Jul 9, 2018 at 4:07 PM, subramgr wrote:> Hi,>...
   Author: Tathagata Das , 2018-07-10, 00:36
[Structured Streaming] Custom StateStoreProvider - Spark - [mail # user]
...Note that this is not public API yet. Hence this is not very documented. Souse it at your own risk :)On Tue, Jul 10, 2018 at 11:04 AM, subramgr wrote:> Hi,>> This looks very dauntin...
   Author: Tathagata Das , 2018-07-11, 03:40
[expand - 1 more] - [Structured Streaming] Application Updates in Production - Spark - [mail # user]
...Yes indeed, we dont directly support schema migration of state as of now.However, depending on what stateful operator you are using, you can workaround it. For example, if you are using mapG...
   Author: Tathagata Das , 2018-03-23, 03:01
Apache Spark Structured Streaming - Kafka Consumer cannot fetch records for offset exception - Spark - [mail # user]
...Structured Streaming AUTOMATICALLY saves the offsets in a checkpointdirectory that you provide. And when you start the query again with thesame directory it will just pick up where it left o...
   Author: Tathagata Das , 2018-03-23, 03:15
How to branch a Stream / have multiple Sinks / do multiple Queries on one Stream - Spark - [mail # user]
...Hey all,In Spark 2.4.0, there will be a new feature called *foreachBatch* whichwill expose the output rows of every micro-batch as a dataframe, on whichyou apply a user-defined function. Wit...
   Author: Tathagata Das , 2018-07-05, 23:01
Dataset - withColumn and withColumnRenamed that accept Column type - Spark - [mail # user]
...Yes. Yes you can.On Tue, Jul 17, 2018 at 11:42 AM, Sathi Chowdhury <[EMAIL PROTECTED]lid> wrote:> Hi,> My question is about ability to integrate spark streaming with multiple>...
   Author: Tathagata Das , 2018-07-18, 01:00
Does Spark Structured Streaming have a JDBC sink or Do I need to use ForEachWriter? - Spark - [mail # user]
...Actually, we do not support jdbc sink yet. The blog post was just anexample :) I agree it is misleading in hindsight.On Wed, Jun 20, 2018 at 6:09 PM, kant kodali  wrote:> Hi All,>...
   Author: Tathagata Das , 2018-06-21, 07:08
[expand - 2 more] - Lag and queued up batches info in Structured Streaming UI - Spark - [mail # user]
...Structured Streaming does not maintain a queue of batch like DStream.DStreams used to cut off batches at a fixed interval and put in a queue,and a different thread processed queued batches. ...
   Author: Tathagata Das , 2018-06-21, 01:07
Recommendation of using StreamSinkProvider for a custom KairosDB Sink - Spark - [mail # user]
...This is interface is actually unstable. The v2 of DataSource APIs is beingdesigned right now which will be public and stable in a release or two. Sounfortunately there is no stable interface...
   Author: Tathagata Das , 2018-06-25, 21:33
[expand - 1 more] - Spark can't identify the event time column being supplied to withWatermark() - Spark - [mail # user]
...Try to define the watermark on the right column immediately before calling`groupByKey(...).mapGroupsWithState(...)`. You are applying the watermarkand then doing a bunch of opaque transforma...
   Author: Tathagata Das , 2018-06-08, 20:28