clear query| facets| time Search criteria: author:"Michael Armbrust".   Results from 1 to 10 from 1483 (0.0s).
Loading phrases to help you
refine your search...
Dataset API inconsistencies - Spark - [mail # user]
...I wrote Datasets, and I'll say I only use them when I really need to (i.e.when it would be very cumbersome to express what I am trying to dorelationally).  Dataset operations are almost...
   Author: Michael Armbrust , 2018-01-10, 09:48
[VOTE] Spark 2.3.0 (RC4) - Spark - [mail # dev]
...I'm -1 on any changes that aren't fixing major regressions from 2.2 at thispoint. Also in any cases where its possible we should be flipping newfeatures off if they are still regressing, rat...
   Author: Michael Armbrust , 2018-02-21, 20:26
Prefer Structured Streaming over Spark Streaming (DStreams)? - Spark - [mail # user]
...At this point I recommend that new applications are built using structuredstreaming. The engine was GA-ed as of Spark 2.2 and I know of several verylarge (trillions of records) production jo...
   Author: Michael Armbrust , 2018-01-31, 23:12
Max number of streams supported ? - Spark - [mail # user]
...-dev +user> Similarly for structured streaming, Would there be any limit on number of> of streaming sources I can have ?>There is no fundamental limit, but each stream will have a t...
   Author: Michael Armbrust , 2018-01-31, 20:40
[VOTE] Spark 2.3.0 (RC5) - Spark - [mail # dev]
...+1 all our pipelines have been running the RC for several days now.On Mon, Feb 26, 2018 at 10:33 AM, Dongjoon Hyun wrote:> +1 (non-binding).>> Bests,> Dongjoon.>>>> O...
   Author: Michael Armbrust , 2018-02-26, 20:09
DataSourceV2: support for named tables - Spark - [mail # dev]
...I am definitely in favor of first-class / consistent support for tables anddata sources.One thing that is not clear to me from this proposal is exactly what theinterfaces are between: -...
   Author: Michael Armbrust , 2018-02-03, 00:11
SQL logical plans and DataSourceV2 (was: data source v2 online meetup) - Spark - [mail # dev]
...>> So here are my recommendations for moving forward, with DataSourceV2 as a> starting point:>>    1. Use well-defined logical plan nodes for all high-level operatio...
   Author: Michael Armbrust , 2018-02-02, 23:23
[Structured Streaming] Deserializing avro messages from kafka source using schema registry - Spark - [mail # user]
...This isn't supported yet, but there is on going work at spark-avro to enable this usecase.  Stay tuned.On Fri, Feb 9, 2018 at 3:07 PM, Bram  wrote:> Hi,>> I couldn't ...
   Author: Michael Armbrust , 2018-02-10, 00:20
[Structured Streaming] Commit protocol to move temp files to dest path only when complete, with code - Spark - [mail # user]
...We didn't go this way initially because it doesn't work on storage systemsthat have weaker guarantees than HDFS with respect to rename.  That said,I'm happy to look at other options if ...
   Author: Michael Armbrust , 2018-02-10, 00:19
select count * doesnt seem to respect update mode in Kafka Structured Streaming? - Spark - [mail # user]
...Those options will not affect structured streaming.  You are looking for.option("maxOffsetsPerTrigger", "1000")We are working on improving this by building a generic mechanism into theS...
   Author: Michael Armbrust , 2018-03-21, 00:01