clear query| facets| time Search criteria: .   Results from 1 to 10 from 20 (0.0s).
Loading phrases to help you
refine your search...
[expand - 2 more] - [Proposal] New feature: reconfigurable number of partitions on stateful operators in Structured Streaming - Spark - [mail # dev]
...I'd agree it might make sense to bundle this into an API. We'd have tothink about whether it's a common enough use case to justify the APIcomplexity.It might be worth exploring decoupling st...
   Author: Joseph Torres , 2018-08-03, 18:10
code freeze and branch cut for Apache Spark 2.4 - Spark - [mail # dev]
...Full continuous processing aggregation support ran into unanticipatedscalability and scheduling problems. We’re planning to overcome those byusing some of the barrier execution machinery, bu...
   Author: Joseph Torres , 2018-07-31, 14:11
JDBC Data Source and customSchema option but DataFrameReader.assertNoSpecifiedSchema? - Spark - [mail # dev]
...I guess the question is partly about the semantics ofDataFrameReader.schema. If it's supposed to mean "the loaded dataframe willdefinitely have exactly this schema", that doesn't quite match...
   Author: Joseph Torres , 2018-07-16, 14:27
TextSocketMicroBatchReader no longer supports nc utility - Spark - [mail # dev]
...I tend to agree that this is a bug. It's kinda silly that nc does this, buta socket connector that doesn't work with netcat will surely seem broken tousers. It wouldn't be a huge change to d...
   Author: Joseph Torres , 2018-06-04, 16:48
Design proposal for streaming APIs in data source V2 - Spark - [mail # dev]
...Hi all,https://docs.google.com/document/d/1VzxEuvpLfuHKL6vJO9qJ6ug0x9J_gLoLSH_vJL3-ChoI've finished a full design proposal for streaming APIs in data source V2,following up on my earlier doc...
   Author: Joseph Torres , 2018-05-24, 17:43
Design for continuous processing shuffle - Spark - [mail # dev]
...Hi all,A few of us have been working on a design for how to do shuffling incontinuous processing. Feel free to chip in if you have any comments orquestions.doc:https://docs.google.com/docume...
   Author: Joseph Torres , 2018-05-04, 18:27
[Structured streaming, V2] commit on ContinuousReader - Spark - [mail # user]
...In the master branch, we currently call this method inContinuousExecution.commit().Note that the ContinuousReader API is experimental and undergoing activedesign work. We will definitely inc...
   Author: Joseph Torres , 2018-05-03, 19:04
[expand - 3 more] - Datasource API V2 and checkpointing - Spark - [mail # dev]
...I agree that Spark should fully handle state serialization and recovery formost sources. This is how it works in V1, and we definitely wouldn't wantor need to change that in V2.* The questio...
   Author: Joseph Torres , 2018-05-01, 17:59
Transform plan with scope - Spark - [mail # dev]
...Is there some transformation we'd want to apply to that tree, but can'tbecause we have no concept of scope? It's already possible for a plan ruleto traverse each node's subtree if it wants.O...
   Author: Joseph Torres , 2018-04-24, 17:37
[expand - 1 more] - [discuss][data source v2] remove type parameter in DataReader/WriterFactory - Spark - [mail # dev]
...ContinuousReader and friends haven't yet gotten a design doc becausethey're significantly constrained by the (incomplete) implementationof SPARK-20928. Since continuous processing queries ne...
   Author: Joseph Torres , 2018-04-20, 20:53