clear query| facets| time Search criteria: .   Results from 1 to 10 from 16 (0.0s).
Loading phrases to help you
refine your search...
[structured-streaming][parquet] readStream files order in Parquet - Spark - [mail # user]
...My parquet files are first partitioned by environment and then by date like:env=testing/   date=2018-03-04/          part1.parquet      &nbs...
   Author: karthikjay , 2018-06-14, 13:59
[expand - 1 more] - [Beginner][StructuredStreaming] Using Spark aggregation - WithWatermark on old data - Spark - [mail # user]
...My data looks like this:{  "ts2" : "2018/05/01 00:02:50.041",  "serviceGroupId" : "123",  "userId" : "avv-0",  "stream" : "",  "lastUserActivity" : "00:02:50", ...
   Author: karthikjay , 2018-05-24, 15:20
[expand - 1 more] - [structured-streaming]How to reset Kafka offset in readStream and read from beginning - Spark - [mail # user]
...Chris,Thank you for responding. I get it. But, if I am using a console sink without checkpoint location, I do not seeany messages in the console in IntellijIDEA IDE. I do not explicitly spec...
   Author: karthikjay , 2018-05-23, 06:02
[Beginner][StructuredStreaming] Console sink is not working as expected - Spark - [mail # user]
...I have the following code to read and process Kafka data using StructuredStreaming   object ETLTest {  case class record(value: String, topic: String)  def main(args: Array...
   Author: karthikjay , 2018-05-23, 05:54
Spark job terminated without any errors - Spark - [mail # user]
...We have created multiples spark jobs (as far JAR) and run it usingspark-submit in a nohup mode. Most of the jobs quits after a while. We triedto harness the logs for failures but the only me...
   Author: karthikjay , 2018-05-18, 20:44
[structured-streaming] foreachPartition alternative in structured streaming. - Spark - [mail # user]
...I am reading data from Kafka using structured streaming and I need to savethe data to InfluxDB. In the regular Dstreams based approach I did this asfollows:      val messages:...
   Author: karthikjay , 2018-05-17, 20:30
[structured-streaming][kafka] Will the Kafka readstream timeout after connections.max.idle.ms 540000 ms ? - Spark - [mail # user]
...Hi all,We are running into a scenario where the structured streaming job is exitingafter a while specifically when the Kafka topic is not getting any data.From the job logs, I see this conne...
   Author: karthikjay , 2018-05-16, 01:25
[Structured-Streaming][Beginner] Out of order messages with Spark kafka readstream from a specific partition - Spark - [mail # user]
...On the producer side, I make sure data for a specific user lands on the samepartition. On the consumer side, I use a regular Spark kafka readstream andread the data. I also use a console wri...
   Author: karthikjay , 2018-05-10, 00:05
[beginner][StructuredStreaming] Null pointer exception - possible serialization errors. - Spark - [mail # user]
...I am getting a null pointer exception when trying to implement a connectionpooling mechanism in Apache Spark. Any help appreciated. https://stackoverflow.com/questions/50205650/spark-connect...
   Author: karthikjay , 2018-05-07, 00:17
[Structured Streaming] [Kafka] How to repartition the data and distribute the processing among worker nodes - Spark - [mail # user]
...Any help appreciated. please find the question in the link:https://stackoverflow.com/questions/49951022/spark-structured-streaming-with-kafka-how-to-repartition-the-data-and-distribu--Sent f...
   Author: karthikjay , 2018-04-20, 23:49