clear
query|
facets|
time |
Search criteria: author:"Mich Talebzadeh".
Results from 1 to 10 from
2342 (0.0s).
|
|
|
Loading phrases to help you refine your search...
|
[expand - 4 more]
[collapse]
-
Hive and Impala -
Hive - [mail # user]
|
...Just to clarify the statement in quotes was made by the author of the
article
"We can access all objects from Hive data warehouse with HiveQL which
leverages the map-reduce architecture... |
... in background for data retrieval and
transformation and this results in latency."
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw... |
...
*
http://talebzadehmich.wordpress.com
On 1 March 2016 at 11:33, Mich Talebzadeh wrote:
> I have not heard of Impala anymore. I saw an article in LinkedIn titled
>
> "Apache Hive Or Cloudera... |
....
>
> Anyway the question is with Hive on Spark or Spark using Hive metastore
> what we cannot achieve that we can achieve with Impala?
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https... |
|
|
|
|
Hive and Impala - Hive - [mail # user]
|
...] stages:
INFO : 2
INFO : 3
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com... |
... suffer
> from hive's performance even for ETL job. As result, we'll switch to
> implala + spark/ flink.
>
> Thanks,
> Dayong
>
> On Mar 2, 2016, at 10:35 AM, Mich Talebzadeh
> wrote:
>
> I... |
... forgot besides LLAP you are going to have Hive Hybrid Procedural SQL On
> Hadoop (HPL/SQL) which is going to add another dimension to Hive
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https... |
...://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 2 March 2016 at 15:30, Mich Talebzadeh
> wrote:
>
>> SQL plays... |
... is essentially a query tool.
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http... |
|
|
|
Hive and Impala - Hive - [mail # user]
|
...I forgot besides LLAP you are going to have Hive Hybrid Procedural SQL On
Hadoop (HPL/SQL) which
is going to add another dimension to Hive
Dr Mich Talebzadeh
LinkedIn * https... |
...://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 2 March 2016 at 15:30, Mich Talebzadeh wrote:
> SQL plays an increasing important... |
... tool.
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 2... |
... need for many other products in the same
>>> space. It is good to keep things simple.
>>>
>>> Warmest
>>>
>>>
>>> On Tuesday, 1 March 2016, 11:33, Mich Talebzadeh <
>>> mich.talebzadeh... |
... on Spark or Spark using Hive metastore
>>> what we cannot achieve that we can achieve with Impala?
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id... |
|
|
|
Hive and Impala - Hive - [mail # user]
|
... a query tool.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 2 March 2016 at 13... |
...
>>
>>
>> On Tuesday, 1 March 2016, 11:33, Mich Talebzadeh <
>> [EMAIL PROTECTED]> wrote:
>>
>>
>> I have not heard of Impala anymore. I saw an article in LinkedIn titled
>>
>> "Apache Hive Or Cloudera... |
....
>>
>> Anyway the question is with Hive on Spark or Spark using Hive metastore
>> what we cannot achieve that we can achieve with Impala?
>>
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn * https... |
|
|
|
Hive and Impala - Hive - [mail # user]
|
... with Impala?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
... |
|
|
|
|
|
[expand - 1 more]
[collapse]
-
SAS_TO_SPARK_SQL_(Could be a Bug?) -
Spark - [mail # user]
|
... double quotes. So simply
spark-sql> select prod_id, cust_id from sales limit 2;
17 28017
18 10419
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
...Assuming I understood your query, in Spark SQL (that is you log in to spark
sql like spark-sql --master spark://:7077 you do not need
double quotes around column names for sql to work... |
|
|
|
|
SAS_TO_SPARK_SQL_(Could be a Bug?) - Spark - [mail # user]
|
....
It is similar to your code but in Scala. Note that I do not enclose my
column names in double quotes.
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import... |
...)
println ("\nFinished at"); sqlContext.sql("SELECT
FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss')
").collect.foreach(println)
sys.exit()
}
}
Dr Mich Talebzadeh
LinkedIn * https... |
|
|
|
|
|
Does Spark support updates or deletes on underlying Hive tables -
Spark - [mail # user]
|
...Hi,
Remember that acidity and transactional support was added to Hive 0.14
onward because of advent of ORC tables.
Now Spark does not support transactions because quote "there is a piece... |
... in
the execution side that needs to send heartbeats to Hive metastore saying a
transaction is still alive". That has not been implemented in Spark yet to
my knowledge.
HTH
Dr Mich Talebzadeh
LinkedIn... |
|
|
|
|
[expand - 10 more]
[collapse]
-
Spark support for Complex Event Processing (CEP) -
Spark - [mail # user]
|
... is not a member of package org
import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
...---Stratio decision could do the job
> https://github.com/Stratio/Decision
>
> From: Alonso Isidoro Roman
> To: Mich Talebzadeh
> Cc: Mario Ds Briggs/India/IBM@IBMIN, Luciano Resende ,
> "user... |
... proceso de introducirlos..."
> - Edsger Dijkstra
>
> My favorite quotes (today):
> "If debugging is the process of removing software bugs, then programming
> must be the process of putting... |
... ..."
> - Edsger Dijkstra
>
> "If you pay peanuts you get monkeys"
>
>
> 2016-04-20 7:55 GMT+02:00 Mich Talebzadeh <*[EMAIL PROTECTED]*
> >:
>
> Thanks a lot Mario. Will have a look... |
....
>
> Regards,
>
>
> Dr Mich Talebzadeh
>
> LinkedIn
> *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw*
>
>
> *http://talebzadehmich.wordpress.com... |
|
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
...Hi Mario, I sorted that one out with Ted's help thanks
scalatest_2.11-2.2.6.jar
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
.../package/databricks/spark-csv
>
>
> thanks
> Mario
>
> [image: Inactive hide details for Mich Talebzadeh ---21/04/2016 08:34:08
> pm---Hi, Following example in]Mich Talebzadeh ---21/04/2016 08:34... |
...:08
> pm---Hi, Following example in
>
> From: Mich Talebzadeh
> To: Mario Ds Briggs/India/IBM@IBMIN
> Cc: Alonso Isidoro Roman , Luciano Resende <
> [EMAIL PROTECTED]>, "user @spark... |
...
> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
> LinkedIn
> *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw... |
...
> Isidoro Roman ---20/04/2016 02:24:39 pm---Stratio decision could do the job
> *https://github.com/Stratio/Decision*
>
>
> From: Alonso Isidoro Roman >
> To: Mich Talebzadeh... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
...Thanks a lot Mario. Will have a look.
Regards,
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
.../src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>
> Your feedback is appreciated.
>
>
> thanks
> Mario
>
> [image: Inactive hide details for Mich Talebzadeh... |
... ---19/04/2016 12:45:52
> am---great stuff Mario. Much appreciated. Mich]Mich Talebzadeh
> ---19/04/2016 12:45:52 am---great stuff Mario. Much appreciated. Mich
>
> From: Mich Talebzadeh... |
...)
> ------------------------------
>
>
>
> great stuff Mario. Much appreciated.
>
> Mich
>
> Dr Mich Talebzadeh
>
> LinkedIn
> *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw*
>
>
> *http... |
...
> Mario
>
> ----- Message from Mich Talebzadeh <*[EMAIL PROTECTED]*
> > on Sun, 17 Apr 2016 19:17:38 +0100 -----
>
> *To:*
> Luciano Resende >
>
> *cc:*
> "user @spark... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
...great stuff Mario. Much appreciated.
Mich
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
...://talebzadehmich.wordpress.com
On 18 April 2016 at 20:08, Mario Ds Briggs wrote:
> Hey Mich, Luciano
>
> Will provide links with docs by tomorrow
>
> thanks
> Mario
>
> ----- Message from Mich Talebzadeh on Sun... |
....
>
> Regards
>
> Dr Mich Talebzadeh
>
> LinkedIn
> *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw*
>
>
> *http://talebzadehmich.wordpress.com*
>
>
>
>
> On 17 April... |
... to reply here with more details.
>
> On Sun, Apr 17, 2016 at 1:54 AM, Mich Talebzadeh <
> *[EMAIL PROTECTED]* > wrote:
> Hi,
>
> Has Spark got libraries for CEP using Spark... |
... Mich Talebzadeh
>
> LinkedIn
> *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw*
>
>
> *http://talebzadehmich.wordpress.com... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
...Thanks Luciano. Appreciated.
Regards
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
... with more details.
>
> On Sun, Apr 17, 2016 at 1:54 AM, Mich Talebzadeh <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> Has Spark got libraries for CEP using Spark Streaming with Kafka by any... |
...
>> chance?
>>
>> I am looking at Flink that supposed to have these libraries for CEP but I
>> find Flink itself very much work in progress.
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
... work in progress.
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
... are sending data it is never truly
continuous. We are sending discrete messages.
HTH,
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
...=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 28 April 2016 at 17:22, Mich Talebzadeh
wrote:
> In a commerical (C)EP like say StreamBase, or for example its competitor
> Apama... |
... microseconds
> level can sometimes matter in financial trading but rarely.
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id... |
....
>>
>> HTH
>>
>>
>> On Apr 27, 2016, at 4:34 PM, Mich Talebzadeh
>> wrote:
>>
>> couple of things.
>>
>> There is no such thing as Continuous Data Streaming as there is no such
>> thing... |
..., Spark micro batching is perfectly adequate.
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
...
value and does not make commercial sense.
Now with regard to your needs, Spark micro batching is perfectly adequate.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view... |
..., 2016, at 7:03 AM, Mich Talebzadeh
> wrote:
>
> please see my other reply
>
> Dr Mich Talebzadeh
>
>
> LinkedIn *
> https://www.linkedin.com/profile/view?id... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
...please see my other reply
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com... |
|
|
|
Spark support for Complex Event Processing (CEP) - Spark - [mail # user]
|
... indeed true that latencies down to the single digit microseconds
level can sometimes matter in financial trading but rarely.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com... |
... streaming. It could also be done using half a dozen other tools
> because the rate of data generation is much slower than 500ms.
>
> HTH
>
>
> On Apr 27, 2016, at 4:34 PM, Mich Talebzadeh
> wrote... |
... is of little
> value and does not make commercial sense.
>
> Now with regard to your needs, Spark micro batching is perfectly adequate.
>
> HTH
>
> Dr Mich Talebzadeh
>
>
> LinkedIn *
> https... |
... a better potential for communication between
>> workers.
>> So here you can handle CEP events.
>>
>> HTH
>>
>> On Apr 27, 2016, at 7:03 AM, Mich Talebzadeh <
>> [EMAIL PROTECTED]> wrote... |
...:
>>
>> please see my other reply
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>> http... |
|
|
|
|
|
[expand - 42 more]
[collapse]
-
Using Spark on Hive with Hive also using Spark as its execution engine -
Hive - [mail # user]
|
...I just read further notes on LLAP.
As Gopal explained LLAP has more to do that just in-memory and I quote
Gopal:
"... LLAP is designed to be hammered by multiple user sessions running... |
... Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all... |
... be liable for any monetary damages arising from
such loss, damage or destruction.
On 12 July 2016 at 15:59, Mich Talebzadeh wrote:
... |
|
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
...Thanks Alan. Point taken.
In mitigation, here are members in Spark forum who have shown (interest) in
using Hive directly and I quote one:
"Did you have any benchmark for using Spark... |
... Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk... |
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
... Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 31 May 2016 at 21:48, Gopal Vijayaraghavan... |
|
|
|
答复: 答复: Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
...Wjich version of Hive and Spark please?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
...I suggest that you try it for yourself then
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
|
|
|
答复: 答复: 答复: Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
...fine which version of spark are using for Hive execution/query engine
please?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
...thanks Marcin.
What Is your guesstimate on the order of "faster" please?
Cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Spark - [mail # user]
|
... partitions etc. The rest is cold
data.
cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
.... You can have both.
>
> On May 30, 2016, at 12:49 PM, Mich Talebzadeh
> wrote:
>
> yep Hortonworks supports Tez for one reason or other which I am going
> hopefully to test it as the query... |
... to support another stack like Tez.
>
> Cloudera support Impala instead of Hive but it is not something I have
> used. .
>
> HTH
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com... |
... execution engine?
>> Then LLAP is the in-memory piece to speed up Tez?
>>
>> HTH
>>
>> -Mike
>>
>> On May 29, 2016, at 1:35 PM, Mich Talebzadeh
>> wrote:
>>
>> thanks I think the problem... |
... is that the TEZ user group is exceptionally
>> quiet. Just sent an email to Hive user group to see anyone has managed to
>> built a vendor independent version.
>>
>>
>> Dr Mich Talebzadeh... |
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Hive - [mail # user]
|
... partitions etc. The rest is cold
data.
cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
.... You can have both.
>
> On May 30, 2016, at 12:49 PM, Mich Talebzadeh
> wrote:
>
> yep Hortonworks supports Tez for one reason or other which I am going
> hopefully to test it as the query... |
... to support another stack like Tez.
>
> Cloudera support Impala instead of Hive but it is not something I have
> used. .
>
> HTH
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com... |
... execution engine?
>> Then LLAP is the in-memory piece to speed up Tez?
>>
>> HTH
>>
>> -Mike
>>
>> On May 29, 2016, at 1:35 PM, Mich Talebzadeh
>> wrote:
>>
>> thanks I think the problem... |
... is that the TEZ user group is exceptionally
>> quiet. Just sent an email to Hive user group to see anyone has managed to
>> built a vendor independent version.
>>
>>
>> Dr Mich Talebzadeh... |
|
|
|
Using Spark on Hive with Hive also using Spark as its execution engine - Spark - [mail # user]
|
... but it is not something I have
used. .
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 30 May... |
... PM, Mich Talebzadeh
> wrote:
>
> thanks I think the problem is that the TEZ user group is exceptionally
> quiet. Just sent an email to Hive user group to see anyone has managed to
> built... |
... a vendor independent version.
>
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
> http... |
..., Mich Talebzadeh
>> wrote:
>>
>> Hi Jorn,
>>
>> I started building apache-tez-0.8.2 but got few errors. Couple of guys
>> from TEZ user group kindly gave a hand but I could not go very far... |
... of
>> Hive etc as I am sure you already know.
>>
>> Cheers,
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw... |
|
|
|
|
|
Escaped Quotes -
Hive - [mail # user]
|
...' Diner");
0: jdbc:hive2://rhes564:10010/default> select * from testme;
+---------------+--+
| testme.col1 |
+---------------+--+
| Dougs' Diner |
+---------------+--+
Dr Mich Talebzadeh... |
... use JDBC and the beeline client to interact with a Hive 1.2.1 store.
>
> I have some string data that contains single quotes (as well as other
> special characters) that requires being escaped... |
... in order to be inserted
> into my tables. For example, the string "Doug's Diner":
>
> INSERT INTO my_table (some_col) VALUES ('Doug\'s Diner')
>
> What I am finding is that the single quote inside... |
|
|
|
|
[expand - 1 more]
[collapse]
-
Parquet versus ORC -
Hive - [mail # user]
|
...Hi,
Thanks for that link.
It appears that the main advantages of Parquet is stated as and I quote:
"Parquet is built to be used by anyone. The Hadoop ecosystem is rich with
data... |
... Dec 15) and I quote
"Whether ORC is the best format for what you're doing depends on the data
you're storing and how you are querying it. If you are storing data where
you know the schema... |
... database does).
Cheers.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 6 March... |
... supports in the
> docs http://parquet.apache.org/documentation/latest/
>
>
> n 06/03/2016 15:26, Mich Talebzadeh wrote:
>
> Hi.
>
> I have been hearing a fair bit about Parquet versus ORC tables... |
..., Parquet etc.
>
> Appreciate any verification or experience on this.
>
> Thanks
> ,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id... |
|
|
|
|
Parquet versus ORC - Hive - [mail # user]
|
... and don't
really see any reason why I should adopt others like Avro, Parquet etc.
Appreciate any verification or experience on this.
Thanks
,
Dr Mich Talebzadeh
LinkedIn * https... |
|
|
|
|
|
[expand - 4 more]
[collapse]
-
Efficient filtering on Spark SQL dataframes with ordered keys -
Spark - [mail # user]
|
...
and I quote
"registerTempTable()
registerTempTable() creates an in-memory table that is scoped to the
cluster in which it was created. The data is stored using Hive's
highly-optimized... |
..., in-memory columnar format."
So on the face of it tempTable is an in-memory table
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
|
|
|
|
[expand - 1 more]
[collapse]
-
where clause on Phoenix view built on Hbase table throws error -
HBase - [mail # user]
|
...-Apr-08 | 49664486 |
+-----------+-----------+
1 row selected (0.016 seconds)
BTW I believe double quotes in enclosing phoenix column names are needed
for case sensitivity on Hbase?
Also... |
... does Phoenix have type conversion from VARCHAR to integer etc? Is
there such document
Regards
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
|
|
|
|
where clause on Phoenix view built on Hbase table throws error - HBase - [mail # user]
|
...)
org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
Undefined column. columnName=1-Apr-08
Why does it think a predicate "1-Apr-08" is a column.
Any ideas?
Thanks
Dr Mich Talebzadeh
LinkedIn * https... |
|
|
|
|
|
[expand - 4 more]
[collapse]
-
Spark streaming issue -
Spark - [mail # user]
|
...)]
= org.apache.spark.streaming.kafka.DirectKafkaInputDStream@6d2d3b21
So that opens data stream. What next?
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http... |
...://talebzadehmich.wordpress.com
On 1 April 2016 at 22:37, Mich Talebzadeh wrote:
> yes I noticed that
>
> scala> val kafkaStream = KafkaUtils.createStream(ssc, "rhes564:2181",
> "rhes564:9092", "newtopic", 1... |
...:2181",
> "rhes564:9092", "newtopic", 1)
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http... |
...://talebzadehmich.wordpress.com
>
>
>
> On 1 April 2016 at 22:25, Cody Koeninger wrote:
>
>> You're not passing valid Scala values. rhes564:2181 without quotes
>> isn't a valid literal, newtopic isn't a list... |
... of strings, etc.
>>
>> On Fri, Apr 1, 2016 at 4:04 PM, Mich Talebzadeh
>> wrote:
>> > Thanks Cody.
>> >
>> > Can I use Receiver-based Approach here?
>> >
>> > I have created the topic newtopic... |
|
|
|
|
Spark streaming issue - Spark - [mail # user]
|
...,'BLIi9a_n7Pfyc7r3nfzKfaNRa4Hmd9NlHEVDPkQS4xbgUWqU2bJeI6b8b1IMoStnmjMHhYLtFf4TQyJcpn85PSwFksggNVnQl1oL',105)
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 1 April 2016 at 23... |
...:26, Mich Talebzadeh wrote:
> I adopted this approach
>
> scala> val conf = new SparkConf().
> | setAppName("StreamTest").
> | setMaster("local[12... |
...: org.apache.spark.streaming.dstream.InputDStream[(Nothing,
> Nothing)] =
> org.apache.spark.streaming.kafka.DirectKafkaInputDStream@6d2d3b21
>
> So that opens data stream. What next?
>
> Thanks
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn... |
... * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 1 April 2016 at 22:37, Mich Talebzadeh
> wrote... |
... to (org.apache.spark.streaming.StreamingContext,
>> String, String, String, Int)
>> val kafkaStream = KafkaUtils.createStream(ssc, "rhes564:2181",
>> "rhes564:9092", "newtopic", 1)
>>
>>
>> Dr Mich Talebzadeh... |
|
|
|
Spark streaming issue - Spark - [mail # user]
|
... kafkaStream = KafkaUtils.createStream(ssc, "rhes564:2181",
"rhes564:9092", "newtopic", 1)
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id... |
...=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 1 April 2016 at 22:25, Cody Koeninger wrote:
> You're not passing valid Scala values. rhes564:2181 without quotes
> isn't a valid literal... |
..., newtopic isn't a list of strings, etc.
>
> On Fri, Apr 1, 2016 at 4:04 PM, Mich Talebzadeh
> wrote:
> > Thanks Cody.
> >
> > Can I use Receiver-based Approach here?
> >
> > I have created... |
...,
> > rhes564:2181, rhes564:9092, newtopic 1)
> >
> >
> >
> >
> >
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn
> >
> https://www.linkedin.com/profile/view?id... |
... to connect to a
> >> zookeeper port, which won't work.
> >>
> >> Look at spark.apache.org/docs/latest/streaming-kafka-integration.html
> >>
> >> On Fri, Apr 1, 2016 at 3:03 PM, Mich Talebzadeh... |
|
|
|
Spark streaming issue - Spark - [mail # user]
|
... Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*
http://talebzadehmich.wordpress.com
On 1 April 2016 at 21:13, Cody... |
...
>
> On Fri, Apr 1, 2016 at 3:03 PM, Mich Talebzadeh
> wrote:
> >
> > Hi,
> >
> > I am just testing Spark streaming with Kafka.
> >
> > Basically I am broadcasting topic every minute to Host:port... |
... got the port wrong in this case or the set up is incorrect?
> >
> >
> > Thanks
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn
> >
> https://www.linkedin.com/profile/view?id... |
|
|
|
Spark streaming issue - Spark - [mail # user]
|
...> -------------------------------------------
Time: 1459541760000 ms
-------------------------------------------
But no values
Have I got the port wrong in this case or the set up is incorrect?
Thanks
Dr Mich Talebzadeh
LinkedIn... |
|
|
|
|
|
|