Hi Folks,

I have enabled below listed configurations within my spark streaming
application but I did not gain performance benefit even after setting these
parameters ,can you please help me is there a way to validate whether
vectorization is working as expeced/enabled correctly !

Note: I am using Spark 2.3 and converted all the data within my application
in orc format.

    sparkSqlCtx.setConf("spark.sql.orc.filterPushdown", "true")
    sparkSqlCtx.setConf("spark.sql.orc.enabled", "true")
    sparkSqlCtx.setConf("spark.sql.hive.convertMetastoreOrc", "true")
    sparkSqlCtx.setConf("spark.sql.orc.char.enabled", "true")
    sparkSqlCtx.setConf("spark.sql.orc.impl","native")
    sparkSqlCtx.setConf("spark.sql.orc.enableVectorizedReader","true")

Thanks,
Umar

--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [EMAIL PROTECTED]