Subject: How to retrieve multiple columns values (in one row) to variables in Spark Scala method


Hi,

Pretty basic question.

I use Spark on Hbase to retrieve the last  14 prices average and standard
deviation  for a security (ticker) from an Hbase table.

However, the call is expensive in Spark streaming where these values are
used to indicate buy and sell and subsequently the high value prices are
stored to MongoDB in the same spark streaming program.

I have defined these two methods to get the average and STDDEV of prices.

This method gets the average of prices

  def tickerAvg(dfHbase: org.apache.spark.sql.DataFrame, ticker: String):
Double = {
    var priceTicker = dfHbase.filter(col("ticker") ===
ticker).sort(col("timeissued").desc).
                                   limit(14).
                                   select(round(avg(col("price")),3)).
                                   collect.apply(0).getDouble(0)
    return priceTicker
  }
And this one STDDEV

  def tickerStddev(dfHbase: org.apache.spark.sql.DataFrame, ticker:
String): Double = {
    var priceTicker = dfHbase.filter(col("ticker") ===
ticker).sort(col("timeissued").desc).
                                   limit(14).
                                   select(round(stddev(col("price")),3)).
                                   collect.apply(0).getDouble(0)
    return priceTicker
  }

Note that each returns one value

However as these calls are expensive,  I want to return both AVG and STDDEV
in one call and from one method as follows
  def tickerGeneric(dfHbase: org.apache.spark.sql.DataFrame, ticker:
String): Double = {
    var priceTicker = dfHbase.filter(col("ticker") ===
ticker).sort(col("timeissued").desc).
                                   limit(14).

select(round(avg(col("price")),3),round(stddev(col("price")),3)).
                                   collect.?

   return ?
}

How this can be achieved to return both AVG and STDDEV values with one call
to method?

thanks

Dr Mich Talebzadeh

LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.