Subject: Reading configuration file in Spark Scala throws error


Many thanks Chris.

In my Spark streaming I would like to use the config file to read the
parameters in. Taking your example, I have

       val globalConfig = ConfigFactory.load()
    val conf       = globalConfig.getConfig(sparkAppName)  // extract out
top level key from top level namespace
   val printEntry = new
java.util.function.Consumer[java.util.Map.Entry[String,com.typesafe.config.ConfigValue]]
{
    override def accept(a:
java.util.Map.Entry[String,com.typesafe.config.ConfigValue]): Unit = {
      val key = a.getKey
      val value = a.getValue.unwrapped
      //val value2 = conf.getAnyRef(a.getKey)
       println( s"$key = $value")
    }
  }
conf.entrySet.iterator.forEachRemaining (printEntry)
And my config file for spark is like below

 cat md_AerospikeAerospike.conf
md_AerospikeAerospike {
  dbDatabase = "trading"
  dbPassword = "mongodb"
  dbUsername = "trading_user_RW"
  bootstrapServers = "rhes75:9092, rhes75:9093, rhes75:9094, rhes564:9092,
rhes564:9093, rhes564:9094, rhes76:9092, rhes76:9093, rhes76:9094"
  schemaRegistryURL = "http://rhes75:8081"
  zookeeperConnect = "rhes75:2181, rhes564:2181, rhes76:2181"
  zookeeperConnectionTimeoutMs = "10000"
  rebalanceBackoffMS = "15000"
  zookeeperSessionTimeOutMs = "15000"
  autoCommitIntervalMS = "12000"
  topicsValue = "md"
  memorySet = "F"
  enableHiveSupport = null
  enableHiveSupportValue = "true"
  sparkStreamingReceiverMaxRateValue = "0"
  checkpointdir = "/checkpoint"
  mongodbHost = "rhes75"
  mongodbPort = "60100"
  zookeeperHost = "rhes75"
  zooKeeperClientPort = "2181"
  batchInterval = 2
  tickerWatch = "VOD"
  priceWatch = 300.0
  op_type = 1
  currency = "GBP"
  tickerType = "short"
  tickerClass = "asset"
  tickerStatus = "valid"
}

So I want them to be imported into Spark program.

Using the above println( s"$key = $value"), I get

zookeeperHost = rhes75
zookeeperSessionTimeOutMs = 15000
memorySet = F
topicsValue = md
currency = GBP
rebalanceBackoffMS = 15000
tickerStatus = valid
enableHiveSupportValue = true
autoCommitIntervalMS = 12000
mongodbPort = 60100
bootstrapServers = rhes75:9092, rhes75:9093, rhes75:9094, rhes564:9092,
rhes564:9093, rhes564:9094, rhes76:9092, rhes76:9093, rhes76:9094
zookeeperConnect = rhes75:2181, rhes564:2181, rhes76:2181
zookeeperConnectionTimeoutMs = 10000
dbUsername = trading_user_RW
dbPassword = mongodb
tickerWatch = VOD
tickerClass = asset
checkpointdir = /checkpoint
mongodbHost = rhes75
schemaRegistryURL = http://rhes75:8081
tickerType = short
zooKeeperClientPort = 2181
priceWatch = 300
batchInterval = 2
op_type = 1
sparkStreamingReceiverMaxRateValue = 0
dbDatabase = trading

Two things please. They are read in in a different order and secondly *the
String values are not quoted.** like currency = GBP as opposed to currency
= "GBP"*

What would be the easiest way of resolving the above please

Regards,

Dr Mich Talebzadeh

LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 4 Aug 2019 at 01:55, Chris Teoh <[EMAIL PROTECTED]> wrote: