Subject: Query takes more time in Solr 8.5.1 compare to 6.1.0 version

Did you create Solrconfig.xml for the collection from scratch after upgrading and reindexing? Was it based on the latest template?
If not then please try this. Maybe also you need to increase the corresponding caches in the config.

What happens if you reexecute the query?

Are there other processes/containers running on the same VM?

How much heap and how much total memory you have? You should only have a minor fraction of the memory as heap and most of it „free“ (this means it is used for file caches).