Subject: ram estimate for docvalues is incorrect

To be clear, there is no plan to remove RAM accounting from readers yet,
this is just something that I have been thinking about recently, so your
use-case caught my attention.

Given how low the memory usage is nowadays, I believe that it would be
extremely hard to make sure that RAM estimates are wrong by less than 2x in
most cases. This is due to the fact that everything counts now, so things
that we historically ignored for memory usage such as the maps you pointed
out can no longer be ignored. Unfortunately, these maps are not the only
thing that we ignored that we should no longer ignore: there are also field
infos, segment infos, BufferedIndexInput buffers, ... not to mention things
that we cache in threadlocals, as we would now need to count the number of
threads that have a cache entry.

I'd suggest looking into opportunities to better contain the number of
fields and segments as David suggested, and caching a fixed number of
readers at most instead of relying on RAM accounting.
On Wed, May 27, 2020 at 11:21 PM John Wang <[EMAIL PROTECTED]> wrote: