Sorry I missed your reply. is it possible to stop the elasticsearch, do a full reboot without elasticsearch and see the `free -h`?
My best guest is that if ES has 3G and total used is `5.1GB` the you have about 2GB memory that is being used by the system itself. So you have to subtract that from 11GB and you will left out with 9/2 = `4.5GB`. Have you try this? What is the maximum heap memory you can have by try and fail? (like start with 3.5G and see where it crashes) Also, it could be due to the Host, sometimes you give the VM memory that the host doesn't have it. Something like vm ballooning where you don't expect all the VMs using all their CPU and memory so you give away x2 than you have available.
But, starting with clean system after boot with no ES running and see the free -h can help. Adding this result to what is the highest heap memory before it crashes will be the answer.
With what the system takes from memory I thought 4G should be OK. Then the only thing I can say is either system reserve some memory apart from what it uses which makes it around 4GB and it gives you less than 3.5G or the host is not really giving you 11G.