High memory utilization

[cuzzo333]

We're currently running Vault 1.6.0 in Google Cloud Run. At the time I inherited this deployment several weeks ago our Vault Cloud Run instance had 512MB of memory assigned and it would frequently hit this limit causing further requests hitting Vault to time out or otherwise fail.

In order to remediate the issue we've gradually increased the memory limit for our Vault service and it now sits at the Cloud Run limit of 8GB while at the same time actual memory utilization of Vault has increased and regularly sits at 85% utilization with frequent spikes to 99%, causing services to time out when making requests to Vault.

Having no prior experience with Vault I'm at a bit of a loss as to what to look for. As we're running on Google Cloud Run, I can't exec into or otherwise get into the Vault container to poke around.

We have a fairly low number of requests hitting Vault at any one time, perhaps 40 requests/min at max and many times it's near below 10 requests/min or near 0. I've also deleted the Vault Cloud Run service and deployed fresh and the issue persists.

At this point I'm not sure what else to look for. My first instinct was to try and figure out if Vault is logging locally and placing those logs into memory but that doesn't seem to be the case, at least as far as I can tell. I also don't see any other issues raised or bug fixes indicating there is or was any sort of memory leak.

I also can't recreate the issue by running a Vault container locally in something like Docker but it's also not an exact replication of the issue as I can't direct our dev/stg/prod systems to hit my local Vault container.

Thanks for any help!