Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Another way for fstats to leak #131

Open
evanmcc opened this issue Jan 10, 2014 · 0 comments
Open

Another way for fstats to leak #131

evanmcc opened this issue Jan 10, 2014 · 0 comments

Comments

@evanmcc
Copy link
Contributor

evanmcc commented Jan 10, 2014

thanks to @joecaswell for finding this.

This leak is slightly hard to trigger, but certain pathological behavior patterns might trigger it.

Merge creates files, and thus creates fstats entries. These are not synchronized with the write thread in any way, but once a value within them is read, they're added to the write thread's read_files state, from where it is trimmed. So, to reproduce this, you need to have a file that is created by merge, never read from, and then merged away again.

The scenario where we saw this in the wild seemed to be:

  1. a few very hot keys in a partition
  2. a remainder of other values in file smaller than the small files threshold.
  3. since we have hot keys, the main writing file gets read as extremely fragmented, so when needs_merge comes around, both files are selected for the partial merge.
  4. a new merge file is created, containing the data from the small file, thus creating another small file.
  5. the cycle repeats, as nothing is ever added to the small file, other than the occasional value, since the hot keys represent most writes to the partition.
@slfritchie slfritchie added this to the 2.0-RC milestone Mar 24, 2014
@slfritchie slfritchie added the Bug label Mar 24, 2014
@slfritchie slfritchie added test and removed resilience labels May 1, 2014
@slfritchie slfritchie modified the milestones: 2.1, 2.0-RC May 1, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants