Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ruler: add support for caching rule group contents #9386

Open
2 of 5 tasks
56quarters opened this issue Sep 23, 2024 · 0 comments
Open
2 of 5 tasks

Ruler: add support for caching rule group contents #9386

56quarters opened this issue Sep 23, 2024 · 0 comments
Assignees
@56quarters
Labels
component/ruler enhancement New feature or request

Comments

@56quarters
Copy link
Contributor

56quarters commented Sep 23, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.

Currently, the contents of rule groups for tenants are not cached. This means an API call to return all configuration for rule groups must perform a list call for object storage (this is sometimes cached) and then N additional object storage requests. For tenants with a large number of rule groups, this can result in a huge number of calls.

Most rule group configurations don't change very often. This makes them a good candidate for caching. However when they are changed, that needs to be reflected immediately in API results. So if we want to add caching for rule groups we also need a way to invalidate their cached contents.

Describe the solution you'd like

We should:

  • Add functionality to CachingBucket to invalidate cached object storage contents on mutations. (PR)
  • Unify "direct" and "cached' ruler storage backends in the Ruler so that cache invalidation works correctly. (PR)
  • Add further cache invalidation logic to prevent data races. (PR)
  • Configure ruler storage to use caching for Get operations
  • Increase the cache TTL for rule groups cached via Get operations to something longer than a single evaluation interval

Describe alternatives you've considered

Doing nothing is simpler and doesn't require opening up the can of worms that is cache invalidation.
@56quarters 56quarters added component/ruler enhancement New feature or request labels Sep 23, 2024
56quarters added a commit that referenced this issue Sep 23, 2024
@56quarters
Invalidate bucket caches on object modification or deletion
8768b33 
Invalidate content, existence, and attributes cached for a particular object
when the object is modified or deleted.

Part of #9386
@56quarters 56quarters mentioned this issue Sep 23, 2024
Merged
4 tasks
56quarters added a commit that referenced this issue Sep 24, 2024
@56quarters
Invalidate bucket caches on object modification or deletion (#9387)
df4cdfa 
Invalidate content, existence, and attributes cached for a particular object
when the object is modified or deleted.

Part of #9386
@56quarters 56quarters self-assigned this Sep 25, 2024
56quarters added a commit that referenced this issue Sep 26, 2024
@56quarters
Unify direct and caching RuleStores in ruler
3e4ad55 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386
@56quarters 56quarters mentioned this issue Sep 26, 2024
Merged
4 tasks
56quarters added a commit that referenced this issue Sep 26, 2024
@56quarters
Unify direct and caching RuleStores in ruler
f7039db 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386
56quarters added a commit that referenced this issue Sep 27, 2024
@56quarters
Unify direct and caching RuleStores in ruler
f3a051a 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386
56quarters added a commit that referenced this issue Sep 30, 2024
@56quarters
Unify direct and caching RuleStores in ruler
f6e69a2 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386

Signed-off-by: Nick Pillitteri <[email protected]>
56quarters added a commit that referenced this issue Sep 30, 2024
@56quarters
Unify direct and caching RuleStores in ruler
c1b4c85 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386

Signed-off-by: Nick Pillitteri <[email protected]>
@56quarters 56quarters mentioned this issue Oct 1, 2024
Open
56quarters added a commit to grafana/dskit that referenced this issue Oct 3, 2024
@56quarters
cache: Add .Set and .Add methods to cache clients
8b287e3 
This change adds a synchronous version of `.Set` to Memcached and Redis
clients as well as the various `Cache` wrapper implementations. This
allows callers to set a key and be sure it exists in the cache. This
change also adds an `.Add` method which conditionally adds an item to
the cache only if it does not already exist.

This change is a prerequisite for grafana/mimir#9386

Signed-off-by: Nick Pillitteri <[email protected]>
@56quarters 56quarters mentioned this issue Oct 3, 2024
Open
2 tasks
56quarters added a commit to grafana/dskit that referenced this issue Oct 3, 2024
@56quarters
cache: Add .Set and .Add methods to cache clients
2f51483 
This change adds a synchronous version of `.Set` to Memcached and Redis
clients as well as the various `Cache` wrapper implementations. This
allows callers to set a key and be sure it exists in the cache. This
change also adds an `.Add` method which conditionally adds an item to
the cache only if it does not already exist.

This change is a prerequisite for grafana/mimir#9386

Signed-off-by: Nick Pillitteri <[email protected]>
56quarters added a commit to grafana/dskit that referenced this issue Oct 3, 2024
@56quarters
cache: Add .Set and .Add methods to cache clients
ebcbf7b 
This change adds a synchronous version of `.Set` to Memcached and Redis
clients as well as the various `Cache` wrapper implementations. This
allows callers to set a key and be sure it exists in the cache. This
change also adds an `.Add` method which conditionally adds an item to
the cache only if it does not already exist.

This change is a prerequisite for grafana/mimir#9386

Signed-off-by: Nick Pillitteri <[email protected]>
56quarters added a commit that referenced this issue Oct 3, 2024
@56quarters
Unify direct and caching RuleStores in ruler
b91a3b3 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386

Signed-off-by: Nick Pillitteri <[email protected]>
56quarters added a commit that referenced this issue Oct 3, 2024
@56quarters
Unify direct and caching RuleStores in ruler
d6abaff 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386

Signed-off-by: Nick Pillitteri <[email protected]>
56quarters added a commit that referenced this issue Oct 4, 2024
@56quarters
Unify direct and caching RuleStores in ruler (#9434)
5c89d3e 
Instead of using two different `RuleStore` implementations within the Ruler,
use a single caching implementation and selectively disable caching when
required.

This change removes the "direct" `RuleStore` implementation from the Ruler's
gRPC and HTTP API layers. Instead, the caching implementation is used for all
calls. In cases where caching returning stale results would not be acceptable,
the caching is disabled _just_ for that call.

This allows rule group contents to be safety cached with the understanding
that it is safe to cache them because they will correctly invalidated when
deleted or modified.

Part of #9386

Signed-off-by: Nick Pillitteri <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@56quarters 56quarters

Labels
component/ruler enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@56quarters