-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmark runner instantiates annotators #510
Conversation
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
@@ -36,26 +36,35 @@ def fake_all_secrets(value="some-value") -> RawSecrets: | |||
return raw_secrets | |||
|
|||
|
|||
class FakeExplodingAnnotator(FakeAnnotator): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pollutes a global variable with test-specific data, and as a general rule tests should not have side effects.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you referring to the registration of the fake annotators? If so, I agree that it's problematic. However, tests' get_annotators
just returns UIDs and then the runners get the actual annotators from the registry now.
Do you have suggestions to avoid test side effects on ANNOTATORS while still testing the runners' annotator-specific functionality?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, I prefer to avoid global variables, which neatly solves this problem. But since fixing that is probably out of scope for this, I think the next-best solution is either setup/teardown (which add the fakes and then restore state after the test has run) or mocking such that the test diverts the production code so that it uses a fake registry which is discarded after the test has run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense! Updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all looks great now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Looks great!
PR #487 did not apply its test annotator instantiation change to modelbench's runner. This fixes that.
I also snuck in a bug fix for the "sxc" prompt issue that was causing some tests to fail.