Update flakiness script #30
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
outofambit
commented
Aug 28, 2024
•
outofambit
commented
- removed checking for empty responses
- use a more markdown centric output format
- group output by test ids
- use a most common set of responses as the comparison set, instead of just using the first set of responses
* WIP: markdown reformatting progress * fix for header section * some changes based on carmen feedback * Total unequal % - carmen request * remove old code comment * remove old code comment * link to header * adding some additional notes to stress test readme * add notes about using a personal non-network fork * simplify join in formatResponses * remove old settle tracking we got this from p-limit * Update stressor/stress-test.mts * Apply suggestions from code review Co-authored-by: Mx Corey Frang <[email protected]> --------- Co-authored-by: cypress evelyn masso <[email protected]>
|
@jugglinmike if you would take a look at this, that would be great! @gnarf already took a look and added a lot of great improvements via #34. |
jugglinmike
requested changes
Oct 2, 2024
|
|
||
| Running the script can take a while, as it is constrained by GitHub Actions availability and speed. | ||
| 1. It is prefered for you to run the stress test against your own personal "non-fork" of this repo (create a personal repo and push to it instead of using "fork" so it isn't part of the "network") to limit the number of action runs against the main branch. | ||
| 2. Update the stress-test.mts file `owner`, `repo`, and `defaultBranch` definitintions near the top, as well as setting up the tests / matrix you want to test. |
There was a problem hiding this comment.
Suggested change
| 2. Update the stress-test.mts file `owner`, `repo`, and `defaultBranch` definitintions near the top, as well as setting up the tests / matrix you want to test. | |
| 2. Update the stress-test.mts file `owner`, `repo`, and `defaultBranch` definitions near the top, as well as setting up the tests / matrix you want to test. |
| Running the script can take a while, as it is constrained by GitHub Actions availability and speed. | ||
| 1. It is prefered for you to run the stress test against your own personal "non-fork" of this repo (create a personal repo and push to it instead of using "fork" so it isn't part of the "network") to limit the number of action runs against the main branch. | ||
| 2. Update the stress-test.mts file `owner`, `repo`, and `defaultBranch` definitintions near the top, as well as setting up the tests / matrix you want to test. | ||
| 3. Run it with `npm run --slient stress-test | tee some-output-file.md`. |
There was a problem hiding this comment.
Suggested change
| 3. Run it with `npm run --slient stress-test | tee some-output-file.md`. | |
| 3. Run it with `npm run --silent stress-test | tee some-output-file.md`. |
| return testCombosForTestPlan.map(spawnAndCollectWorkflows); | ||
| })); | ||
|
|
||
| clearInterval(logStatusInterval); |
There was a problem hiding this comment.
Mind putting this in a finally block?
Comment on lines
+326
to
+328
| const isRowEqual = | ||
| JSON.stringify(resultResponses) === | ||
| JSON.stringify(baselineResponses); |
There was a problem hiding this comment.
Object serialization semantics being what they are, this seems a bit fragile to me. What do you think of writing a comparator function for this?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.