[bug] The test runner could not access the Storybook channel

[IanVS]

Describe the bug
I'm not sure what can be done to fix this, but I've gotten an error in a single test, which was resolved upon a re-run of the job:

Message:
     The test runner could not access the Storybook channel. Are you sure the Storybook is running correctly in that URL?

      at __test (<anonymous>:230:17)
      at async <anonymous>:228:30
      at src/screens/loading/LoadingAuthPage.stories.tsx:1[68](https://github.com/xyz/abc/actions/runs/6732566520/job/18299559702?pr=2057#step:9:69):58
      at step (src/screens/loading/LoadingAuthPage.stories.tsx:109:23)
      at Object.next (src/screens/loading/LoadingAuthPage.stories.tsx:50:20)
      at asyncGeneratorStep (src/screens/loading/LoadingAuthPage.stories.tsx:4:28)
      at _next (src/screens/loading/LoadingAuthPage.stories.tsx:22:17)
      at src/screens/loading/LoadingAuthPage.stories.tsx:27:13
      at src/screens/loading/LoadingAuthPage.stories.tsx:19:16
      at testFn (src/screens/loading/LoadingAuthPage.stories.tsx:220:49)
      at Object.<anonymous> (src/screens/loading/LoadingAuthPage.stories.tsx:233:33)
      at step (src/screens/loading/LoadingAuthPage.stories.tsx:109:23)
      at Object.next (src/screens/loading/LoadingAuthPage.stories.tsx:50:20)
      at asyncGeneratorStep (src/screens/loading/LoadingAuthPage.stories.tsx:4:28)
      at _next (src/screens/loading/LoadingAuthPage.stories.tsx:22:17)
      at src/screens/loading/LoadingAuthPage.stories.tsx:27:13
      at Object.<anonymous> (src/screens/loading/LoadingAuthPage.stories.tsx:19:16)

To Reproduce
No idea, it's not something I've seen often, and appears to be intermittent. The story is nothing special: no args, no play function, just static content.

Expected behavior
I wouldn't expect flakey behavior from the test runner.

Screenshots
If applicable, add screenshots to help explain your problem.

System

  System:
    OS: macOS 14.0
    CPU: (12) arm64 Apple M2 Max
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.16.0 - ~/Library/Caches/fnm_multishells/73160_1698691812056/bin/node
    npm: 9.5.1 - ~/Library/Caches/fnm_multishells/73160_1698691812056/bin/npm
    pnpm: 8.8.0 - ~/Library/Caches/fnm_multishells/73160_1698691812056/bin/pnpm <----- active
  Browsers:
    Safari: 17.0
  npmPackages:
    @storybook/addon-a11y: 7.5.0 => 7.5.0
    @storybook/addon-essentials: 7.5.0 => 7.5.0
    @storybook/addon-interactions: 7.5.0 => 7.5.0
    @storybook/jest: ^0.2.3 => 0.2.3
    @storybook/react: 7.5.0 => 7.5.0
    @storybook/react-vite: 7.5.0 => 7.5.0
    @storybook/test-runner: ^0.13.0 => 0.13.0
    @storybook/testing-library: ^0.2.2 => 0.2.2
    @storybook/types: 7.5.0 => 7.5.0
    chromatic: ^6.17.3 => 6.24.1
    eslint-plugin-storybook: ^0.6.15 => 0.6.15
    storybook: 7.5.0 => 7.5.0

Additional context
This isn't probably immediately actionable, but I wanted to open an issue to see if others are hitting this as well, in which case we might want to look into why it could happen.

[yannbf]

Hey @IanVS thanks for opening this issue! I'll elaborate things a bit so others might have enough context when looking at this issue in the future.

The test-runner works by accessing the "iframe" URL of your Storybook, which has just a barebones, isolated version of Storybook with no stories, for instance like here. After that, it accesses Storybook specific APIs to select the story it needs to test. If it can't find Storybook APIs (in this case, the Storybook channel), it will error with that message you got.

First, it's important to know whether you're running the test-runner against a local Storybook or a deployed Storybook. It might also help to know whether this happens only in CI or not.

Second, this can mean a few things:

1. You are running the test-runner against a deployed Storybook that needs authentication, or some type of pre-navigation. Imagine the test-runner trying to access Storybook APIs, but in the login page of Chromatic, for instance.
2. You are running the test-runner against a URL that, for some reason, redirects you to another page. This would behave similarly than use case number 1.
3. You are running the test-runner against a local URL, that isn't actually Storybook. It's possible if you run apps/etc in similar port numbers
4. You ran Storybook (locally) and the test-runner at the same time, and the test-runner tried to access the Storybook URL before it was available. This could be a timing issue, and if this is the case and we find a way to reproduce this properly, we can try to improve the mechanism to retry or something like that.

If it's none of the above, I really would like to know how to reproduce this so we can fix it!

[IanVS]

Thanks Yann! I've only seen it in CI, but honestly I don't run the test-runner locally because I get very inconsistent failures and timeouts, so I nearly always push to CI and let it run there. Some other notes:

• It is built and running locally in CI, not against a deployed Storybook
• There are no redirects, the story this failed on was a dead-simple presentation-only story with a spinner.
• Not an issue with port conflict, and every other story in the run succeeded except this one.
• It was the last test that ran in the job (I shard across 6 GitHub runners). I wonder if the locally-running preview was shut down before the final test had a chance to complete.

These are the scripts I use to run the tests in CI:

"test:storybook:ci": "run-p --race test:storybook preview-storybook",
"test:storybook": "wait-on http://127.0.0.1:6006 && test-storybook --url http://127.0.0.1:6006 -- --shard=$SHARD",
"preview-storybook": "http-server storybook-static --port 6006 --silent",

I'll keep my eyes open for any further cases, and see if I can find any similarities.

[pedrosousa13]

I'm also getting this if I run test-runner with--browsers webkit it's mostly just happening on webkit.

I also get:

 FAIL   browser: webkit  stories/components/Tabs.stories.tsx
[TEST]   ● Test suite failed to run
[TEST]
[TEST]     page.goto: Timeout 30000ms exceeded.
[TEST]     =========================== logs ===========================
[TEST]     navigating to "http://127.0.0.1:6006/iframe.html", waiting until "load"
[TEST]     ============================================================
[TEST]
[TEST]       at defaultPrepare (node_modules/@storybook/test-runner/dist/index.js:13241:14)
[TEST]       at setupPage (node_modules/@storybook/test-runner/dist/index.js:13273:11)
[TEST]       at CustomEnvironment.setup (node_modules/@storybook/test-runner/playwright/custom-environment.js:8:5)
[TEST]
[TEST]  FAIL   browser: webkit  stories/components/Skeleton.stories.tsx (121.925 s)
[TEST]   ● Components/Skeleton › WithText › smoke-test
[TEST]
[TEST]     thrown: "Exceeded timeout of 30000 ms for a test.
[TEST]     Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."
[TEST]
[TEST]     > 1 | import React from 'react';
[TEST]         |

I tried with both a dev instance of storybook + a built one. this error was with the built one.
chromium works fine. I also increased the timeout but didn't help.

These stories have nothing special, just smoke-tests.

I'm going to try and set up a repo where I can replicate it but might take some time.

We use vite for our build, not sure if that helps.

[unional]

I also face similar issue as @pedrosousa13:

[TEST] FAIL packages/react-ui/src/components/Input/Input.stories.tsx
  ● Test suite failed to run
[TEST] 
    page.goto: Timeout 30000ms exceeded.
[TEST]     Call log:
      - navigating to "http://localhost:7006/iframe.html", waiting until "load"
[TEST] 
      at defaultPrepare (node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]_27kwgapvhdmhdh2inwdzlb3z3e/node_modules/@storybook/test-runner/dist/index.js:13241:14)
      at setupPage (node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]_27kwgapvhdmhdh2inwdzlb3z3e/node_modules/@storybook/test-runner/dist/index.js:13273:11)
      at CustomEnvironment.setup (node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]_27kwgapvhdmhdh2inwdzlb3z3e/node_modules/@storybook/test-runner/playwright/custom-environment.js:8:5)

I am also using vite.
This occurs when I am doing snapshot testing using jest-image-snapshot.

I suspect it is related to postVisit.
Does test-runner wait for the previous story's postVisit before starting a new one?
I suspect page has a promise queue and it got piled up.

In my case, I am running test-storybook with about 1k stories.

concurrently -k -s first -n \"SB,TEST\" -c \"magenta,blue\" \"storybook dev -p 7006 --ci\" \"wait-on http://localhost:7006 && test-storybook --url http://localhost:7006 -u

The snapshot test occurs in the postVisit, and all promises are awaited:

// simplified for illustrative purpose
(
  ...,
  async postVisit(page, context) {
    await ctx.waitForPageReady(page)
    const image = await page.screenshot()
    expect(image).toMatchImageSnapshot(...)

    await switchTheme(page, 'dark')
    const image = await page.screenshot()
    expect(image).toMatchImageSnapshot(...)
}

[unional]

could be related to #444