videobuf2_common: driver bug: stop_streaming operation is leaving buf 0b70dd47 in active state #3791
Comments
|
Hopefully solved by #3790 |
|
Fix should be in latest rpi-update kernel. |
|
Thank you! I updated to 5.4.58-v7+ #1335 SMP Thu Aug 13 22:17:06 BST 2020 armv7l GNU/Linux and will let you known if it appears again. |
|
I've being using 5.4.58-v7+ for more than a week, it looks good, so I'm closing this issue. Thank you! |
|
I think the problem is still there. My use case is slightly different. I am playing a youtube video in the wpewebkit browser, and still reaching these messages in dmesg :( Sorry.. |
|
:-( @timemaster5, it's still good here so far. Anyway I reopened the issue and would be nice if you could give more information for the devs, like your dmesg and "uname -a " output. Maybe there's something in your output that could help. |
|
hm, maybe it has multiple causes.. here is the dmesg. The second line is uname -a basically. Hope that helps. |
Merged
|
there has been some progress in suspiciously similar issue #3325 (comment), is it helpful? |
|
@timemaster5 This might be fixed in firmware release edf2e9c318863999c97c50cdb74eee235ede3af5 which will hopefully reach your Pi with a regular update soon. |
|
@hailfinger thank you for letting me know.. I use a custom distribution build with Yocto, so will try to pull this version manually |
|
however, struggling with finding this.. is it a commit id in the raspberrypi/firmware repo? |
|
That's probably the build version (commit hash for the source code) - the correct hash for the firmware repo is raspberrypi/firmware@20081d8, or Hexxeh/rpi-firmware@c0cf93a for rpi-update. |
|
Facing the same issue here, the system crashes when streaming video via any browser. Sometimes there is no log available. This is from my crash: uname: firmware: FWIW, I've tested the aarch64 (beta) release from RaspiOS and the issue is still there. EDIT: I've just faced this crash just by merely opening Midori (without loading any page). |
|
I'm still getting similar error while decoding h264 from IP cameras using gstreamer, no luck so far. Kernel "4.19.118-v7+ #1311 SMP Mon Apr 27 14:21:24 BST 2020" seems stable, I'm not hitting this problem after several weeks. I'll upgrade to Hexxeh/rpi-firmware@c0cf93a as soon as possible. |
|
Do we have any progress or clues here? Or even workaround from the others. Last week, I did some tests with the userland driver and was able to play videos in a loop without significant issues. Nothing in the kernel log at least :) Only GStreamer had something to tell, but I guess it wasn't substantial enough to break something. I tried even the newest bullseye firmware, but the situation is the same. The only noticeable difference between the old kernel 4.19 and 5.4 and 5.10 was that in 4.19, after the video crashed, the new one didn't start. On the newer kernel, the new loop starts playing, but kernel log keeps growing of those messages :( |
|
Try adding Despite claims to the contrary, these buffers will accumulate (every time the error message gets repeated, the reported number of buffers increases) and eventually playing videos will fail. Since my devices are in production, those which cannot be fixed even with over_voltage=6 get regular scheduled reboots. That's a horrible workaround, but at least it's a workaround. |
|
Hi @hailfinger, thank you for your response. I found your thread somewhere else some time ago and immediately thought, hey, that's my missing bit, but it wasn't :( I tried over_voltage up to 6, then even more, but the problem is still there. And it is immediate. I mean, this error log appears the first time the video plays and then with every loop. I don't need to wait hours for it :( That is why I am quite surprised no one can sort it. If I'm correct, you were on raspberry pi 4, is that right?. Here I am on the Compute Module 3+. |
|
Ah yes, that might explain the difference. I'm on a Raspberry Pi 4B. |
|
I believe it is an issue with the hardware itself. All is fine with userland on CM3+ or Rpi3b+, so the decoder is not "broken", but as userland is the closed source, I guess the vc4 driver does something potentially right from the general point of view, but BCM requires something specific, maybe more non-standard to work correctly in this case. The reason why this is haven't been fixed, I guess, is because of this difference in board versions. Many people successfully resolved this issue with the newer firmware release, but they were on Rpi 4. On Rpi 3, this has never been fixed and maybe not even looked into. |
It's not.
The V4L2 driver uses exactly the same parts of the firmware, but there is a path that I believe I know what the issue is, but haven't had time to investigate (partly as I don't have a reliable route to reproduce). I've said it multiple times, |
I am sorry for the misunderstanding. The context is further in the following sentence. I believe it is hardware-specific, not a hardware bug, just a specific way we need to treat BCM here. I don't know much about documentation available for this specific bit, but in general, I know it is let's say more problematic with Broadcom.
I have to disagree here. This cause system lockup and the only variable is time. If you have a short 10s video you want to play in a loop, the number of buffers left in the active state increase every 10s. It is worse on the 4.19 kernel, where the new video loop doesn't even start, and you can't reboot the system to fix it because it will not boot again. The only way to fix it is to power cycle. It is much better on 5.4. and up, where the video can play, but at some point, the number of buffers left is just too high. I don't know whether the memory runs out or there is no new buffer available for the next video loop.
I am happy to help. If you want, I can build you an image with all the tools you need and some example video so you can just flash and immediately see what is wrong. |
|
Same on |
|
Facing the same issue when looping a 30s video from WPEWebkit on latest kernel 6.1.13-v7l. |
|
I was experiencing crashes streaming videos from Youtube to the "Tubecast" app on Kodi. Both from the android Youtube App and Chromium's 'cast' feature. I get this error message repeating in syslog when casting from Chromium, right before the system becomes unresponsive. When I stream from the youtube app on android, it gives stuff like this in syslog: and here is another one Eventually the system will crash, which doesn't show any helpful messages, but does give a lot of this: "@^@^@^@^@^@^" |
|
I found a similar issue. kernel 6.1.21-v8+ on RPi 4B |
|
Still have the issue as well on latest kernel 6.1.51-v8 with gstreamer 1.22.2 and wpewebkit 2.40.3. |
|
The warnings are just that - warnings, not errors. If you have an issue, they are symptoms and not the cause. |
|
@6by9 Sorry I wasn't specific enough, the video still plays but after a few hours it stops with the following error: |
|
|
|
After some verification I've noticed that the allocated GPU memory was only 78Mb. There's some confusion on how to configure it for KMS driver. Config.txt: I thought "cma-256" would dynamically allow memory to the GPU when needed but it seems not the case. |
|
The default gpu_mem setting should be good for Level 4.1 H.264. |
|
Only one stream, here are the specs: I'll have a look to compile vclog to the target device |
|
Is there anything your code is doing that may allocate additional gpu_mem? Are you confident increasing gpu_mem really fixes the issue completely (rather than making the failure take ~4x longer to occur?) |
|
Hi, better late than never. We are still facing the issue with small length videos. When looped continuously seem to cause the buffer not to clean up correctly, eventually leading to a system freeze after several hours. Tested on latest kernel 6.6.30-v8 on RPI3, RPI3B+, RPI4. Below is the Python script used to instantiate a playlist of videos in a loop: The media_playlist.json contains 2 videos: Here is a sample of the kernel message logged at each video stop/start. And here is vclog output from boot till +2 hours of video loop: |
|
I managed to retrieve some kernel messages ater the crash/freeze. Unfortunately It's not complete (missing the exact crash timestamp).: This message is repeating itself every minute: |
|
Could we please find a solution for this issue? Looping videos that are at least 15 seconds each does seem to stabilize things somewhat, but we’re still seeing numerous warnings in the logs. This makes it quite challenging to read and diagnose other potential issues. Any assistance would be greatly appreciated! |
|
The WARN triggered by GStreamer (and other apps using VIDIOC_CREATE_BUFS) has been fixed with #6348 (merged last week). |
|
Thank you @6by9 ! Testing now and indeed the message is gone, no issue so far ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I hit this "driver bug" after stopping an IP camera stream that was left open overnight. The software uses OpenCV to read camera frames using a gstreamer pipeline like this: "rtspsrc protocols=tcp tcp-timeout=2000000 retry=2 do-retransmission=false location=rtsp://192.168.42.10:554/user=admin&password=&channel=1&stream=0.sdp ! rtpjitter
buffer ! rtph264depay ! queue ! h264parse ! v4l2h264dec ! queue ! appsink sync=1"
Model: Raspberry Pi 3B
$ cat /etc/rpi-issue
Raspberry Pi reference 2020-05-27
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 825107f04027269db77426046f5085475b1ea22f, stage2
$ vcgencmd version
Jul 17 2020 11:04:27
Copyright (c) 2012 Broadcom
version 21a15cb094f41c7506ad65d2cb9b29c550693057 (clean) (release) (start)
$ uname -a
Linux br0001 5.4.51-v7+ #1327 SMP Thu Jul 23 10:58:46 BST 2020 armv7l GNU/Linux
[Thu Aug 13 10:12:33 2020] bcm2835-codec bcm2835-codec: bcm2835_codec_stop_streaming: Timeout waiting for buffers to be returned - 11 outstanding
[Thu Aug 13 10:12:33 2020] ------------[ cut here ]------------
[Thu Aug 13 10:12:33 2020] WARNING: CPU: 0 PID: 13924 at drivers/media/common/videobuf2/videobuf2-core.c:1882 __vb2_queue_cancel+0x210/0x288 [videobuf2_common]
[Thu Aug 13 10:12:33 2020] Modules linked in: cmac bnep hci_uart btbcm bluetooth ecdh_generic ecc 8021q garp stp llc brcmfmac brcmutil sha256_generic libsha256 cfg80211 raspberrypi_hwmon rfkill i2c_bcm2835 bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) v4l2_mem2mem snd_bcm2835(C) bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops snd_pcm videobuf2_v4l2 videobuf2_common snd_timer videodev snd mc vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables ipv6 nf_defrag_ipv6
[Thu Aug 13 10:12:33 2020] CPU: 0 PID: 13924 Comm: python Tainted: G WC 5.4.51-v7+ #1327
[Thu Aug 13 10:12:33 2020] Hardware name: BCM2835
[Thu Aug 13 10:12:33 2020] Backtrace:
[Thu Aug 13 10:12:33 2020] [<8010d458>] (dump_backtrace) from [<8010d750>] (show_stack+0x20/0x24)
[Thu Aug 13 10:12:33 2020] r6:8bcf6000 r5:00000000 r4:80d93ff4 r3:82a41469
[Thu Aug 13 10:12:33 2020] [<8010d730>] (show_stack) from [<808b2324>] (dump_stack+0xe0/0x124)
[Thu Aug 13 10:12:33 2020] [<808b2244>] (dump_stack) from [<8011fd24>] (__warn+0xec/0x104)
[Thu Aug 13 10:12:33 2020] r8:0000075a r7:00000009 r6:7f0ca7c4 r5:00000000 r4:00000000 r3:82a41469
[Thu Aug 13 10:12:33 2020] [<8011fc38>] (__warn) from [<8011fdf4>] (warn_slowpath_fmt+0xb8/0xc0)
[Thu Aug 13 10:12:33 2020] r9:7f0ca7c4 r8:0000075a r7:7f0c519c r6:00000009 r5:00000000 r4:80d04f48
[Thu Aug 13 10:12:33 2020] [<8011fd40>] (warn_slowpath_fmt) from [<7f0c519c>] (__vb2_queue_cancel+0x210/0x288 [videobuf2_common])
[Thu Aug 13 10:12:33 2020] r9:afa8b40c r8:40045613 r7:7f165ee0 r6:afa8b40c r5:00000009 r4:afa8b40c
[Thu Aug 13 10:12:33 2020] [<7f0c4f8c>] (__vb2_queue_cancel [videobuf2_common]) from [<7f0c6b00>] (vb2_core_streamoff+0x44/0xb4 [videobuf2_common])
[Thu Aug 13 10:12:33 2020] r10:b1bd27c8 r9:00000020 r8:40045613 r7:7f165ee0 r6:afa8b40c r5:00000009
[Thu Aug 13 10:12:33 2020] r4:afa8b40c r3:00000009
[Thu Aug 13 10:12:33 2020] [<7f0c6abc>] (vb2_core_streamoff [videobuf2_common]) from [<7f0e6084>] (vb2_streamoff+0x40/0x60 [videobuf2_v4l2])
[Thu Aug 13 10:12:33 2020] r4:afa8b400 r3:00000000
[Thu Aug 13 10:12:33 2020] [<7f0e6044>] (vb2_streamoff [videobuf2_v4l2]) from [<7f1d8c08>] (v4l2_m2m_streamoff+0x3c/0xf4 [v4l2_mem2mem])
[Thu Aug 13 10:12:33 2020] [<7f1d8bcc>] (v4l2_m2m_streamoff [v4l2_mem2mem]) from [<7f1d8ce0>] (v4l2_m2m_ioctl_streamoff+0x20/0x24 [v4l2_mem2mem])
[Thu Aug 13 10:12:33 2020] r10:b1bd27c8 r9:00000020 r8:40045613 r7:7f165ee0 r6:80d04f48 r5:00000000
[Thu Aug 13 10:12:33 2020] r4:b1bd24a0 r3:97229000
[Thu Aug 13 10:12:33 2020] [<7f1d8cc0>] (v4l2_m2m_ioctl_streamoff [v4l2_mem2mem]) from [<7f165f08>] (v4l_streamoff+0x28/0x2c [videodev])
[Thu Aug 13 10:12:33 2020] [<7f165ee0>] (v4l_streamoff [videodev]) from [<7f16acb4>] (__video_do_ioctl+0x240/0x470 [videodev])
[Thu Aug 13 10:12:33 2020] [<7f16aa74>] (__video_do_ioctl [videodev]) from [<7f16cd54>] (video_usercopy+0x220/0x60c [videodev])
[Thu Aug 13 10:12:33 2020] r10:7f16aa74 r9:40045613 r8:00000000 r7:8bcf7e04 r6:00000004 r5:00000000
[Thu Aug 13 10:12:33 2020] r4:80d04f48
[Thu Aug 13 10:12:33 2020] [<7f16cb34>] (video_usercopy [videodev]) from [<7f16d160>] (video_ioctl2+0x20/0x24 [videodev])
[Thu Aug 13 10:12:33 2020] r10:00000036 r9:8bcf6000 r8:00000026 r7:8e89a540 r6:b1371690 r5:021f1200
[Thu Aug 13 10:12:33 2020] r4:80d04f48
[Thu Aug 13 10:12:33 2020] [<7f16d140>] (video_ioctl2 [videodev]) from [<7f164144>] (v4l2_ioctl+0x4c/0x60 [videodev])
[Thu Aug 13 10:12:33 2020] [<7f1640f8>] (v4l2_ioctl [videodev]) from [<802f8540>] (do_vfs_ioctl+0xbc/0x804)
[Thu Aug 13 10:12:33 2020] [<802f8484>] (do_vfs_ioctl) from [<802f8ccc>] (ksys_ioctl+0x44/0x6c)
[Thu Aug 13 10:12:33 2020] r10:00000036 r9:8bcf6000 r8:00000026 r7:40045613 r6:8e89a540 r5:021f1200
[Thu Aug 13 10:12:33 2020] r4:8e89a541
[Thu Aug 13 10:12:33 2020] [<802f8c88>] (ksys_ioctl) from [<802f8d0c>] (sys_ioctl+0x18/0x1c)
[Thu Aug 13 10:12:33 2020] r8:801011c4 r7:00000036 r6:4ca637d0 r5:4ca63000 r4:6a308648 r3:76d72510
[Thu Aug 13 10:12:33 2020] [<802f8cf4>] (sys_ioctl) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[Thu Aug 13 10:12:33 2020] Exception stack(0x8bcf7fa8 to 0x8bcf7ff0)
[Thu Aug 13 10:12:33 2020] 7fa0: 6a308648 4ca63000 00000026 40045613 021f1200 76d72510
[Thu Aug 13 10:12:33 2020] 7fc0: 6a308648 4ca63000 4ca637d0 00000036 6a307300 6a4785f0 706e492c 74c73ce8
[Thu Aug 13 10:12:33 2020] 7fe0: 708cfc60 7eb630b4 4ca2b954 76d7251c
[Thu Aug 13 10:12:33 2020] ---[ end trace abd9ddf252533015 ]---
[Thu Aug 13 10:12:33 2020] videobuf2_common: driver bug: stop_streaming operation is leaving buf 0b70dd47 in active state
[Thu Aug 13 10:12:33 2020] videobuf2_common: driver bug: stop_streaming operation is leaving buf b0d5ce36 in active state
[Thu Aug 13 10:12:33 2020] videobuf2_common: driver bug: stop_streaming operation is leaving buf da61dea1 in active state
[Thu Aug 13 10:12:33 2020] videobuf2_common: driver bug: stop_streaming operation is leaving buf d4e41263 in active state
[Thu Aug 13 10:12:33 2020] videobuf2_common: driver bug: stop_streaming operation is leaving buf db4367ac in active state
[Thu Aug 13 10:12:33 2020] videobuf2_common: driver bug: stop_streaming operation is leaving buf 08347a89 in active state
The text was updated successfully, but these errors were encountered: