kernel update 5.15.5-76051505 breaks zfs

[lynxnot]

Distribution (run cat /etc/os-release):
Pop!_OS 21.04

Related Application and/or Package Version (run apt policy $PACKAGE NAME):
linux-image-5.15.5-76051505-generic:
Installed: 5.15.5-76051505.202111250933163820157921.0409f1aa7
Candidate: 5.15.5-76051505.202111250933163820157921.0409f1aa7
Version table:
*** 5.15.5-76051505.202111250933163820157921.04~09f1aa7 1001
1001 http://ppa.launchpad.net/system76/pop/ubuntu hirsute/main amd64 Packages
100 /var/lib/dpkg/status

linux-modules-5.15.5-76051505-generic:
Installed: 5.15.5-76051505.202111250933163820157921.0409f1aa7
Candidate: 5.15.5-76051505.202111250933163820157921.0409f1aa7
Version table:
*** 5.15.5-76051505.202111250933163820157921.04~09f1aa7 1001
1001 http://ppa.launchpad.net/system76/pop/ubuntu hirsute/main amd64 Packages
100 /var/lib/dpkg/status

Issue/Bug Description:
zfs module is missing, no mounts after boot
sudo modprobe zfs fails with no zfs module found

Steps to reproduce (if you know):

• setup zfs volumes with kernel 5.13
• update kernel to 5.15.5

Expected behavior:
zfs module is present after update to 5.15
zfs volumes are mounted at boot

[e-garbage]

Same problem here, i tried this solution: https://github.com/pop-os/beta/issues/285
It worked for one day, then after 2 reboot, the issue was back.
Help would be much appreciated since I'm not very confident to deal with kernel stuffs.

[e-garbage]

as mention in the previous link one solution can be like this (i added some modifications):

Remove both zfs-dkms and zfsutils-linux (in case they are installed):

sudo apt remove zfs-dkms
sudo apt remove zfsutils-linux
sudo apt autoremove

Download and install the zfs-dkms package from 21.10:

wget -P ~/Download http://archive.ubuntu.com/ubuntu/pool/universe/z/zfs-linux/zfs-dkms_2.0.6-1ubuntu2.1_all.deb
sudo dpkg -i ~/Download/zfs-dkms_2.0.6-1ubuntu2.1_all.deb
sudo /sbin/modprobe zfs

Re-install zfsutils-linux:

sudo apt install zfsutils-linux

But this solution is falling apart every time I reboot my computer

Another solution is to downgrade the kernel (boot in 5.13 instead of 5.15) but still, it is a temporary solution that will break at the next update...

[flingtoad]

I also ran into this issue. I ended up reverting to a previous system snapshot. Unfortunately, the update also seemed to have changed to initrd process to no longer prompt for my passphrase when decrypting one of my ZFS datasets. After manually decrypting the datasets (zfs load-key -a) and mounting them (zfs mount -a), I installed a different kernel (5.13.0-22-generic) from the Ubuntu repo. I've since pinned my kernel version until this gets sorted out.

[SoenensBram]

There is indeed a problem with zfs after the lateste update.

It seems that the system module "zfs-import-cache.service" fails because it can't find the "/sys/module/zfs" and "zfs-load-module.service" fails because it can't find the zfs module in "/lib/modules/5.15.5-76051505-generic".

I found this after scrolling through syslog:
"systemd-udevd[537]: sdb1: Process '/sbin/modprobe zfs' failed with exit code 1.
systemd-udevd[549]: sda1: Process '/sbin/modprobe zfs' failed with exit code 1.
udevadm[588]: systemd-udev-settle.service is deprecated. Please fix zfs-import-cache.service, zfs-load-module.service not to pull it in."

[pdemarino]

Doesn't this issue also affect anybody who uses System76 PPAs, beyond Pop!_OS Users??

[jacobgkau]

Doesn't this issue also affect anybody who uses System76 PPAs, beyond Pop!_OS Users??

Yes, the same kernel is shipped by both Pop!_OS and the System76 Ubuntu PPA that is used for hardware enablement on System76 products.

According to the most recent OpenZFS changelog (2.1.1), OpenZFS does not yet have support for kernel 5.15 (the highest "supported" kernel is 5.14, although it looks like work is actively being done in the project to support 5.15 in the future): https://github.com/openzfs/zfs/releases/tag/zfs-2.1.1

[dustin-hawkins]

https://github.com/openzfs/zfs/releases/tag/zfs-2.1.2

ZFS 2.1.2 was released with kernel support to 5.15.

Not sure how long this will take to work its way into the repos

[curiousercreative]

Any updates here? Can't update my desktop until this is resolved.

[SoenensBram]

I update this week to "Pop!_OS 21.10 x86_64" and did followed these instructions

as mention in the previous link one solution can be like this (i added some modifications):

Remove both zfs-dkms and zfsutils-linux (in case they are installed):

sudo apt remove zfs-dkms sudo apt remove zfsutils-linux sudo apt autoremove

Download and install the zfs-dkms package from 21.10:

wget -P ~/Download http://archive.ubuntu.com/ubuntu/pool/universe/z/zfs-linux/zfs-dkms_2.0.6-1ubuntu2.1_all.deb sudo dpkg -i ~/Download/zfs-dkms_2.0.6-1ubuntu2.1_all.deb sudo /sbin/modprobe zfs

Re-install zfsutils-linux:

sudo apt install zfsutils-linux

But this solution is falling apart every time I reboot my computer

Another solution is to downgrade the kernel (boot in 5.13 instead of 5.15) but still, it is a temporary solution that will break at the next update...

And that seems to fix it on the 2 systems that i have that run pop-os and zfs it mounts at boot and the performance is the same as before.

[curiousercreative]

I haven't tested thoroughly or for any length of time, but ZFS seems to be installing and loading with the below. But really haven't tested this beyond what I've described below. This system doesn't boot from ZFS nor does it mount any ZFS (it's just a laptop!). In a couple weeks I'll be back at my desktop that runs ZFS and know whether this is really functioning and stable. FWIW, there are ZFS 2.0 (currently 2.0.6) and ZFS 2.1 (currently 2.1.1) PPAs here. I'm running the 2.0 on a Raspberry Pi 4 on Ubuntu 20.04 Kernel 5.4 and found that 2.1 was horrendously unstable for my UASP enclosures and couldn't get past 0.5% of a scrub without leaving 1/4 of my disks UNAVAIL and the pool suspended until reboot.

1. Apply all Pop! Shop updates on 21.04
2. Run 21.10 OS upgrade
3. Upon reboot into 21.10, notice that zfs --version returns an error
4. Uninstall ZFS

   sudo apt remove zfsutils-linux zfs-dkms
   sudo apt install zfsutils-linux zfs-dkms

5. Reboot and zfs --version should return "zfs-2.0.6-1ubuntu2"
6. Reboot again and notice it still working

[chuck-bz]

I'm running into the same problem.
I'm using ZFS for the root so it makes it difficult to diagnose. Glad this bug report was here.
When this update came out I had to resort to booting the previous kernel (5.13)

I was hoping a couple weeks later that there would be a fix...
I just tried 5.15.8 (latest currently available), no change. Still fails to load ZFS. Back to good ol' 5.13!

[curiousercreative]

I'm running into the same problem. I'm using ZFS for the root so it makes it difficult to diagnose. Glad this bug report was here. When this update came out I had to resort to booting the previous kernel (5.13)

I was hoping a couple weeks later that there would be a fix... I just tried 5.15.8 (latest currently available), no change. Still fails to load ZFS. Back to good ol' 5.13!

@chuck-bz Have you tried reinstalling zfs and zfs-dkms while booted in 5.13 and then trying 5.15 again?

sudo apt remove zfsutils-linux zfs-dkms
sudo apt install zfsutils-linux zfs-dkms

[lynxnot]

@curiousercreative I think there is some confusion here. This bug is about zfs-2.1.2 not being present in the repo with a dependency on kernel >=5.15.

Using zfs-dkms changes the method to manage kernel modules. It is infact possible to build zfs-2.0.6 for kernel 5.15 by installing zfs-dkms.

$ cat /etc/pop-os/os-release | head -n2
NAME="Pop!_OS"
VERSION="21.10"

$ uname -r
5.15.8-76051508-generic

$ sudo apt install zfs-dkms
-> reboot

$ zfs --version
zfs-2.0.6-1ubuntu2
zfs-kmod-2.0.6-1ubuntu2

The problem here is that zfs-2.0.6 and kernel 5.15 are an unsupported combination, which with filesystems is a risk not worth taking.

Also the PPA proposed seems to rename zfsutils-linux to zfs-linux, so to install zfs-2.1.1 with the PPA the command should be:

$ sudo apt remove zfsutils-linux
$ sudo apt install zfs-linux zfs-dkms

EDIT: fixed incomplete zfs version zfs-2.1 -> zfs-2.1.2

[chuck-bz]

@curiousercreative No I have not tried to re-install zfs on 5.15 kernel. From what @lynxnot said it doesn't sound like that would be a good idea.

Also note that it looks like 2.0.7 is released with 5.15 support ( https://github.com/openzfs/zfs/releases?page=1 )

@lynxnot have you tried the PPA with zfs-2.1.2?

Sounds like our options are:

• Run unsupported zfs version with kernel 5.15 (sounds like a bad idea)
• Use the PPA to run zfs-2.1.2 with kernel 5.15
• Continue to use kernel 5.13 and quit receiving kernel updates until zfs-2.1.2 or zfs-2.0.7 makes it through the repos
• Switch to a mainline kernel (5.10?)

One thing to note, I'm running 20.04 so on my system zfs --version outputs:

zfs-0.8.3-1ubuntu12.13
zfs-kmod-2.0.6-1ubuntu2

I think that means my zfs version is 0.8.3 but the kernel module is 2.0.6. I'm not sure how that works...

[exstral]

Thanks for this discussion, I'm having the same problem. Considering giving up on my ZFS partition and reformatting it since it looks like ZFS is not a priority of PopOS at all :(

Which PPA are you referring to that has zfs 2.0.7 or 2.1.2 with proper support for kernel 5.15 ?

[curiousercreative]

@jacobgkau as you mentioned here this seems to be resolved? Do I need to do anything to be sure this comes in when I run my Pop! Shop updates? I don't see any new packages for zfs listed in Pop! Shop updates ready to apply to my 21.04 system (up-to-date as of late Nov or early Dec). I'm happy to update to 21.10 (that was my plan) but I follow the advice of upgrading packages prior to dist-upgrade. Of course, any user that updates to kernel 5.15 will need a zfs package that supports it, regardless of whether they're on 21.10 or prior.

[jacobgkau]

@curiousercreative The version of zfs-dkms that Ubuntu was shipping with 21.04 did not work with version 5.15. The version they're currently shipping on 21.10, which is the same version we're shipping, does work with kernel 5.15 (even though the documentation for that version says it doesn't*). We have backported that 21.10 version to 20.04 LTS, so yes, this issue should be resolved.

* The DKMS module compiles and I'm able to create and mount a pool.

[pdemarino]

@jacobgkau So to be 100% clear - what are the steps that we need to undertake to install it? Currently I have:

sudo dpkg --list |grep -i zfs
ii  libzfs2linux                               0.8.3-1ubuntu12.13                                        amd64        OpenZFS filesystem library for Linux
ii  libzfs4linux                               2.0.6-1ubuntu2pop0~1642612525~20.04~51ca23c~dev           amd64        OpenZFS filesystem library for Linux
ii  libzpool2linux                             0.8.3-1ubuntu12.13                                        amd64        OpenZFS pool library for Linux
ii  libzpool4linux                             2.0.6-1ubuntu2pop0~1642612525~20.04~51ca23c~dev           amd64        OpenZFS pool library for Linux
ii  zfs-auto-snapshot                          1.2.4-2                                                   all          ZFS automatic snapshot service
ii  zfs-initramfs                              2.0.6-1ubuntu2pop0~1642612525~20.04~51ca23c~dev           amd64        OpenZFS root filesystem capabilities for Linux - initramfs
ii  zfs-zed                                    2.0.6-1ubuntu2pop0~1642612525~20.04~51ca23c~dev           amd64        OpenZFS Event Daemon
ii  zfsutils-linux                             2.0.6-1ubuntu2pop0~1642612525~20.04~51ca23c~dev           amd64        command-line tools to manage OpenZFS filesystems

And nothing shows up in terms of modules.

[jacobgkau]

@pdemarino The package is zfs-dkms, so from a fresh system, run sudo apt install zfs-dkms. That package pulls in zfsutils-linux and other necessary dependencies.

On my 20.04 system, zfs2linux and libzpool2linux were removed when updating to the newer ZFS version, leaving only the two newer versions of those packages; you may want to remove the old ones manually if they don't get uninstalled on your system.

[curiousercreative]

I've been running zfs-dkms on 5.15 for a couple days now and haven't noticed any instability.

[curiousercreative]

Does this duplicate pop-os/linux#88?

[jacobgkau]

I've been running zfs-dkms on 5.15 for a couple days now and haven't noticed any instability.

Thank you for the confirmation. I'm closing this issue since the currently-shipped version of ZFS supports the currently-shipped kernel.

Does this duplicate pop-os/linux#88?

Yes. Although that one was in a more specific repository, given that the ZFS repo didn't exist yet when the issue started.