Tekton Buildah Task failed No Space left On device

[tppalani]

Expected Behavior

Buildah docker build task from tekton source workspace inside this i have Dockerfile to build the app layers. I have 10Gi space you can refer the workspace path location /workspace/source

Actual Behavior

Steps to Reproduce the Problem

disk space check i have noticed overlay is used for 100% but main workspace we have lot of space is availbile.

Filesystem                         Size  Used Avail Use% Mounted on
overlay                             59G   59G     0 100% /
tmpfs                               64M     0   64M   0% /dev
tmpfs                              7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1                      60G  8.0G   53G  14% /workspace
tmpfs                               15G  8.0K   15G   1% /tekton/creds
/dev/mapper/vg.docker-lv.dockerfs   59G   59G     0 100% /etc/hostname
tmpfs                               15G  4.0K   15G   1% /workspace/dockerconfig
shm                                 64M     0   64M   0% /dev/shm
/dev/nvme2n1                       9.8G   55M  9.7G   1% /workspace/source
tmpfs                               15G  8.0K   15G   1% /tekton/creds-secrets/bitbucket-secret
tmpfs                               15G   12K   15G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                              7.8G     0  7.8G   0% /proc/acpi
tmpfs                              7.8G     0  7.8G   0% /sys/firmware

Tekton Task yaml

buildah --storage-driver=$(params.STORAGE_DRIVER) bud --format=$(params.FORMAT) --no-cache -f $(params.DOCKERFILE) -t $(params.IMAGE)

Error logs

Copying blob sha256:64c1fd37906bb3f2ed67579acf26337afe3999f7bcac3851b73f2252dfea2cec
Error: creating build container: copying system image from manifest list: writing blob: adding layer with blob "sha256:148a3ed2f70ea72bafab6a24598d47682ae82214e6b84b09c69960fdd727465b": processing tar file(mkdir /usr/lib/.build-id/bb: no space left on device): exit status 1

[vdemeester]

Hey @tppalani, so this is a "buildah task" issue but definitely not a tektoncd/pipeline issue, I'll move it to tektoncd/catalog.

I am not sure which buildah Task you are using (can you link us to it ?), but the one I know, use the workspace for the source, but buildah itself uses /var/lib/containers to store container's related data (volume, layers, …). So if the image you are building is getting big, it will take space in /var/lib/container (which, in your case is in the partition that is mounted at /) and not in the workspace.

According to buildah --help, you could set the --root and --run-root flags to point to somewhere in the workspace for buildah to use it instead of `/var/lib/containers).

[tppalani]

Hi @vdemeester

Thank for the right direction.

Sorry I'm new to the tekton, According to your note i have modified my build command but still its failing can you please help me.

buildah --storage-driver=$(params.STORAGE_DRIVER) --root=/workspace/source/ --runroot=/workspace/source/ bud --format=$(params.FORMAT) --no-cache -f $(params.DOCKERFILE) -t $(params.IMAGE)

Filesystem                         Size  Used Avail Use% Mounted on
overlay                             59G   54G  2.5G  96% /
tmpfs                               64M     0   64M   0% /dev
tmpfs                              7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1                      60G  8.1G   52G  14% /workspace
tmpfs                               15G  8.0K   15G   1% /tekton/creds
/dev/mapper/vg.docker-lv.dockerfs   59G   54G  2.5G  96% /etc/hostname
tmpfs                               15G  4.0K   15G   1% /workspace/dockerconfig
shm                                 64M     0   64M   0% /dev/shm
/dev/nvme2n1                       9.8G  216K  9.8G   1% /workspace/source
tmpfs                               15G  8.0K   15G   1% /tekton/creds-secrets/bitbucket-secret
tmpfs                               15G   12K   15G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                              7.8G     0  7.8G   0% /proc/acpi
tmpfs                              7.8G     0  7.8G   0% /sys/firmware
STEP 1/7: FROM node:16-alpine
Resolved "node" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull [docker.io/library/node:16-alpine](http://docker.io/library/node:16-alpine)...
Getting image source signatures
Copying blob sha256:d9059661ce70092af66d2773666584fc8addcb78a2be63f720022f4875577ea9
Copying blob sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de
Copying blob sha256:eee371b9ce3ffdbb8aa703b9a14d318801ddc3468f096bb6cfeabbeb715147f9
Copying blob sha256:93b3025fe10392717d06ec0d012a9ffa2039d766a322aac899c6831dd93382c2
Copying config sha256:2573171e0124bb95d14d128728a52a97bb917ef45d7c4fa8cfe76bc44aa78b73
Writing manifest to image destination
time="2024-10-02T10:33:28-04:00" level=error msg="Unmounting /workspace/source/overlay/85647c97c7047897ad409013f2f8f62edca1a20ee715f401fdb69468da1785e9/merged: invalid argument"
Error: mounting new container: mounting build container "d8ff5cc3c82cbd70cc668d9942cfdb40605a01d0bc9259326d1c1fbdffdaeede": creating overlay mount to /workspace/source/overlay/85647c97c7047897ad409013f2f8f62edca1a20ee715f401fdb69468da1785e9/merged, mount_data="lowerdir=/workspace/source/overlay/l/QEPNRTLMA7JHFJ63QR4FZWAELF:/workspace/source/overlay/l/KUPXF25IOJD2UGYWKWTWXN4ZX4:/workspace/source/overlay/l/KEMX32IGUECE6SL3RHAPDZCZFT:/workspace/source/overlay/l/CXT3AVOOHZMH7VK2MOVFO47HTG,upperdir=/workspace/source/overlay/85647c97c7047897ad409013f2f8f62edca1a20ee715f401fdb69468da1785e9/diff,workdir=/workspace/source/overlay/85647c97c7047897ad409013f2f8f62edca1a20ee715f401fdb69468da1785e9/work,volatile": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1

[tppalani]

Hi @vdemeester

Above fuse issue resolved after adding security context values in the buildah task

 securityContext:
        privileged: true

Now its moved to another issue file path is too long..

error: building at STEP "COPY . .": reading "/workspace/source": error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/workspace/source", rootPrefix:"/workspace/source", Directory:"/", preservedDirectory:"/workspace/source", Globs:[]string{"/"}, preservedGlobs:[]string{"/workspace/source"}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(0xc000014750), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc000014760), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:true, StripSetgidBit:true, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: get: "/"("/"): lstat 

217bfc2d3d1847c604cb4c359a854322ff87a/diff/app/overlay/966da02b3fe0f09756581a0b384217bfc2d3d1847c604cb4c359a854322ff87a/diff/app/overlay/4693057ce2364720d39e57e85a5b8e0bd9ac3573716237736d6470ec5b7b7230/diff: file name too long