-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial version of aarch64 container with Vulkan #270
base: main
Are you sure you want to change the base?
Conversation
@sroecker please sign your commit, this is failing the DCO build:
to sign an old commit. |
Signed-off-by: Steffen Roecker <[email protected]>
40672af
to
3e41c4e
Compare
I would prefer these all to be based off a base image with all of the python tools required to run ramalama and then rocm, vulcan, ... can all share the lower layer. |
@sroecker is llama.cpp working properly with you with a container generated from this Containerfile? Which models have you tested? I'm asking because the Vulkan backend hasn't worked for me since March, which is the reason why I started favoring the Kompute (which also uses Vulkan) backend. |
I had to test a smaller model due to machine constraints:
I can check with the kompute backend tomorrow. |
Tested with Mistral-7B and Wizard-Vicuna-13B and got random answers with both of them. Sadly, the Vulkan backend is still broken for Apple Silicon GPUs upstream. I think we're going to need to stay for a while with the Kompute backend, as implemented in #235. |
Initial version for aarch64 container with Vulkan support that runs on libkrun containers on MacOs