bare-base rock that bundles and runs prism-ml/bonsai-1.7B model through llama cpu server. bind a host port, get a chat window -- and an OpenAI-compatible API on the same port.
read more about about the bonsai rock in a ubuntu discourse post
docker run -d --rm --name bonsai -p 8080:8080 ghcr.io/rockcrafters/bonsai-rock/bonsai:1.7B
curl http://localhost:8080/v1/models | jq '.models[].name'
docker rm -f bonsai
llama-server runs under pebble and serves everything on :8080:
/-- llama.cpp's built-in chat web ui/v1/models,/v1/chat/completions-- OpenAI-compatible API, incl. streaming and tool-calls
- https://github.com/PrismML-Eng/Bonsai-demo
- https://github.com/ggml-org/llama.cpp
- https://github.com/PrismML-Eng/Bonsai-demo/blob/main/1-bit-bonsai-8b-whitepaper.pdf
| component | licence | in the image |
|---|---|---|
| bonsai-1.7B weights | Apache-2.0 | /usr/share/doc/bonsai-1.7B/{LICENSE,NOTICE.txt,PROVENANCE} |
llama.cpp (llama-server, statically linked) |
MIT | /usr/share/doc/llama.cpp/LICENSE |
| glibc, libstdc++, libgomp | LGPL / GPL+exception | /usr/share/doc/<pkg>/copyright |
the Bonsai logo is (c) Prism ML, Inc., used under Apache-2.0 to identify the model.