Repository navigation
Clef on IBM Cloud Code Engine Fleets - #500
Luke-Roy-IBM wants to merge 3 commits into
Conversation
a1f7090 to
24569df
Compare
Signed-off-by: Luke Roy <luke.roy@ibm.com>
24569df to
83e1aca
Compare
reggeenr
left a comment
reggeenr
left a comment
There was a problem hiding this comment.
Hi @Luke-Roy-IBM , great example!
Looking forward to see it live :)
So far fleets tutorials are maintained in serverless-fleets/tutorials. I suggest to move this one to the tutorials folder, too. Apart from that I suggest to rework the function integration and make use of Trusted Profiles instead of using an API Key to access COS
| # This container calls the clef model via the ollama HTTP API and | ||
| # prints a JSON result to stdout. | ||
| # --------------------------------------------------------------------------- | ||
| FROM registry.access.redhat.com/ubi10/ubi-minimal |
There was a problem hiding this comment.
I suggest to slightly rework this one and use a distroless runtime image. Similarly like done here: https://github.com/IBM/CodeEngine/blob/main/logging/python-structured/Dockerfile
| -v ollama:/root/.ollama \ | ||
| -p 11434:11434 \ | ||
| --name ollama \ | ||
| docker.io/ollama/ollama:latest |
There was a problem hiding this comment.
Do we know / have an alternative which is not hosted on docker.io?
There was a problem hiding this comment.
Only docker is the official image or we build the image ourselfs or run directly on host: https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image
| ) | ||
| CE_PROJECT_ID = os.environ.get("CE_PROJECT_ID", "") | ||
| FLEET_ID = os.environ.get("FLEET_ID", "") | ||
| CE_API_KEY = os.environ.get("CE_API_KEY", "") |
There was a problem hiding this comment.
I suggest to rework this function and make use of Trusted Profiles instead of a long living API Key
There was a problem hiding this comment.
good point Trusted Profiles is definitely the way to go
| # Ensure secret exists or create it if CE_API_KEY was supplied in env | ||
| if [[ -n "${CE_API_KEY}" ]]; then | ||
| print_msg "Creating/updating secret '${SECRET_NAME}'..." | ||
| if ibmcloud ce secret get --name "${SECRET_NAME}" >/dev/null 2>&1; then | ||
| ibmcloud ce secret update --name "${SECRET_NAME}" --from-literal CE_API_KEY="${CE_API_KEY}" | ||
| else | ||
| ibmcloud ce secret create --name "${SECRET_NAME}" --from-literal CE_API_KEY="${CE_API_KEY}" | ||
| fi | ||
| else | ||
| if ! ibmcloud ce secret get --name "${SECRET_NAME}" >/dev/null 2>&1; then | ||
| print_error "Secret '${SECRET_NAME}' does not exist and CE_API_KEY environment variable is not set." | ||
| echo "Create the secret with:" | ||
| echo " ibmcloud ce secret create --name ${SECRET_NAME} --from-literal CE_API_KEY=<your-ibmcloud-api-key>" | ||
| echo "or set export CE_API_KEY=<key> before running." | ||
| exit 1 | ||
| fi | ||
| fi |
There was a problem hiding this comment.
This should be replaced by the creation of a trusted profile that grants required permissions for the target COS instance
There was a problem hiding this comment.
yes Trusted Profiles is definitely the way to go keeps slipping my mind that we have it
| `ibmcloud ce pds create --name fleet-output-store ...` | ||
| - Subnetpool `fleet-subnetpool` provisioned | ||
| - Container registry secret `ce-auto-icr-private-eu-de` (or set `FLEET_REGISTRY_SECRET`) | ||
| - An IBM Cloud API key with Code Engine Writer and COS access (set as `CE_API_KEY` or stored in `clef-fn-secrets`) |
There was a problem hiding this comment.
As mentioned before, we should replace the API Key by a Trusted Profile integration
Co-authored-by: Enrico Regge <36001299+reggeenr@users.noreply.github.com>
Co-authored-by: Enrico Regge <36001299+reggeenr@users.noreply.github.com>
|
@reggeenr if you are happy with this being v1 and getting merged i can work on the trusted profile support in the next iteration |
This branch has not been deployed
2 participants
This sample demonstrates generic AI decision classification using clef, a lightweight decision model served via Ollama, running on IBM Cloud Code Engine GPU Fleets. Pass in any text with a plain-language question and true/false criteria — clef returns a probability score and a boolean verdict. No prompt engineering, no fine-tuning required.
What it does
A Code Engine Function acts as the HTTP gateway: it validates the request, submits a task to the fleet via the CE REST API, and either polls Cloud Object Storage for the result within a configurable timeout (returning HTTP 200 synchronously) or returns HTTP 202 with a task_id and a ready-to-use retrieve_url for async polling. Workers stay idle at zero cost between jobs and scale up on demand. The 18 GB clef model is loaded into GPU VRAM once via a worker prehook — verified and warmed up before the first task ever arrives.
Immediate use cases include spam and phishing detection, content moderation, document field extraction, ticket routing, and policy compliance checks — anything that can be expressed as a yes/no question.
Why this is more than a clef demo
The architecture is a reusable blueprint for any interactive serverless fleet workload:
Sync/async duality from a single endpoint. The Function layer abstracts away fleet mechanics entirely. Callers get instant results when workers are hot and a task ID when they are not — with no changes to the caller code.
Worker-local heavy services. The prehook pattern — starting Ollama via Podman once per worker and sharing it across many lightweight task containers — applies to any workload that benefits from a warm, expensive process: database engines, embedding servers, simulation runtimes, or video processing pipelines.
COS as the task result bus. Using object storage as a decoupled result store means the Function, the fleet, and the caller are never directly coupled. Any consumer can retrieve results at any time with a single GET.
GPU at zero idle cost. Workers scale to zero between batches. The prehook ensures the GPU and model are ready before tasks start, so scale-up is transparent to callers.
Swap clef for any Ollama model, any inference server, or any long-running local process — the scaffolding stays identical.