Skip to main content

One model, one queue, every client

llm-queue

A single-worker priority queue for local LLMs, exposed as an OpenAI-compatible HTTP service — so cron jobs, browser extensions and OpenAI clients all share one serialized queue against one model instead of fighting over the GPU.

The challenge

One machine, one model, and several things that want it at once. Ollama will happily accept concurrent requests and then thrash, so the second caller makes the first one slower rather than waiting its turn.

My role

Sole author. Built the queue and its priority handling, wrapped it in an OpenAI-compatible HTTP surface so existing clients need no code change, and published it to npm with a CLI.

Result

Backs the LLM work in my own tooling — the job-board monitor and the on-device content classifier both queue through it.

Built with

  • Node.js
  • TypeScript
  • Ollama
  • HTTP API
  • CLI
  • npm
Source

I'm available for full-time remote work in EU timezones. Happy to talk through anything here in more detail.

Get in touch