---
title: "llm-queue — an OpenAI-compatible queue for local LLMs"
description: "Case study: a single-worker priority queue for Ollama, exposed as an OpenAI-compatible HTTP service so every process on the machine shares one queue against one model."
canonical: "https://asmyshlyaev177.dev/projects/llm-queue"
image: "https://asmyshlyaev177.dev/og_image.png"
---

[All projects](https://asmyshlyaev177.dev/projects.md)

One model, one queue, every client

# llm-queue

A single-worker priority queue for local LLMs, exposed as an OpenAI-compatible HTTP service — so cron jobs, browser extensions and OpenAI clients all share one serialized queue against one model instead of fighting over the GPU.

## The challenge

One machine, one model, and several things that want it at once. Ollama will happily accept concurrent requests and then thrash, so the second caller makes the first one slower rather than waiting its turn.

## My role

Sole author. Built the queue and its priority handling, wrapped it in an OpenAI-compatible HTTP surface so existing clients need no code change, and published it to npm with a CLI.

## Result

Backs the LLM work in my own tooling — the job-board monitor and the on-device content classifier both queue through it.

## Built with

-   Node.js
-   TypeScript
-   Ollama
-   HTTP API
-   CLI
-   npm

[Source](https://github.com/asmyshlyaev177/llm-queue)

More on this

[Why I built llm-queue: one local model, one queue](https://asmyshlyaev177.dev/blog/why-i-built-llm-queue.md)

One local LLM (Ollama) behind an OpenAI-compatible HTTP API, so every process shares one serialized queue instead of thrashing the model. On my 6 GB GPU that made everything roughly four times faster.

I'm available for full-time remote work in EU timezones. Happy to talk through anything here in more detail.

[Get in touch](https://asmyshlyaev177.dev/#contact)

---

Site summary for LLMs: <https://asmyshlyaev177.dev/llms.txt> · Blog: <https://asmyshlyaev177.dev/blog>