AI Infrastructure_Parallaxis Internal PlatformLocal inference gateway

Making one office GPU usable as a governed AI inference service_

We packaged quantized models behind an OpenAI-compatible gateway, added request limits and health checks, and separated interactive chat, embeddings, and batch jobs by workload class. The office GPU becomes a reusable de

Local LLM infrastructure
OpenAI-compatible API
Quantized model profiles

Challenge_

Developers needed shared access to an RTX 3060 Ti without copying models, fighting for VRAM, or embedding machine-specific endpoints in every application.

Approach_

We packaged quantized models behind an OpenAI-compatible gateway, added request limits and health checks, and separated interactive chat, embeddings, and batch jobs by workload class.

Outcome_

The office GPU becomes a reusable development service with predictable model choices and fallback behavior, while sensitive experiments can remain on premises.

Overview

Developers needed shared access to an RTX 3060 Ti without copying models, fighting for VRAM, or embedding machine-specific endpoints in every application.

We packaged quantized models behind an OpenAI-compatible gateway, added request limits and health checks, and separated interactive chat, embeddings, and batch jobs by workload class.

The engineering decision

Eight gigabytes of VRAM rewards discipline. Smaller models, quantization, bounded context, and queued batch work produce more useful throughput than loading the largest model that barely fits.

How the system works

The implementation separates intake, validation, state changes, side effects, and reporting. That separation makes failures visible and allows one layer to change without rewriting the entire workflow.

Operational users see explicit statuses and exceptions; technical teams retain identifiers, timestamps, versions, and logs needed to reproduce a result.

Outcome