How to Build a Local AI Drafting Engine for a B2B Service Firm

Most B2B service firms route every draft, every research query, and every client document through a hosted AI subscription. Each seat costs the firm a fixed monthly fee, and each query leaks client data to a third party. A local drafting engine replaces that seat with a workstation the firm owns. The system is one machine, a stack of open containers, and a runbook. A founder with one technical operator on staff can install it in a week.

The problem

Hosted AI tools charge per seat. A ChatGPT Team or Claude Team plan runs $200 to $300 per user per month. For a twenty-person professional services firm, that is $4,000 to $6,000 per month of recurring software cost. The cost scales with headcount, so the firm pays more as it grows.

Client briefs, contracts, and proposals pass through a vendor's servers. The vendor's terms grant the vendor a license to use the data for model improvement in many cases. A regulated client, a financial auditor, a healthcare buyer, or a government procurement officer will refuse to sign when their data sits on someone else's hardware. The firm loses the deal, or it spends two weeks negotiating a data processing agreement that excludes the AI tool. Both outcomes are wasted motion.

The hosted model also has a context ceiling. Most chat products cap context at 200K tokens. A 400-page contract with exhibits breaks that limit. The team is forced to chunk, summarize, and lose detail. A local model with one million tokens of context handles the same document in one pass, with the full clause structure intact.

Finally, the hosted model is a single point of failure. When the vendor throttles, deprecates a model, changes pricing, or has an outage, the firm scrambles. A local model on owned hardware has no such cliff. The firm controls the model version, the inference rate, and the uptime.

Who this is for

This build is for a B2B service founder in Southeast Asia running a $1M to $5M firm. The team is five to twenty people. The work is research, drafting, and document review. The buyer profile is a professional services firm, an agency, a consultancy, or a small manufacturer with technical sales engineers who write proposals and parse specifications.

The non-fit is a solo operator who writes one email a week. A single user does not need a dedicated GPU workstation. A hosted free or low-cost tier is enough at that scale. The non-fit also includes a firm with no IT operator on staff. The build needs one technical owner to install it, patch the OS, update the model, and recover from GPU driver crashes. Without that owner, the workstation decays and the team loses trust in the system.

The fit firm is also data-sensitive. Healthcare, legal, finance, and government-facing work are strong fits because the data must not leave the building. The fit firm also drafts at volume. A consultancy that writes ten proposals a week will feel the cost and time savings within the first month. The fit firm also expects to grow. A team of five that plans to be twenty in eighteen months gets more value from a shared workstation than from per-seat scaling. The cost curve is the same whether the team is five or fifty, because the GPU is the bottleneck, not the seat count.

System architecture

The system is one workstation connected to the firm's local network. The workstation runs a stack of containers. The team reaches the workstation through a browser, a desktop client, or a chat interface that runs on the same LAN. No traffic leaves the building.

Inputs are the firm's documents, the team's prompts, and the firm's internal knowledge base. Documents enter through a watched folder, a manual upload, or an email-to-ingest rule. Prompts enter through a chat interface that binds to the workstation's local address. The knowledge base is a vector store populated by an ingestion job that runs nightly and indexes every PDF, DOCX, and Markdown file in a designated share.

Outputs are drafted text, summaries, and answers grounded in the firm's own documents. The system also exposes an OpenAI-compatible API on a local port so other internal tools (a CRM, a proposal tool, an internal chatbot) can call the model directly. The API is on the LAN only.

The flow:

1. The user opens the chat interface at the workstation's local URL.
2. The user attaches a document or selects a knowledge base.
3. The orchestrator routes the request to the right model on the GPU.
4. The model returns a draft or an answer.
5. The user copies the output into the firm's standard tools, or an internal tool calls the API directly.

Build steps

1. Procure a workstation with one NVIDIA GPU holding at least 24 GB of VRAM. A used RTX 3090, a new RTX 4090, or an NVIDIA A5000 fits. Budget $2,000 to $4,000 for the GPU and the rest of the machine. Add 64 GB of system RAM and a 2 TB NVMe drive for the vector store.

2. Install Ubuntu 22.04 LTS. Linux is the cleaner path for inference runtimes and Docker. Plan one full day for the OS install, the NVIDIA driver, and the CUDA toolkit. Windows 11 with WSL2 is an alternative if the technical operator is Windows-first, but Linux gives fewer surprises.

3. Install Docker Engine and Docker Compose. The system runs as a stack of containers. Docker isolates the model server, the vector store, the chat interface, and the orchestration layer. A single docker-compose.yml file describes the whole stack.

4. Deploy Ollama or vLLM as the inference server. Ollama is the simpler install and has a one-line model pull. vLLM is faster for multi-user workloads and supports more concurrent requests. Choose one. Both expose an OpenAI-compatible API on a local port, so any chat client that speaks the OpenAI API can connect.

5. Pull two open models from a public hub like Hugging Face. Use a 7B to 9B parameter chat model in Q4 quantization for general drafting, summarization, and Q&A. Use a long-context model (1M tokens) for document work. The Q4 versions fit in 24 GB of VRAM and run at 20 to 40 tokens per second on a 4090. Pin the model versions in the docker-compose file so updates are deliberate.

6. Deploy Open WebUI as the chat interface. Bind it to the LAN address of the workstation. The team reaches it at http://workstation.local:8080. Set basic auth and per-user accounts. No external access. No public URL.

7. Deploy ChromaDB or Qdrant as the local vector store. Run a nightly ingestion job that watches a shared drive and indexes new PDFs, DOCX, and Markdown files. Store the embeddings on the NVMe drive, not on a network share. Plan for 5 GB to 20 GB of vector data at steady state for a 10-person firm.

8. Wire the inference server to the vector store through a retrieval augmented generation (RAG) layer. Use a small Python script or a low-code tool like Rowboat. The script takes the user's question, retrieves the top five chunks from the vector store, and feeds them to the model with the prompt. The system prompt instructs the model to answer from the retrieved context only.

9. Set role-based access and network policy. The workstation is on the LAN only. Add a reverse proxy with basic auth if the team works remotely over VPN. Do not expose the port to the public internet. Block outbound traffic from the workstation to common hosted AI domains at the firewall, so a misconfigured client cannot leak data.

10. Write a one-page runbook. Cover: how to start the Docker stack, how to pull a new model, how to back up the vector store, how to restart the inference server after a crash, and who to call when the GPU fans scream. Place the runbook in the firm's shared drive and review it once a quarter.

Tools and costs

Hardware: workstation with 24 GB VRAM GPU, $2,000 to $4,000 one-time.

OS: Ubuntu 22.04 LTS, free.

Container runtime: Docker Engine and Docker Compose, free for community edition.

Inference server: Ollama, free. vLLM, free.

Models: open weights from public hubs, free. Pull a 9B chat model in Q4 for general work and a long-context model for document tasks. Both are permissively licensed for commercial use.

Chat interface: Open WebUI, free.

Vector store: ChromaDB, free. Qdrant community edition, free.

Orchestration: Rowboat, free. n8n self-hosted, free.

RAG layer: a small Python script, free. Or a managed open-source tool like Haystack, free.

Electricity: one workstation idles at 80 to 150 watts and peaks at 450 watts under load. At $0.12 per kWh in most Southeast Asian markets, the monthly run cost is $20 to $40 in electricity. Add $5 to $10 for cooling.

Total monthly run cost: under $50. The one-time hardware cost is $2,000 to $4,000. Compared to a hosted team plan at $200 to $300 per seat, the firm breaks even in month one at three users and saves the difference every month after.

Failure modes

1. Model invents a fact that is not in the source document. The fix is to enable the RAG layer on every document task. Pass the retrieved chunks to the model and instruct it in the system prompt to answer from the chunks only. Add a refusal instruction: if the answer is not in the context, say so. Test the refusal behavior with a known out-of-scope question once a week.

2. GPU runs out of memory on a long document. The fix is to lower the model quantization to Q4, or to split the document into chunks before retrieval, or to upgrade the GPU to 48 GB. A 24 GB card runs a 9B Q4 model and a 70K-token context cleanly. A 48 GB card runs a 70B Q4 model and a 200K-token context. Pick the card that matches the longest document the team handles.

3. Vector store goes stale and the team gets outdated answers. The fix is to run the nightly ingestion job on a cron and to write a health check that emails the operator when the job fails. Review the indexed file count once a week. Reindex the full corpus once a quarter, because the embedding model will change and old embeddings will lose compatibility.

4. Team bypasses the local system and uploads a client file to a hosted AI tool. The fix is to write a one-page data policy, train the team on it, and add a web filter at the office firewall that blocks uploads to common hosted AI domains. Audit the firewall log once a month. Make the local system easier to use than the hosted tool, and the team will route through it.

5. Inference server crashes during a client deadline. The fix is to run the inference server under a process manager (Docker restart policy, systemd, or a container orchestrator) that restarts on crash. Keep a backup of the model weights on a second internal drive. Document the restart procedure in the runbook. Run a canary request through the API once a day so the operator notices a crash within 24 hours.

What good looks like

First draft turnaround moves from hours to minutes. A consultant produces a structured first draft in ten minutes and spends the saved time on review, on client calls, and on scoping.

Per-seat AI cost falls from $200 to $300 per month to a shared workstation cost under $50 per month. A team of ten sees the difference on the monthly software line. The number holds as the team grows.

Client data stays on the firm's hardware. The firm signs data processing agreements without carve-outs for AI vendors. The firm wins regulated clients that require on-premise processing.

The system handles a 400-page contract in one pass with a long-context model. The team reads, summarizes, and extracts clauses without chunking. The contract review moves from two days to two hours.

The firm owns the system. When the model provider releases a new version, the firm pulls it on its own schedule. When the vendor changes pricing, the firm is unaffected. When the firm hires user number twenty-one, the firm adds a user account, not a subscription seat.

A local drafting engine is not a research project. It is a workstation, a stack of open containers, and a one-page runbook. The founder who installs it on Monday has a margin engine by Friday.