How we built our context engine

The problem

Ion's AI is only as good as the context it can draw from. If you ask "what's blocking the launch?", the AI needs to find every relevant doc, ticket, and thread in your workspace — fast. For a 200-person engineering org, that's 4M+ documents and growing.

Our constraint: p95 query latency under 200ms, with fresh data (no more than 60 seconds stale), across a graph that updates in real-time as your team works.

Why we didn't use a vector database

We started with Pinecone. It worked fine for semantic search, but it couldn't do the structured filtering we needed (e.g., "only docs from the last 30 days, in the engineering channel, that mention the launch"). We'd have to fetch a large candidate set and filter in application code — slow and expensive.

We tried Postgres + pgvector next. Better — we could combine semantic and structured filters in a single query. But as our corpus grew past 1M docs, query latency crept up. Index rebuilds became painful.

The architecture we settled on

Three pieces:

1. Postgres for structured data — doc metadata, ticket metadata, thread metadata. Indexed on workspace_id, channel, author, created_at.
2. pgvector for embeddings — stored in the same Postgres instance, queried with HNSW indexes.
3. Redis for real-time updates — when a new message lands in Slack, we update Redis immediately (so the next query sees it), and asynchronously write to Postgres + recompute embeddings.

The query path

When you ask Ion a question:

1. We embed your query with a small, fast model (50ms).
2. We do a hybrid query in Postgres: semantic similarity (HNSW) + structured filters (B-tree). Returns top 50 candidates in 80–120ms.
3. We re-rank with a cross-encoder model (30ms) and return the top 10 to the AI.
4. The AI generates an answer with citations.

Total: 160–200ms. We're happy with it for now.

What we'd do differently

Three things, in hindsight:

• Start with Postgres + pgvector. We wasted two months on Pinecone. Vector DBs are great if you only need semantic search, but they're bad at the hybrid queries real products need.
• Don't embed everything. Half our corpus was noise — automated messages, bot spam, one-word replies. A simple quality filter before embedding cut our index size by 40% and improved result quality.
• Cache aggressively. 80% of queries in a workspace are the same 20 questions. A 5-minute cache hit rate of 60% drops your effective p95 from 200ms to 80ms.

What's next

We're experimenting with fine-tuned retrieval models per workspace — early results suggest a 15–20% lift in retrieval quality for technical vocabularies. More to come.