AI Interview · 2025 architecture

Building a multi-agent interviewer from first principles.

Before I could rely on an agent SDK, I built the runtime myself: agent queues, interview state, specialist roles, live coding analysis, memory, and evaluation.

Soumil ChughSeptember 20266 minute read
01 · What I built

An interview is a live system, not a sequence of prompts.

The platform had to listen, decide who should speak, follow an interview plan, react to code, remember earlier answers, and produce evidence-based feedback.

I built AI Interview in 2025 as a full-stack product. A React client handled the interview room, audio, video, and coding workspace. A Python and FastAPI backend ran the agents and streamed events over WebSockets.

There were three product modes: candidate practice, company-led screening, and invited candidate interviews. The same runtime had to support different jobs, interview plans, panelist personalities, and model providers.

The important constraint

This work predates the agent SDKs I use today. Agent lifecycle, routing, state, memory, and messages were explicit application code.

02 · The agent system

One coordinator, several focused agents.

The Master Agent owned the interview. It did not answer every question itself. It kept the shared state and sent the right context to the right specialist.

01

Configuration Agent

Before the call, it turned the job details and examples into an interview plan, coding activity, starter code, and panelist profiles.

02

Master Agent

During the call, it tracked the round, topic, subtopic, time, last speaker, waiting state, and audio playback state.

03

Panelist Agents

Each interviewer had a professional profile and personality. Only the selected panelist received a turn command.

04

Activity + Evaluation Agents

One watched the coding task while it happened. The other combined the transcript, criteria, code analysis, and panelist assessments after the interview.

03 · Information flow

Follow one event through the system.

Choose an event to see the path it takes. The browser and agents never communicate directly; the Master remains the control point.

For every conversational turn, the Master asks the model for two bounded decisions: is this topic complete? and who should speak next? Older context is summarized, the current exchange stays detailed, and advice for multiple panelists is generated in parallel.

The selected panelist receives one structured message. Its live path combines internal reasoning and the spoken answer in a single model call, then returns a typed response to the Master. That avoids extra model round trips while keeping the wider workflow explicit.

04 · Manual orchestration

The SDK was queues, tasks, and typed messages.

Each panelist and the Activity Agent ran as its own asynchronous task. The Master wrote commands to their queues, and agents returned results through a shared receiving queue.

Deterministic shell

Code owns control

Locks, queues, timeouts, state transitions, database writes, and WebSocket events are regular code.

Model judgment

LLMs make narrow choices

The model chooses the next speaker, judges topic completion, writes advice, and produces a response.

Concurrency

Work happens in parallel

Panelists remain alive as background tasks, and per-panelist advice is generated together.

Provider boundary

Models remain replaceable

A shared provider interface supports different OpenAI, Gemini, Groq, Grok, DeepSeek, and Perplexity paths.

05 · Memory and context

Keep the current moment rich. Compress the past.

The interview plan doubles as a memory index. Dialogue is stored under its round, topic, and subtopic. As the conversation advances, the system creates subtopic and topic summaries.

This gave the Master and panelists enough evidence for a coherent follow-up without sending the entire interview transcript on every model call. The same memory graph is persisted so evaluation can reconstruct the evidence later.

06 · Evaluation

Feedback is a second workflow, not an afterthought.

The Evaluation Agent reloads the interview plan and memory, then grades each topic and subtopic against its own criteria. Coding performance is included as evidence rather than scored in isolation.

For deeper technical discussion, the evaluator can generate focused subqueries and extract supporting details before scoring. Question-level results are then merged into criterion-level scores and an overall report.

07 · What I learned

Agent quality comes from the system around the model.

One owner for shared stateSpecialists stay focused when one coordinator owns the interview timeline and routing.
Use models for judgmentSpeaker choice and follow-up quality need reasoning. Locks and lifecycle management do not.
Context needs structureA hierarchy of topics and summaries is more useful than one growing transcript.
Latency shapes architectureCombining related reasoning steps and parallelizing independent work keeps a live conversation moving.
Side channels add realismThe Activity Agent lets interviewers react to what the candidate is building, not only what they say.
Explicit code teaches the contractsBuilding orchestration manually made agent messages, state, memory, and failure boundaries visible.
Looking back

A modern agent framework can now provide parts of this runtime. The hard product decisions remain the same: what each agent owns, what information it sees, when a model should decide, and how evidence survives the run.

Open source

Read the implementation behind the architecture.

View the repository ↗