Designing a Multi-Agent Job Finder: Pipeline, HyDE and Observability

Agents, agentic AI, multi-agent systems: the vocabulary is everywhere, and every week brings another model, framework or orchestration pattern. Prompting, RAG, tool calling, MCP, memory layers. The ecosystem moves faster than most practitioners can follow. Underneath the noise, the hard questions haven't changed:

• Where should a problem be cut into pieces of reasoning?
• Which calls belong to an LLM, and which should never be left to one?
• How would anyone confirm the system does what they believe it does?

A new framework release doesn't make those easier. To keep things concrete, this case study uses a problem millions of people have: finding the right jobs in an overwhelming pile of listings. It designs an end-to-end AI system for it and uses the design to answer three practical questions. What does agentic mean in architectural terms? When do several agents add value, and when are they overhead? And what makes an AI system observable, reliable and fit for production?

Job hunting isn't really search

Most people treat a job hunt as a search task. It's closer to a matching task. Nobody is really after postings that contain certain keywords. The underlying question is harder: given this experience, these skills and this career direction, which openings are a genuine fit?

Job boards handle it as search. You type a few terms, add a location and hope something relevant comes up. A resume, though, carries years of experience, domain knowledge, technical depth, transferable skills, progression and context, and a search box squeezes all that into a handful of words while assuming that matching words means matching intent.

Everyone knows how that goes:

• Scrolling hundreds of listings across several platforms
• Judging relevance one posting at a time
• Running into duplicates, expired ads and broken links
• Wondering whether the target roles were the right ones to begin with

The tedium isn't the core issue. The real gap lies between what a candidate's history says and what a posting actually wants, and closing it takes interpretation, context and judgment. In short, it takes reasoning, which is where large language models are strong. Suppose a system could read a resume, infer sensible career paths, find openings and rank them by real fit instead of keyword overlap. That's the problem this system was built for.

The pipeline, step by step

It begins with one action: upload a resume. After that nobody touches a search box, scrolls listings or decides which roles are worth applying to. The system takes over and runs nine steps, each with a defined job.

1. Resume Parser Agent. Acting like a veteran recruiter, it turns the resume into a structured profile instead of a keyword list. The profile covers jobs held, skills, areas of expertise, the arc of the career and a short ATS-style narrative that all later agents treat as the canonical description of the candidate. Model: Claude Sonnet.
2. Profile Recommender Agent. It proposes two or three target job titles that fit the candidate's level, domain and skills, along with search-query variants for each. Model: Claude Haiku.
3. Human confirmation gate. This is the only point where a person decides anything. The user reviews the suggested profiles, picks the one that feels right, optionally adds location and seniority preferences, and confirms. The pipeline waits until then. No model is involved, only human judgment.
4. Job search. With the confirmed profile, parallel queries hit five live job sources at once and return roughly 100 to 150 raw listings. No single source has everything, and running them side by side keeps latency down while widening coverage. No LLM, just API calls.
5. URL validator. A light HTTP check tests each listing's URL. Dead links, 404 pages and aggregator redirects are more frequent than you'd expect, and anything that doesn't resolve cleanly is dropped. No LLM.
6. Deduplication. Different sources often return the same job, so duplicates (same company, title and location) are removed before any costly step. Clean input, clean output. No LLM, just in-memory logic.
7. Semantic prefilter. This step is worth understanding in detail. It uses Hypothetical Document Embeddings (HyDE). Instead of comparing the resume with listings directly, the system first generates hypothetical job descriptions from the resume, then finds the real listings closest in meaning to those synthetic ones. Listings under a minimum similarity threshold are dropped before any pricey LLM scoring. The reasoning for why this works, and why direct comparison doesn't, appears in design choice 5 below. Models: Claude Sonnet for generating descriptions, Voyage AI voyage-3 for embeddings.
8. Ranker Agent. The LLM compares each job description to the candidate's complete parsed profile and rates four dimensions: experience fit, skill overlap, domain alignment and education requirements. Those merge into a single fit score, and each score comes with a rationale plus a gap analysis between profile and requirements, so the result is an explanation and not only an ordering. Model: Claude Haiku.
9. Finalization and observability logging. Ranked results go back to the UI, and every metric of the run, including token counts, latency and cost, is written to an MLflow dashboard for later inspection. No LLM.

That makes nine steps, four LLM calls, one human decision and a total runtime under three minutes.

The technology map

The diagram above places each major component of a current AI stack within this pipeline. Every system is a bundle of choices, so the next sections cover what was considered, what was rejected and what it cost.

Eight design decisions

1. LangGraph over CrewAI, AutoGen or a ReAct loop

The usual candidates were LangGraph, CrewAI and AutoGen, and inside LangGraph there's a further split between a directed graph and a ReAct loop.

CrewAI emphasizes autonomous collaboration, with role-playing agents that negotiate and delegate among themselves. AutoGen centers on conversational multi-agent dialogue. Both suit open-ended workflows. Both also bring coordination overhead and non-determinism that a fixed pipeline can't afford to debug.

ReAct loops shine when the number of steps isn't known upfront, as with an LLM browsing the web that can't predict how many searches it needs. Job finding isn't like that. The structure of the problem fixes the stages: parse before recommending, search before filtering, deduplicate before embedding. Nothing is discovered at runtime.

A directed graph fits all of that. Every transition is explicit, the human-in-the-loop interrupt is a first-class primitive, and state is a shared Pydantic model each node reads and writes. When something breaks, the failing node is obvious.

2. Specialized agents over one mega-prompt

Writing one big prompt that parses the resume, suggests titles and scores listings is tempting, and it's quicker to build. It also fails in murky ways that compound. A failure anywhere becomes a failure everywhere. It's impossible to tell which instruction caused a bad output, tuning one part risks regressions in another, and the context fills with instructions irrelevant to the task at hand.

Separate agents avoid this. A weak ranker score gets debugged by looking at the ranker prompt alone. Each agent has its own test script, so the parser can be regression-tested without the whole pipeline. And different agents can be routed to different models, as design choice 6 shows.

3. A human-in-the-loop gate

Most agent tutorials treat human review as optional polish, something to add so users feel involved. Here the gate is a core architectural decision.

The recommender offers two or three titles, and whichever profile the user confirms steers every later stage: the searches, the filters, the scoring. Let the system pick automatically and a wrong guess poisons all of it, producing a neatly ranked list for a profile the candidate never wanted. No other decision in the pipeline carries as much risk, and none is better handled by a person, since candidates know things about their own direction that a parser can't extract.

4. A prefilter ahead of the ranker

Real-world data is never clean, and listings from five APIs arrive with predictable problems:

• Dead links and 404s, from postings that expired and were never removed
• Duplicates, since the same job surfaces from several APIs at once
• Aggregator redirects, where a URL lands on a job board homepage and not on the listing

These aren't edge cases. They're the normal state of external API data. The validator and deduplication steps exist to clean the pool before anything expensive happens, so what reaches the semantic prefilter is valid, unique and real.

Then cost matters. Across production runs, the ranker accounts for 70 to 80 percent of total LLM spend. The parser costs a flat ₹2.7 to 3.0 no matter how many jobs flow through, and the recommender about ₹0.30, also flat. The ranker swings from ₹3 to ₹11 depending on how many jobs reach it, putting the total per run between ₹6 and ₹14. The prefilter keeps only the listings most aligned with the candidate, so the ranker works on a high-signal subset instead of the noisy raw output of five job APIs.

5. HyDE instead of direct comparison

The obvious approach is to embed the candidate's ATS summary and compare it with every job description by cosine similarity. It's simple, fast and needs no extra LLM calls. In testing, the results disappointed.

The cause is a mismatch in embedding space. A candidate summary reads as a first-person story, something like "shipped ML pipelines, owned model deployment, collaborated across teams." A posting reads as a third-person list of demands, such as "requires 3+ years of ML work, PyTorch skills and good communication." Even when both describe the same abilities, the two styles land in different regions, so direct comparison keeps underrating real matches.

HyDE sidesteps this by asking a different question: what would the ideal job description for this candidate look like? An LLM writes two hypothetical descriptions from the resume.

• JD1 is domain-anchored, written as if a company in the candidate's exact field were hiring someone with that background.
• JD2 focuses on transferable skills, written as if domain expertise were secondary and general capability the headline.

These synthetic descriptions sit in the same region as real ones, because they are job descriptions, only hypothetical. Comparing them with real listings becomes a like-for-like semantic match. Each real listing then goes to whichever synthetic description it's closer to. Closer to JD1 puts it in section 1, roles in the candidate's domain. Closer to JD2 puts it in section 2, broader opportunities.

One constraint held throughout: only general-purpose embeddings, no domain-specific models. Domain specificity comes from the HyDE content, which the LLM bakes into the synthetic descriptions, and not from the choice of embedding model. That keeps the system portable across professions.

6. Different models for different agents

Not every task needs the same reasoning power. The parser and the HyDE generator cost about ₹2.7 to 3.0 and ₹0.50 per run, fixed regardless of volume. Their quality drives everything after them, since a weak parse corrupts the profile and a weak hypothetical description corrupts the embedding match, so Sonnet is the right choice.

Scoring happens per job in the ranker, so it's the stage whose cost grows with volume, and it already eats 70 to 80 percent of the budget. Sonnet there would roughly triple that bill without better structured scores. Haiku does the job. Here's the routing:

7. One config file for every tunable

Agentic systems eventually develop a tuning problem. Swapping a model, tightening a threshold or changing a concurrency limit shouldn't mean hunting across five agent files for the right constant.

This system keeps a single YAML file that owns every knob in the pipeline, so no constants sit hardcoded in agent code and no magic numbers hide in prompts. It governs:

• Model per agent. Replacing Claude Haiku with a newer model is a one-line edit.
• Token limits and temperature, tuned agent by agent. Structured extraction in the parser uses 0.1 so output stays consistent, while the recommender uses 0.4 to vary its title suggestions a little.
• Concurrency and rate limits: semaphore_size: 4, batch_size: 4 and batch_delay_secs: 2, explained next.

8. Concurrent ranking, shaped by rate limits

The first ranker was sequential: one LLM call, wait, next job. At 40 jobs and about 2.5 seconds a call, that's 100 seconds of wall-clock time. Pipeline data bears it out, with the ranker alone taking around 95 seconds of a typical 166-second run.

Firing all 40 calls at once looks like the answer, and it collides with Anthropic's rate limits. Tier 1 access caps requests per minute, so 40 to 60 simultaneous calls exceed it immediately and come back as 429 errors. The earliest production run lacking any throttling handed back only part of the results: jobs 1 through 70 got scored before the API stopped answering.

The answer is a three-part concurrency design, controlled entirely from the config file:

• A semaphore (semaphore_size: 4) caps the calls in flight at any moment. Four at a time stays well within Tier 1 limits.
• Batching (batch_size: 4) scores jobs in groups of four, and each group finishes before the next starts.
• A batch delay (batch_delay_secs: 2) adds a two-second pause between groups for headroom against burst limits.

The payoff: 60 jobs across 15 batches score in roughly 90 to 95 seconds, about four times faster than the sequential baseline at the same job count, without hitting limits.

Observability

A system nobody can observe is a hope, not a system. A real incident shows it. In the first full production run, VOYAGE_API_KEY was missing on the EC2 instance. The HyDE prefilter quietly skipped embedding and forwarded all 89 jobs to the ranker, which tripped the rate limit at job 70 and delivered a partial list. With no observability that would look like random, baffling behavior. With it, tracing the cause took under a minute.

Observability here works on two levels:

• LangSmith traces every LLM call: the exact prompt, the response, per-call latency and token counts. It gives micro-level visibility into reasoning and is used during development to catch prompt regressions before they affect the whole pipeline.
• MLflow records every run with per-agent latency, total cost and token counts. Its job_audit.json artifact is the most useful diagnostic, a full record for each job that entered the pipeline: which HyDE section it landed in, its similarity score, which filter dropped it and why, its fit-score components and its final rank.

What maturity looks like

An AI system's maturity isn't measured by how autonomous it is. It's measured by how well people understand what it does, why it does it and where it goes wrong. Talk about agentic AI has optimized for the wrong metric. Autonomy makes an impressive demo and often a poor engineering property, because autonomous behavior is costly to observe and costly to debug.

For anyone building an AI system today, the useful questions aren't how agentic it is or how many agents it has. They are these:

• Can you observe what it's doing at each step?
• Can you explain a specific output without opening the source code?
• Can you recalibrate it when its behavior drifts?
• Can you tell good performance from silent failure?

A system that passes all four is a working ML system. Everything else is a prototype with good marketing.