An Open-Source PR Review Bot: LangGraph, Node.js and GitHub Webhooks

Pull requests pile up on almost every team. Senior engineers are usually the only ones who spot the subtle problems: the N+1 query, the hardcoded secret, the edge case nobody handled. They're also the busiest people around, so reviews get hurried or postponed, and either way something gets through.

Commercial AI review tools tend to be closed-source or expensive, often both. An alternative is a bot a small team can run on its own repositories, with no code sent to a black box. The design below is an open-source bot that listens for new pull requests on GitHub, reviews each one with a multi-step LangGraph agent and posts specific, line-level feedback as a PR comment.

What happens when a PR opens

1. A GitHub webhook delivers the event.
2. The GitHub API, through Octokit, fetches the diff.
3. A LangGraph agent examines the diff stage by stage for security issues, performance problems and architectural concerns.
4. The bot posts a comment on the PR with feedback that points at specific lines.
5. The review is saved to a database, so history accumulates over time.

The stage-by-stage part is the point. One prompt asking an LLM to "review this code" tends to yield vague, generic remarks. Splitting the job into a security pass, a performance pass and an architecture pass produces sharper results, since each stage has exactly one responsibility.

The stack

The stack stays close to what most Node.js teams already run. Showing off a new framework wasn't the goal. Something a team can host and trust was.

Flow and structure

The pipeline runs from the opened PR through a verified webhook into the agent, then back out as a comment:

GitHub PR opened
        │
        ▼
 Webhook (signed, verified)
        │
        ▼
 Express API ──► Fetch diff (Octokit)
        │
        ▼
 LangGraph Agent
   ├── Security check
   ├── Performance check
   └── Architecture check
        │
        ▼
 Post PR comment + save to PostgreSQL

Each check in the LangGraph flow is its own node. A team can test, tune or switch off one check without affecting the others, which helps when people care only about security findings and don't want architecture opinions on every PR.

Two code pieces worth showing

First, webhook signature verification. Without it, anyone who discovers the webhook URL could fire fake events and trigger bogus reviews:

function verifySignature(payload: string, signature: string) {
  const hmac = crypto.createHmac("sha256", process.env.GITHUB_WEBHOOK_SECRET!);
  const digest = "sha256=" + hmac.update(payload).digest("hex");
  return crypto.timingSafeEqual(Buffer.from(digest), Buffer.from(signature));
}

It's tiny, yet leaving it out is a common mistake in webhook-driven tools.

Second, the LangGraph flow is built from separate nodes rather than a single large prompt:

const graph = new StateGraph(ReviewState)
  .addNode("security", securityCheckNode)
  .addNode("performance", performanceCheckNode)
  .addNode("architecture", architectureCheckNode)
  .addEdge("security", "performance")
  .addEdge("performance", "architecture");

Every node gets the diff and returns only its own findings, and those are merged into one comment. Debugging gets easier this way. If the performance check returns something strange, there's one node to inspect, not a giant prompt.

What worked

• Splitting the review into separate security, performance and architecture checks gave clearly better, more specific feedback than a combined prompt. It was the biggest quality gain in the whole project.
• Verifying webhook signatures from day one meant security was part of the design, not something added later.
• Keeping review history in PostgreSQL, even in simple form, made patterns visible afterwards, such as which kinds of issues kept appearing across PRs.

What stayed hard

• False positives come with the territory. LLM-based review sometimes flags things that aren't problems, and the docs should say so openly. Trust counts for more than an impressive feature list.
• Cost adds up. Three LLM calls per PR are more accurate than one, but they cost more and take longer. A small team needs a balance between review quality and price it can afford.
• Making comments read like something a person wrote took more prompt iteration than the LangGraph logic did. Comments should be short, specific and tied to a line number, because nobody reads a five-paragraph AI essay in a PR thread.

The bot is one in a family of small open-source experiments that apply LLM orchestration with LangChain and LangGraph to concrete workflow problems, such as document Q&A with RAG, policy Q&A for citizen-facing services and turning voice notes into Agile user stories, rather than building generic demos. It isn't a finished, deploy-anywhere product. It's a working starting point that a team can build on and harden for its own setup.

Trying it locally

The repository is quick to run. Clone it, install dependencies and start the dev server:

git clone https://github.com/Srameshgitnow/agentic-code-reviewer.git
cd agentic-code-reviewer
npm install
npm run dev

The README covers the full setup, the environment variables and how to connect the GitHub webhook. Feedback on false positives or ideas for the LangGraph flow are the kinds of reports that help most.