Diagnosing Slow Node.js APIs: Measure Latency Before You Optimize

An endpoint takes close to a second, yet CPU sits around a third of capacity, memory is unremarkable and the database dashboard is green. Tuning whatever you suspect rarely helps. This guide walks a request stage by stage so you can locate where the time goes, plus the instrumentation habits that keep the investigation cheap and trustworthy.

Time the request before blaming the runtime

Take a simple route that loads an order by its id and returns it as JSON.

app.get("/orders/:id", async (req, res) => {
  const order = await getOrder(req.params.id);
  res.json(order);
});

Suppose responses take roughly 900ms. Rather than blaming Node.js, wrap the awaited call with performance.now() and see how much of the total it accounts for.

const start = performance.now();

const order = await getOrder(req.params.id);
console.log(
  `getOrder: ${performance.now() - start}ms`
);
res.json(order);

The numbers usually settle the question quickly:

Request:       910ms
getOrder():    870ms
Node.js:        40ms

Nearly all of the latency lives inside getOrder(); the surrounding JavaScript costs about 40ms. The time is being spent waiting on something downstream.

Low CPU does not mean a healthy service

A utilization panel like the following looks reassuring:

CPU: 32%
Memory: 48%

CPU measures work being done, and a request parked waiting burns almost none. Typical things a Node.js process waits on:

• acquiring a database connection
• the query itself
• another HTTP service
• general network I/O
• a message queue
• a lock

A slow service with idle CPU is usually not busy; it is blocked.

Watch event-loop delay, not just CPU percentage

Every callback in a Node.js process shares one event loop, so the gap between when a timer should fire and when it does is a direct health signal. monitorEventLoopDelay from node:perf_hooks samples that gap into a histogram; here it uses a 20ms resolution and prints p95 and p99 every five seconds.

import { monitorEventLoopDelay } from "node:perf_hooks";

const histogram = monitorEventLoopDelay({
  resolution: 20
});
histogram.enable();
setInterval(() => {
  console.log({
    p95: histogram.percentile(95),
    p99: histogram.percentile(99)
  });
}, 5000);

If the output looks like this, the loop is fine and will not explain a 900ms request:

p95 event-loop delay: 8ms
p99 event-loop delay: 14ms

Numbers in the hundreds of milliseconds tell a very different story:

p95: 180ms
p99: 650ms

Now something is monopolizing the loop. A typical culprit is a synchronous, CPU-heavy function called directly inside a handler:

app.get("/report", (req, res) => {
  const result = expensiveCalculation();
  res.json(result);
});

While expensiveCalculation() runs, no other request on that process progresses. On a multi-core host the aggregate CPU graph can still look moderate, because one saturated core is averaged with idle ones, which is why loop delay is often more honest than CPU percentage. The usual remedy is moving such work to a worker thread, a job queue or a separate service. A related failure mode, where microtask scheduling rather than computation starves the loop, is covered in how process.nextTick can starve the event loop.

When the database looks guilty

Next, instrument the query the same way.

const start = performance.now();
const result = await db.query(
  "SELECT * FROM orders WHERE customer_id = $1",
  [customerId]
);
console.log(
  `DB query: ${performance.now() - start}ms`
);

The breakdown now points squarely at the database:

Total request: 850ms
Node.js:        15ms
DB query:      810ms
Serialization:  25ms

Before rewriting SQL, check whether the query was slow or the request was waiting for a connection.

Pool exhaustion masquerading as a slow query

Consider a fleet sized like this:

API instances:        10
Connections/instance: 20

Total connections:   200

Then a burst of traffic arrives:

Requests:          500
Active DB queries:  20
Waiting requests: 480

Only 20 queries can run per instance, so the rest queue inside the pool. A query might take 30ms, but a request at the back of the line waits hundreds of milliseconds before reaching the database. The application timer reports:

DB call = 500ms

The database itself reports:

Query = 30ms

A slow query calls for indexes or a better plan; a long wait calls for pool sizing, shorter transactions or fewer round trips. Measure the phases separately instead of one "database latency" figure:

Connection wait
      +
Query execution
      +
Result processing

Most pool libraries expose idle, total and waiting counts; exporting them is the quickest way to tell the cases apart.

Downstream services and sequential awaits

Many endpoints assemble a response from several dependencies, one after another:

const customer = await customerService.get(id);

const payment = await paymentApi.get(
  customer.paymentId
);
const orders = await orderService.get(id);

Timing each call separately exposes the outlier:

Customer API:    40ms
Payment API:    700ms  ← 🚨
Orders DB:       35ms
Node.js:         15ms

The payment service dominates, and because each await waits for the previous one, latencies stack:

40ms + 700ms + 35ms + 15ms
≈ 790ms

Independent calls can start together. The customer and order lookups only need id, so Promise.all overlaps them:

const [customer, orders] = await Promise.all([
  customerService.get(id),
  orderService.get(id)
]);

The payment call still waits, since it needs customer.paymentId. Do not parallelize by reflex: extra concurrency adds pressure on pools, APIs and memory, and under load can worsen the tail.

Percentiles tell you what averages hide

A dashboard reporting this seems fine:

Average latency: 120ms

The percentile view of the same traffic is less comforting:

p50:  80ms
p95: 180ms
p99: 2.8s

The average describes an ordinary request; the tail describes users who hit a saturated pool or slow dependency. Track the distribution:

p50
p95
p99

Instrumentation that helps instead of hurting

Measurement costs CPU, storage and attention. A few habits keep it useful.

Measure at boundaries, not in every helper

Timing every function this way produces more noise than insight:

const start = performance.now();
// ...
console.log(performance.now() - start);

Measure where a request crosses into another system, because that is where waiting happens:

HTTP request
   ↓
Database
   ↓
External API
   ↓
Queue
   ↓
Response

Keep per-request logging under control

A log line on every request becomes expensive at high throughput:

console.log({
  path: req.path,
  duration,
  userId: req.user.id
});

Use structured logging with levels and sampling, and keep these out of logs entirely:

passwords
tokens
authorization headers
payment information

A duration is not a diagnosis

Seeing a figure like this does not prove the SQL statement took that long:

DB: 700ms

That span can include several distinct costs:

Connection acquisition
+
Query execution
+
Network
+
Result transfer
+
Client-side processing

When a number surprises you, split it into those pieces before acting on it.

Avoid high-cardinality metric labels

Tagging a latency metric with a user id looks harmless:

http_request_duration{user_id="123456"}

With millions of users that means millions of time series. Use bounded dimensions instead:

route
method
status_code
service

A well-labelled series then looks like this:

http_request_duration{
  route="/orders/:id",
  method="GET",
  status="200"
}

Per-request detail belongs in logs and traces.

Sample deliberately

A common policy traces a fraction of normal traffic and everything interesting:

Normal traffic → sample a percentage
Errors         → capture aggressively
Slow requests  → capture aggressively

The ratio depends on traffic, tooling and budget; the goal is enough evidence to explain failures, not completeness.

Compare before and after, across several signals

"It feels faster" is not evidence. Record the percentile before a change:

Before:
p95 = 820ms

And again after it:

After:
p95 = 410ms

Then confirm that nothing else moved in the wrong direction:

error rate
CPU
memory
DB load
connection pool usage

A faster query that doubles pool usage or database load has simply moved the problem.

Follow a single request with tracing

A route-level metric only tells you the total:

GET /orders = 980ms

A distributed trace breaks the same request into spans, so the cost of each stage is visible at a glance:

GET /orders
│
├── Node.js processing       15ms
├── DB connection wait      120ms
├── PostgreSQL query         40ms
├── Payment API             700ms
├── JSON serialization       20ms
└── Network                   5ms
                              ───
                              900ms

This division of labor is worth remembering: metrics alert you that something is wrong, and traces show you where it is wrong.

A checklist for walking the request path

When latency climbs, resist adding servers or CPU first. Trace the path a request takes and measure each hop:

Request
   ↓
Queue / Load Balancer
   ↓
Node.js
   ↓
Event Loop
   ↓
Connection Pool
   ↓
Database
   ↓
External APIs
   ↓
Network
   ↓
Response

At each stage, ask a concrete question:

Is the event loop blocked?

Are requests waiting for connections?

Are database queries slow?

Are downstream APIs slow?

Are we waiting on network I/O?

Is the queue growing?

Is garbage collection causing pauses?

Is the response itself expensive to serialize?

Every one of these can be answered with data, so there is no need to guess.

What the evidence usually shows

Return to the opening scenario:

API latency: 900ms
CPU: 35%
Memory: 48%

After measuring each stage, the picture is completely different:

Event loop:          12ms
DB query:            35ms
Connection wait:    120ms
Payment API:        700ms
Serialization:       20ms
Network:              5ms

No runtime rewrite, bigger instance or extra memory would help. The work is the 700ms payment dependency (caching, a timeout with fallback, or moving it off the request path) and the 120ms connection wait (pool sizing or query volume).

Key takeaways

The instinctive loop looks like this:

Slow API
   ↓
Add more servers

The productive one looks like this:

Slow API
   ↓
Measure
   ↓
Break down latency
   ↓
Find the bottleneck
   ↓
Fix the bottleneck
   ↓
Measure again

• Time the whole request first, then split it at every boundary where it leaves the process.
• Treat event-loop delay as the primary health signal for Node.js; CPU percentage can hide a single blocked core.
• Separate connection wait from query execution before touching SQL.
• Judge latency by p95 and p99, not by averages.
• Keep metrics low-cardinality, logs sampled and free of secrets, and put per-request detail in traces.
• Verify every fix against the same percentiles and neighboring signals it might have affected.

A slow system is manageable once you can see it; until then every change is a guess. For a follow-on view of which fixes to prioritize once the bottleneck is known, see a priority-ordered framework for Node.js API performance.