How Many Requests Can a Node.js Server Really Handle?

Anyone new to Node.js eventually asks the same practical question: how many requests can one production Node.js app handle, and how many server instances does a given amount of traffic call for? The answer depends on a lot more than the framework.

A target worth aiming at

The design platform PicsArt reportedly serves around 40,000 requests per second with a backend that's mostly Node.js.

That number is huge, and it depends on plenty of things besides the backend server. For this exercise those other factors get set aside, leaving only backend hardware and software, and Node.js in particular.

What synthetic benchmarks show

The team behind Fastify published a benchmark in which Express handles roughly 15,000 requests per second, while the bare Node HTTP module reaches about 70,000.

A 2015 MacBook Pro 15" produced almost identical figures. Even 15K rps from Express looks great, but benchmarks like this run against a tiny handler, which is far from a real API. What makes an API "real-world" is the data, how it moves and how components interact, so a realistic handler has to talk to a database or other services.

The raw-versus-Express gap isn't very important anyway. Established frameworks such as Express are usually the better pick, because no single instance is going to serve 100K requests per second in practice. So Express was installed for these tests, with a small handler performing varied operations.

The test handler

The handler runs six steps:

1. Insert a new user into the database.
2. Fetch 20 users from the database.
3. Stringify and parse the array of 20 users.
4. Recursively convert those 20 user objects to camelCase.
5. Sort the array.
6. Do light computation and stdout work, such as Date.now and console.log, which also consume compute.

The response weighed about 12KB. Every user object looked like this, at 609 bytes:

Result 1: Node.js on Heroku with AWS RDS

The app was deployed to Heroku on $25-per-month hardware, and requests were sent at the highest rate that didn't raise average latency. The instance handled 31,000 requests over 60 seconds, an average of 515 requests per second.

Result 2: Django on Heroku with AWS RDS

To see what a technology that isn't single-threaded would do, an equivalent handler was built in Django under identical conditions and put on the same server. It came out a little worse than Node.js, reaching 23,000 requests in 60 seconds, or 389 per second.

Result 3: three Node.js instances on AWS Elastic Beanstalk

A third setup modeled a real production deployment on AWS. Database and backend both lived in Frankfurt (eu-central-1), the nearest region, about 3,000 km from the tester. Three Node.js instances sat behind AWS load balancing. They handled 19,000 requests in 10 seconds, which is 1,858 requests per second. A continuous load test then pushed through 1.5 million successful requests in 13 minutes.

Scaling to 40,000 requests per second

One instance handles about 600 rps, so what would 40,000 require? Assume the average endpoint on a large platform is twice as heavy as the test handler. That gives 40,000 / (600 / 2) = 133 instances spread around the world for a global unicorn-scale app. Not a big deal.

Heavier endpoints could be 3x more demanding, but caching and event-based communication can win throughput back, so 2x is a reasonable middle estimate.

What slows an API down

Countless factors can drag an API's speed down. These are the performance-related ones found in most server-side web apps.

Data size

The volume stored in or fetched from the database matters, especially with relational databases, where large payloads often mean complex queries or heavy work between the DBMS and its disk memory. In every case more bytes mean more transmission time. Storing files in the database is an antipattern, and object storage is the better home. Ideally a record stays within tens of kilobytes, usually 1 to 20KB.

Distance to the database

Greater distance between the backend and the database means longer trips for queries and results. Take two backends, one in Virginia and one in Frankfurt, with a single database in Virginia. A US user's request reaches the US backend in about 10 to 40ms, and the backend-to-database hop takes 1 to 5ms. That's fine. A German user reaches the Frankfurt backend in 10 to 20ms, but the query to Virginia takes 120 to 130ms. A single-region app should therefore keep data and backend logic as neighbors, and a global app must be ready for some complex engineering to stay fast.

Database hardware

Hardware has a big effect too. A small app should buy better hardware to fit its traffic, and otherwise it's back to that complex engineering.

Database configuration

Databases ship with default settings so they're not another headache to manage, but at scale they need attention. Maximum connection count is one setting to watch. It can bottleneck the app or overwhelm the database server, and client libraries such as ORMs often change it. Connection timeout is another.

Number of database calls

A real API may make tens of separate database calls to finish a job, and it may call other services that each make their own tens. One memorable case was a banking monolith API with a 9-second response time in the cloud dev environment. Its handler ran 300 lines with around 30 database calls and about 10 internal service calls. This tends to happen with a "code first" approach and not a "design first" one.

Fixes depend on the business logic. Queues and other asynchronous techniques can move some database work into the background, which requires splitting large services into small ones. Joining queries can also reduce the number of calls.

Hooks

Popular ODMs and ORMs offer hooks or middleware, which are comfortable for handling certain logic. During bulk operations they can become a hidden bottleneck, since plugins such as a camelCase or pascal_case converter use compute and time as well. Hooks help, but what's written in them and where deserves a second thought.

CPU-heavy work

CPU-intensive tasks slow the server and also leave the end user waiting. In Node.js many of them rely on the thread pool. That doesn't block asynchronous I/O, but it still slows the API. A sign-up click, for instance, waits on the thread pool to hash a password, which takes upwards of 100 milliseconds.

Event loop bottlenecks

Node.js executes code on the event loop, on a single thread, so code performance affects the whole application. JSON.parse and JSON.stringify can slow APIs down, and nested loops and recursive functions are dangerous. Ignoring how Node.js works can make an application unavailable, depending on data size and traffic.

Poor infrastructure design

Large infrastructure doesn't just have more problems than a monolith, it has different ones, and they often need complex architectural answers. With many components, a bad architecture hits API speed hard and makes the system resource-inefficient, so more is paid for something that could cost less. When designing complex applications, think about which problems a technology solves and know the alternatives. Consider how requirements might change in two or six months, and test each decision against that guess. Think about the SLA and about non-functional requirements, including what happens if traffic grows.

Component scale and setup

Components added to the infrastructure can slow the whole platform. Take RabbitMQ, which has plenty of settings like any technology. Defaults are fine to start with, but with research and a closer look at a specific component, a setup review or a configuration refactor can improve the infrastructure considerably. Components must also scale with usage so they don't become a bottleneck.