Finding Hidden Prices by Reverse-Engineering Background API Calls

Second installment in the "How Scraping Actually Works" series

In the previous article, we ran into a wall: sometimes the price you need simply isn't present anywhere in the HTML that requests fetches. It's not embedded in JSON-LD, it's not tucked into a meta tag, it's nowhere to be found. That doesn't mean you're stuck — it usually just means the page pulls that piece of data in after the initial load, through a separate background call. The goal here is to track down that call and make it ourselves.

Step 1: Open the Network tab

We'll work through a concrete example: a product page on Target's site, target.com/p/dyson-ball-animal-3-upright-vacuum/-/A-85269291 (this is just for demonstration — the same approach works on virtually any product page).

1. Load up any product page.
2. Right-click on the page and choose Inspect (or press F12).
3. Switch to the Network panel.
4. Reload the page.
5. Apply the Fetch/XHR filter — this hides image, CSS, and font requests, leaving only calls that actually carry data.

Step 2: Locate the request you actually need

Go through the list one entry at a time, clicking each and checking its Response or Preview panel on the right side. You're hunting for a JSON payload that contains the price, the review count, or whatever value you're after.

There's a faster route: use the magnifying-glass search box inside the Network tab and type in a value that's visible on the page itself. For the Target example, typing "449.99" leads straight to a POST call named deferred_enrichment/modules. Its response body is a sizeable JSON object, and somewhere inside it you'll find "formatted_current_price":"$449.99.

Step 3: Reproduce the call yourself

Select that request and grab its full URL. In this case it's a POST request, but everything it needs is already present in the query string:

https://www.target.com/cdui_orchestrations/v1/pages/pdp/deferred_enrichment/modules
    ?tcin=85269291
    &key=9f36aeafbe60771e321a7cc95a78140772ab3e96
    &channel=WEB
    &device_type=DESKTOP

Here's how that looks when reproduced in Python:

import requests

url = "https://www.target.com/cdui_orchestrations/v1/pages/pdp/deferred_enrichment/modules"
params = {
    "tcin": "85269291",       # the product ID, from the page URL
    "key": "9f36aeafbe60771e321a7cc95a78140772ab3e96",
    "channel": "WEB",
    "device_type": "DESKTOP",
}

resp = requests.post(url, params=params, headers={"User-Agent": "Mozilla/5.0"})
data = resp.json()

price = data["modules"][0]["module_data"]["data"]["product"]["price"]["formatted_current_price"]
print(price)  # $449.99

That's the whole trick. No parsing library, no CSS selectors to babysit — you're simply sending the same query the page's own JavaScript sends behind the scenes.

Tip: in DevTools, right-click the request and pick Copy → Copy as cURL. That gives you the precise headers and cookies the browser attached, which you can drop into a client like Postman or convert into Python to figure out exactly which parts are actually required.

Step 4: When the API is GraphQL instead

Some sites route everything through GraphQL rather than a traditional REST setup. You can recognize this pattern because every call lands on the same endpoint (frequently /graphql), and the actual question being asked lives in the request body instead of the URL:

{
  "operationName": "GetPrice",
  "variables": { "productId": "12345" },
  "query": "query GetPrice($productId: ID!) { product(id: $productId) { price } }"
}

A number of sites take this further with a persisted query, where instead of transmitting the full query text, the client sends only a short hash that stands in for it:

/graphql/GetPrice/a1b2c3d4e5f6...?variables={"productId":"12345"}

That hash isn't something you can derive on your own — you have to lift it straight from DevTools. And because it's tied to the site's frontend build, an update on their end can change the hash and quietly break your script. That's a genuine ongoing cost of relying on this technique.

Step 5: Checking whether that API key is really private

Look again at the Target URL from Step 3 — notice key=9f36aeafbe60771e321a7cc95a78140772ab3e96 sitting openly in the query string. Is that something you're not meant to have access to?

Usually not. The quickest way to verify is to open View Page Source (Ctrl+U), then search for apiKey with Ctrl+F. You'll find that exact same key sitting in plain text, inside a configuration block that Target sends to every single visitor.

If a value is already sitting in the page's own markup — with nothing bypassed and no authentication involved — it was never confidential to begin with. It's simply a public key meant to run in the browser. (Occasionally it lives in a separate .js file instead; if View Source turns up nothing, try DevTools' Sources panel and search with Ctrl+Shift+F.)

One good practice: before you go to the trouble of reverse-engineering an endpoint, check whether the company already offers a documented public or partner API. Those tend to be far more stable than something you discovered by poking around in DevTools.

Step 6: When the data isn't coming from the retailer at all

Sometimes the request you track down doesn't even go to the retailer's own domain — instead it points to a third-party service like bazaarvoice.com or judge.me. That's a sign the retailer isn't running its own review infrastructure; it's licensing a vendor's system and simply embedding that vendor's widget on the page.

These vendor-run widget APIs tend to be guarded less tightly than a retailer's primary site, since they're designed to be dropped into thousands of unrelated storefronts rather than to specifically detect and block scrapers.

What this article deliberately leaves out

If you've followed every step above and the request you need still comes back with a 403, an empty body, or something that resembles a security checkpoint rather than actual data, that's not a bug in your code. It means the site has actively detected that you're not a normal browser session. That challenge is substantial enough to deserve its own discussion, which is exactly where the next article picks up.

Glossary

Network tab — the DevTools panel that logs every request a page issues, along with what was sent and what came back.

XHR / Fetch — the two mechanisms JavaScript uses to make background requests without a full page reload; the Network tab can be filtered to show only these.

REST API — a common API pattern where each resource has its own dedicated URL, such as /products/123/price.

GraphQL — an API style built around a single endpoint, where the request body specifies exactly what data is being requested.

Persisted query — a GraphQL shortcut in which the full query text is replaced by a compact hash that the server already recognizes.

API key — a value attached to a request to identify it; some are genuinely private, but plenty are public, client-facing keys meant to be visible in anyone's browser.

Copy as cURL — a DevTools option that converts a captured request into a runnable command-line call, handy for inspecting its exact headers and cookies.

Third-party widget — a feature, such as a reviews section, that a site doesn't build in-house but instead embeds from another company's product.