Agents API Hosted Sandbox Test: CSV Runner, Artifact Checks and Failure Handling

This experiment starts from a concrete question: can an Agents API session accept a file, work on it inside a sandbox and return a result that can be verified? A small Python runner answers it. The input is a transactions CSV with four sales, one refund and one invalid row. The agent writes a JSON summary, and the application works out the expected answer on its own so the two can be compared. All opinions here are personal and come from testing with a personal OpenAI account on a Pro plan, along with the documentation available at that time.

On September 10, 2026 the Agents API went into public beta from OpenAI, exposing a managed Codex harness that drives the model-and-tool loop. The calling application contributes the task and chooses where commands run. What follows traces how the runner is built. The five recorded trials came from an earlier version of it. The revised runner adds reproducibility and saved evidence, and so far it has only been checked offline.

01 How the Agents API fits together

Four pieces need to stay distinct. The agent carries the model, instructions and tools. The environment supplies optional compute. The session holds the work together over time. Events and items report what is happening now and what has been stored. A turn is a single trip around that loop.

The hosted arrangement in this experiment looks like this: the Python runner opens the session and fetches results, while OpenAI operates the harness and the execution sandbox.

The first design question is where the agent loop should live. With the Agents API, OpenAI hosts the harness and keeps sessions durable. The Agents SDK runs the loop in your own application, so you pick storage and integrations. The Responses API offers direct model access and leaves the surrounding workflow to you.

Responses can also run hosted shell commands, so a CSV calculation by itself doesn't need the Agents API. The runner uses the API to exercise the whole path from session to artifact. Every trial creates a fresh session and runs one task. It tests neither multi-turn persistence nor subagents.

02 Why a hosted sandbox

The job needs only somewhere to put the CSV and the Python file the agent generates. It requires no private network and no custom image, so the runner picks openai_hosted with sandbox networking turned off. The alternatives mostly change who owns the workspace and how files come back out.

The value none means there is no environment filesystem and no built-in Bash execution, though application functions and remote MCP servers may still run code elsewhere. With self_hosted, OpenAI continues to run the harness but you operate the execution environment.

Output handling belongs in this choice. Files in a self-hosted environment are not published through OpenAI's Artifacts API, even when the agent writes them under /workspace/outputs. Your application has to retrieve and preserve them itself.

Hosted settings that matter here

The hosted guide also covers environment variables, skills, plugins and capability directories. Here, seeded files and disabled sandbox network access are sufficient. Remote MCP connections start from OpenAI's service, so sandbox egress settings shouldn't be described as policy for every tool connection.

03 Inside the CSV runner

The code sits in openai-agents-api/sandbox_check.py, and the full example is in the openai-agents-api directory of the techforum-repo ai-agents-experimentation repository on GitHub. The runner generates amounts as integer cents and saves its seed. Each CSV includes awkward cases on purpose: currency symbols, thousands separators, stray whitespace, a negative refund and an N/A entry.

The seed-42 input

amount,note
 $981.96 ,sale
" $1,838.10 ",sale
 $42.78 ,sale
-$100.12,refund
N/A,voided
 $155.92 ,sale

The local generator derives the reference below straight from those integer amounts. It is the expected result for this fixture and not a fresh live agent response.

{
  "total_cents": 291864,
  "valid_rows": 5,
  "invalid_rows": 1
}

The instructions sent

The agent is handed the CSV plus a written task, and the expected answer never leaves the application. The gist of the task: create /workspace/outputs/calculate.py and execute it with Python; load /workspace/data.csv; trim whitespace, dollar signs and thousands separators from each amount; ignore values that aren't numeric but keep negative refunds; compute with Decimal or integer cents, never binary floats; produce /workspace/outputs/summary.json with exactly three integer fields named total_cents, valid_rows and invalid_rows; count each data row one time and leave out the header; and retain calculate.py as an output artifact.

The function run_trial() opens a session through client.beta.agents.sessions.create(), passing gpt-6-astra and those instructions as the agent, and seeds /workspace/data.csv through inline base64. The environment is openai_hosted with network.access set to disabled. That is the same API path the original experiment followed.

What gets checked and saved

1. Track the right turn. After fetching the session, the runner polls the root turn. A failed session, a required action, a failed turn, a cancellation and an exhausted polling budget each end in a distinct outcome of their own.
2. Fetch both requested files. Each of summary.json and calculate.py is matched on path and on turn ID, which means an absent or outdated artifact can never pass.
3. Validate the summary. Exactly three integer fields must equal the local reference: the total in cents, the valid-row count and the invalid-row count. The parser refuses extra, missing, duplicate, boolean, float and non-finite values.
4. Keep the evidence. For every trial the runner stores the input, expected answer, request, newest session, turn records, saved items, artifact metadata, fetched files and a result file that machines can parse.

The artifact guide explains how files are tied to turns, and the observability guide covers examining command activity. The automated pass confirms that files exist and the structured output is exact. The saved script and items still deserve a look before anyone claims how the calculation was carried out. Downloaded code is never executed on the application's machine.

04 What the five recorded trials show

The original run recorded its results on September 15, 2026, and each trial used a newly randomized CSV. The figures were kept from that run. They are client-side observations made after the create response, and the timings exclude both session creation and artifact download.

Every one of the five totals reported by the agent agreed with the reference figures. That check used floating-point arithmetic with a tolerance under one cent. The timing records when polling first saw a turn and a settled session, so it measures neither precise server execution time nor end-to-end latency. Five successful trials describe what happened in one small run and say nothing about a general reliability rate.

What the updated runner changes

The revised runner repeats the same small experiment with a tougher definition of success. Integer cents and row counts are compared exactly, each seeded input is stored, and the generated script and session history are kept. Timing begins before session creation, with download and verification clocked on their own. Eight offline tests cover these checks, and the figures above still belong to the earlier live run.

05 Failure handling

The polling loop uses explicit HTTP timeouts, turns off automatic retries and checks the remaining trial budget between requests. That budget isn't a hard process-kill timer. A trial records its outcome before attempting cleanup, and a correct answer paired with failed cleanup still yields a nonzero exit code.

A session can have four statuses: idle, in_progress, requires_action and failed. The CSV task configures no application function tools, so if an action is ever required the runner saves that state and reports it for inspection instead of waiting forever.

Live streams don't replay events that were missed. To rebuild an application's view, reconnect and buffer incoming events while fetching the saved session items, then reconcile by item ID. This restores the display of saved work, though it doesn't guarantee that an interrupted command ran successfully.

One matching incident deserves a note. Earlier investigation notes described sessions sitting idle with no visible turns, and OpenAI's incident report for September 14, 2026 describes delayed or blocked turn starts from 13:30 Pacific that day, later marked resolved. The symptoms agree, but without request-level evidence they don't prove the cause of every local failure. Compare service status against the saved session and turn errors. The runner logs failed sessions, failed turns, missing artifacts, invalid summaries and wrong results as separate outcomes, and it no longer takes an unobserved turn as proof that the old incident returned.

Once available results are saved, the runner asks for cancellation if work is unfinished and tries to delete the session, with at most three attempts when deletion conflicts. Any cleanup error remains recorded in result.json. Should session creation fail before an ID is returned, cleanup by ID can't happen and the runner records that uncertainty.

A hosted workspace can expire after an hour without activity or keep-alives, and connected sandboxes get keep-alives between turns. Published artifacts survive environment expiry, but required outputs should be saved before the session is deleted.

06 When to bring your own compute

A dashed line in the wider architecture diagram applies only to self-hosted compute, and the CSV experiment here stays on OpenAI-hosted compute. Self-hosting starts to pay off if the task requires a specific image, access to a private network or machines already in your fleet. Inside that environment you launch codex exec-server, handing it the remote URL and environment ID that come back along with a restricted key scoped to the environment, and the broader application key should stay outside it.

From there the application is responsible for provisioning, storage, reconnecting and tearing things down. Deleting an OpenAI session doesn't stop provider compute, and reusing an environment ID doesn't bring files back onto a replacement machine.

Self-hosting doesn't make the whole service local either. The current Agents API keeps session state, supports US data residency and doesn't support Zero Data Retention.

Cost belongs in the decision. Model tokens and hosted compute are billed separately, with container pricing based on memory and billing intervals, so record real usage and don't estimate a trial's bill from elapsed seconds alone.

07 Running the experiment

Running it costs real money, because it needs separately billed API usage in the form of credits or an active API billing arrangement, and a Pro subscription doesn't pay for those calls. The runner calls GPT-6 Astra (gpt-6-astra), so many trials or lengthy agent runs can drain credits quickly, and hosted sandbox time adds compute fees. Do one trial first, look at the usage dashboard, and read the Astra pricing page before running more.

Inside the repository, set up a virtual environment and install the pinned SDK. An application API key is needed, with permission to read and write sessions and to run model inference. Every trial spends live API tokens and hosted compute.

cd openai-agents-api
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
export OPENAI_API_KEY="your-key"
python sandbox_check.py --seed 42       # Start with one trial
# python sandbox_check.py 5 --seed 42   # Optional: five trials

The first command runs a single trial with seed 42, and the optional five-trial command runs seeds 42 through 46. Results land in a new runs/<timestamp> directory, with one folder per trial. The SDK pin is openai==3.13.0, and the README documents output files and command options.

trial-001/
  data.csv           # Exact input
  expected.json      # Local reference
  request.json       # Submitted configuration
  session.json       # Latest retrieved session
  turns.json         # Turn status, timestamps, usage
  items.json         # Saved messages and tool calls
  artifacts.json     # Published file metadata
  summary.json       # Downloaded result
  calculate.py       # Downloaded agent script
  result.json        # Checks, timing, errors, cleanup

Failed runs keep whatever files were obtained before the failure. A zero exit code requires exact output checks and confirmed session deletion for every trial. To test the runner locally without any API calls:

python -m unittest discover -s . -p test_sandbox_check.py -v

Without touching the network, the test suite exercises the reference math, malformed model output, stale or absent artifacts, sessions and turns that fail, required actions, request errors, deadline handling and broken cleanup. The revised runner hasn't yet been validated against the live service.

The lasting value of the project is a repeatable way to inspect how the Agents API moves from file to artifact. A completed turn is one signal, while a downloaded and independently checked output is the real acceptance test. Documentation was checked on September 15, 2026, trial values come from the original experiment with no new live measurements, and the linked guides describe the current beta, which may change.