Debugging Serverless Apache Spark with Gemini, MCP and Agent Skills
Debugging serverless Apache Spark with Gemini: from pasting tracebacks into chat to an agent that reads batch logs via an MCP server and fixes PySpark.
This article is published in English.
Sources: Debugging Serverless Apache Spark using Gemini with MCP
Anyone who runs Apache Spark at scale knows how tedious it is to work out why a batch job died. The real cause tends to hide inside hundreds of executor log lines, JVM stack traces and Cloud Logging streams. You end up hopping between browser tabs, hunting through Cloud Storage paths and hand-editing PySpark scripts on your laptop.
Generative models can speed that up, but how much they help comes down to one thing: the context they can see. The sections below walk through four levels of context, from none to a reusable team playbook, using Managed Service for Apache Spark, the Antigravity CLI and Google's remote Spark Model Context Protocol (MCP) server.
Level zero: pasting into a chat window
A PySpark job fails with something vague like a SparkException or AttributeError. The reflex is to copy the traceback into a web chat. Take this one, from a script named spark_job.py:
py4j.protocol.Py4JJavaError: An error occurred while calling o80.load.
org.apache.spark.SparkException: Job aborted due to stage failure.
Traceback (most recent call last):
File "spark_job.py", line 26, in main
df_with_status = df.withColunm("status", lit("active"))
AttributeError: 'DataFrame' object has no attribute 'withColunm'
A chat model will spot the typo quickly and suggest changing withColunm to withColumn. But it works blind:
- It can't open your local
spark_job.py, so you find the line, edit and save by hand. - It can't check serverless batch status or fetch live execution logs.
- It can't look at your buckets, service account permissions or batch settings.
If the failure is a missing bucket or a subnet without Private Google Access, the chat can only offer general guesses.
Level one: file context from the terminal
The Google Antigravity CLI, invoked as agy, is a terminal-first coding agent. It runs in your workspace, so it can read and change your files directly.
Two separate layers of authentication are involved. The first is Application Default Credentials: running gcloud auth application-default login gives gcloud commands, the client libraries and the Spark MCP server a way to act on your project's resources. The second is the CLI's own session. On first launch, agy asks you to sign in with a Google account and trust the workspace, which gives it access to Gemini models.
Then make sure your project ID is exported:
export GOOGLE_CLOUD_PROJECT="$PROJECT_ID"
By default the CLI asks permission before reading or writing files. With the non-interactive print flag (-p), anything needing confirmation is quietly refused unless auto-approval is set up. A settings.json with toolPermission set to always-proceed, plus an allowlist for file operations, lets the agent inspect and fix scripts:
mkdir -p ~/.gemini/antigravity-cli
cat << 'EOF' | tee ~/.gemini/antigravity-cli/settings.json ~/.gemini/jetski/cli/settings.json ~/.gemini/antigravity/settings.json ~/.gemini/settings.json
{
"toolPermission": "always-proceed",
"permissions": {
"allow": [
"read_file(*)",
"write_file(*)",
"mcp(*)"
]
}
}
EOF
Be aware that always-proceed trades safety for convenience, so keep the allowlist narrow and use it in a workspace you trust.
Now ask it to handle the failing script:
agy -p "examine spark_job.py, fix the DataFrame method typo, and save the file"
With local access the agent opens spark_job.py, finds withColunm, changes it to withColumn and saves the file in place. No copying between windows.
Level two: infrastructure context through the Spark MCP server
Syntax errors are the easy case. Pipelines at scale more often fail in the infrastructure layer: missing datasets, permission mismatches, runtime environment problems. That's where MCP comes in. It's an open standard that lets AI agents connect to external APIs and data sources, and Google offers a remote Spark MCP server exposing the Managed Service for Apache Spark API as callable tools.
- Endpoint:
https://dataproc-$REGION.googleapis.com/mcp - Transport: HTTP
- Tools include
list_batches,get_batch,list_jobsandget_job, among others
Register the server in the global config file ~/.gemini/config/mcp_config.json:
mkdir -p ~/.gemini/config
cat << EOF | tee ~/.gemini/config/mcp_config.json
{
"mcpServers": {
"spark": {
"serverUrl": "https://dataproc-${REGION}.googleapis.com/mcp"
}
}
}
EOF
With that in place the agent can see your infrastructure. After a batch fails on Google Cloud, run:
agy -p "inspect my latest failed Spark batch via MCP, identify the root cause, and fix spark_job.py so it succeeds"
Here is the sequence that follows:
- The CLI asks the remote MCP server which tools exist and finds
list_batchesandget_batch. - It calls
list_batchesto locate your most recent run. - It calls
get_batchto read the failure state and pull the JVM error from Cloud Logging, such as aFileNotFoundExceptionfor ags://input path that doesn't exist. - It opens the local
spark_job.py, finds the line reading that bad path and removes the reference. In this example it might print the output rather than save it. - It makes sure resources are released cleanly with
spark.stop()and writes the corrected file.
One prompt, and the agent diagnosed a cloud failure and updated local source.
Level three: playbooks as Agent Skills
Once a troubleshooting routine works, package it as an Agent Skill. A skill is a folder of instructions, scripts and context files, anchored by a SKILL.md, that teaches the agent a recurring operational task.
Create .agents/skills/spark-troubleshooter/ and write the skill:
mkdir -p .agents/skills/spark-troubleshooter
cat << 'EOF' > .agents/skills/spark-troubleshooter/SKILL.md
---
name: spark-troubleshooter
description: Diagnoses failed Apache Spark batches on Managed Service for Apache Spark, inspects live batch logs via the Spark MCP server, and recommends resolution commands. Use when troubleshooting Spark job failures.
---
# Spark Troubleshooter Skill
This skill diagnoses failed Apache Spark batches on Managed Service for Apache Spark and offers rapid solutions.
## Instructions
1. Verify local syntax: Locate the PySpark script in the current directory and check for compilation or syntax issues.
2. Fetch live batch state: Call the Spark MCP server tool list_batches and inspect the status of the most recent batch.
3. Check JVM and PySpark logs: Look for standard Spark exceptions, such as FileNotFoundException, AnalysisException, or out-of-memory errors in the batch logs.
4. Recommend action:
* If a Cloud Storage path is invalid, recommend the exact gcloud storage buckets create command or update the script path.
* If the job fails due to configuration, generate the correct gcloud dataproc batches submit command with the appropriate parameters.
* If a syntax error is detected, fix the code in-place.
* For other errors, recommend a fix.
EOF
Because it lives in the project's .agents/skills/ directory, the CLI finds it on its own. Then:
agy -p "Diagnose why my last Spark batch failed and recommend a fix"
The agent follows your playbook steps and returns a focused plan:
[Spark Troubleshooter] Running diagnostic playbook...
- Local Syntax: OK (spark_job.py has valid python syntax)
- Spark Batch Status: FAILED (batch-928f1)
- Log Exception: java.io.FileNotFoundException for gs://my-missing-bucket/input.csv
Recommendation:
The bucket gs://my-missing-bucket does not exist. Run this command to create it:
gcloud storage buckets create gs://my-missing-bucket --location=$REGION
Once created, submit the batch again with:
gcloud dataproc batches submit pyspark spark_job.py \
--region=$REGION \
--deps-bucket=gs://$BUCKET_NAME
Commit the .agents/skills/ folder to Git and everyone who pulls the repository gets the same automated troubleshooting.
Console option: Gemini in Cloud Logging
If you'd rather stay in the Google Cloud console, there are built-in AI features in Cloud Logging. When a serverless Spark batch fails:
- Open the batches page of the Managed Service for Apache Spark console.
- Choose the failed batch and click View Logs.
- Find the error line in Logs Explorer.
- Click Investigate log, or the Gemini sparkle icon.
Gemini reads the log payload in place, highlights the PySpark exception and suggests fixes, without you copying sensitive traces anywhere.
The progression in short
AI help in data engineering scales with context:
- Zero context, as in a web chat, gives general advice and can't touch code or verify infrastructure.
- File context from the Antigravity CLI reads local code and applies syntax fixes in place.
- Infrastructure context from Spark MCP queries live job state, reads Cloud Logging traces and ties runtime failures back to source.
- Codified context in Agent Skills turns a multi-step playbook into a reusable command for a whole team.
To try it hands-on, Google publishes a codelab on platform engineering for Managed Service for Apache Spark with Gemini and MCP. A similar lab exists for Google Kubernetes Engine covering comparable use cases.