Git to GCS for Spark: An Agent-Ready Scala SBT Build and Upload Pipeline

Anyone who has taken a Spark job from a laptop to production knows the routine. You juggle Git commands, an SBT build, an upload to cloud storage and a handful of infrastructure APIs, and every step is manual. There's a different way to do it: instead of asking an AI for code snippets (prompt engineering), you hand an AI agent the whole lifecycle and let it run the steps itself. With Google's Agent Development Kit (ADK), Gemini 2.5 Flash and Dataproc Serverless, an LLM can act as a DevOps and data engineering partner.

This first stage covers the foundation of that setup: a deterministic Python pipeline that pulls a Scala/Spark repository from Git, builds a fat JAR with SBT under Java 11, and pushes it to Google Cloud Storage. The pipeline is written so an agent can call it as a tool. The agent layer itself, with ADK function tools, a confirmation-gated prompt, Dataproc Serverless interactive sessions and a Streamlit chat front end, builds on top of it afterwards.

The practical result is that a 20-minute, error-prone build and deployment routine becomes a reliable command that an agent can fire from a plain English request.

The pain of the inner loop in Spark work

Web developers get near-instant feedback from hot module reloading. Big data engineers rarely get anything like that. The edit, build, deploy, test cycle, often called the inner loop, tends to stall at several points:

1. Mismatched environments. The Java version on a laptop clashes with the one on the Spark cluster. Java 17 locally against Java 11 bytecode on Dataproc is the classic example.
2. Expensive packaging. Producing a shaded fat JAR with sbt clean compile assembly can chew through CPU and memory for several minutes.
3. Manual trips through the cloud. People switch to the Google Cloud Console or type long CLI commands that are easy to get wrong.
4. Slow notebook wiring. Attaching a fresh JAR to an interactive notebook environment means digging through nested Dataproc properties.

The idea here: what if the entire chain were code, tidily wrapped, and handed to an agent that accepts instructions in plain language? The sections below describe an automated Git-to-GCS Scala/SBT build pipeline designed as an executable tool for LLM agents.

Design rules for agent-safe tooling

For an AI agent to run a build system safely, the tool has to follow three principles:

1. Deterministic execution. The script validates the environment, for instance by confirming Java 11, before it begins any costly compile step.
2. Clean isolation. Builds happen in temporary directories that are removed afterwards, so disk space doesn't pile up.
3. Standard input and output. Each stage prints machine-parseable logs and returns an explicit status.

Build and upload are kept separate so each can be called, tested and retried independently.

Implementation, step by step

Pinning the build to Java 11

Spark runtimes on Dataproc Serverless need binary compatibility with Java 11. Developer machines often default to Java 17 or 21, and that mismatch produces bytecode errors such as UnsupportedClassVersionError: 61.0 vs 55.0.

The ArtifactBuilder class in artifact_builder.py avoids this by inspecting the installed JDKs at runtime and binding JAVA_HOME to the build subprocess alone, so the system default stays untouched:

import os
import subprocess
import logging

class ArtifactBuilder:
    """Builds Scala JAR artifacts from Git repositories using SBT."""

    def __init__(self, repo_url: str, branch: str = "main"):
        self.repo_url = repo_url
        self.branch = branch

    def clone_repo(self, dest_dir: str):
        """Clone the given Git repository to the destination directory."""
        logging.info(f"Cloning {self.repo_url} (branch={self.branch}) into {dest_dir}")
        subprocess.run(["git", "clone", "-b", self.branch, self.repo_url, dest_dir], check=True)

    def find_build_dir(self, root_dir: str) -> str:
        """Recursively search for SBT build file."""
        for dirpath, _, filenames in os.walk(root_dir):
            if "build.sbt" in filenames:
                logging.info(f"Detected SBT build file in {dirpath}")
                return dirpath
        raise FileNotFoundError(f"No build.sbt file found in {root_dir}")

    def check_java_installed(self):
        """Ensure Java 11 is available and active in environment."""
        env = os.environ.copy()
        env["JAVA_HOME"] = "/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home"
        env["PATH"] = f"/opt/homebrew/opt/openjdk@11/bin:{env.get('PATH', '')}"

        result = subprocess.run(
            ["java", "-version"],
            check=True,
            env=env,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True
        )
        output = result.stdout + result.stderr
        if "version" in output:
            version_str = output.split("version")[1].split()[0].strip('"')
            major_version = int(version_str.split(".")[0])
            if major_version != 11:
                raise EnvironmentError(f"Java 11 required, but detected Java {major_version}")
        logging.info("Java 11 verified successfully.")

    def build_artifact(self, repo_dir: str) -> str:
        """Compile and assemble fat JAR."""
        build_dir = self.find_build_dir(repo_dir)
        self.check_java_installed()

        env = os.environ.copy()
        env["JAVA_HOME"] = "/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home"
        env["PATH"] = f"/opt/homebrew/opt/openjdk@11/bin:{env.get('PATH', '')}"

        logging.info("Running: sbt clean compile assembly...")
        subprocess.run(["sbt", "clean", "compile", "assembly"], cwd=build_dir, env=env, check=True)

        target_dir = os.path.join(build_dir, "target")
        return self._find_file(target_dir, ".jar")

    def _find_file(self, directory: str, extension: str) -> str:
        for root, _, files in os.walk(directory):
            for f in files:
                if f.endswith(extension):
                    return os.path.join(root, f)
        raise FileNotFoundError(f"No {extension} file found in {directory}")

Publishing to Cloud Storage reliably

Shell-level gsutil or gcloud calls can fail silently in odd authentication situations. The uploader.py module therefore uses the official Google Cloud Storage Python client library instead, which raises real exceptions the pipeline can catch and report:

from google.cloud import storage
import logging
import os

class ArtifactUploader:
    """Handles secure upload of artifacts to Google Cloud Storage."""

    def __init__(self):
        self.client = storage.Client()

    def upload_to_gcs(self, bucket_name: str, artifact_path: str, dest_path: str):
        if not os.path.exists(artifact_path):
            raise FileNotFoundError(f"Artifact not found: {artifact_path}")

        logging.info(f"Uploading {artifact_path} → gs://{bucket_name}/{dest_path}")
        bucket = self.client.bucket(bucket_name)
        blob = bucket.blob(dest_path)
        blob.upload_from_filename(artifact_path)
        logging.info("Upload to GCS completed successfully.")

Wrapping both in a CLI

Finally, run_build_and_upload.py joins the builder and uploader into one ArtifactPipeline with a deterministic command-line interface.

Nobody has to memorize or type these commands. The developer states the intent in the chat front end, for example "Build my repo on main and upload it to my GCS bucket." The agent's backend extracts the parameters from that sentence and fires the command underneath. This is what it executes in the background:

# Executed autonomously in the background by the Agent's backend tool:
python3 artifact_pipeline/run_build_and_upload.py \
  --repo "https://github.com/<username>/demo-pipeline.git" \
  --branch main \
  --bucket "demo-spark-sandbox-bucket" \
  --dest "spark-jobs/spark-serverless-job_test.jar" \
  --cleanup

A backend terminal log from a live run shows the Google ADK agent working with Gemini 2.5 Flash on Vertex AI. It reads the user's chat message and calls artifact_pipeline/run_build_and_upload.py with the right parameters automatically.

Watching the pipeline run

Cloud Storage check. Once the uploader finishes, the fat JAR sits in the target bucket, ready to attach to Dataproc. In the console, the compiled Spark JAR appears under a path like gs://<bucket-name>/spark-jobs/spark-serverless-job_test.jar, at 186.8 MB.

Driving it from a chat UI. With the conversational Streamlit interface attached, the build and upload run entirely from natural language. The developer supplies the Git repository, the branch, the bucket and the destination path, and the agent plans and executes the build tools. When the artifact build and GCS upload complete, the agent follows its operational state machine and asks for confirmation before launching a serverless notebook.

Failing gracefully

A pipeline meant for production has to fail in a readable way. When the build or the environment breaks, the agent intercepts the underlying stderr and gives actionable guidance instead of crashing.

Git authentication problems. Point the pipeline at a private repository, or hit an authentication failure, and it traps Git's stderr (for example Invalid username or token / Password authentication is not supported). It then explains how to configure SSH or a token.

Runtime and environment errors. If something unexpected goes wrong in the Python environment, such as a missing dependency or a ModuleNotFoundError, the agent isolates the error, reports the exact traceback and suggests ways to recover. The chat session stays alive, with interactive retry prompts, rather than dying with the failure.

Lessons to carry forward

1. Wrap domain complexity. When a process has four distinct steps, fold them into one idempotent command with explicit inputs and outputs.
2. Design for calls from code. Keep interactive prompts out of build scripts, and make every parameter available as an argument or API payload.
3. Give agents a firm base. A deterministic foundation lets an LLM trigger complex builds reliably instead of hallucinating shell commands.
4. Plan for failure. Clean error interception lets the agent diagnose authentication problems, missing files and broken dependencies, then explain them to the user.

Reliable agentic systems grow from the bottom up. Swapping manual compile steps for an isolated, Java 11-compliant Python engine turns a slow, fragile bottleneck into a dependable building block. Once the Git-to-GCS pipeline is packaged as a self-contained, idempotent tool, it can serve as the hands and feet of an AI agent.