Dual-Engine RAG Support Assistant on Google Cloud: Public Docs, Private Runbooks

Anyone working a support queue against an SLA knows the cost of switching tabs: one for Google Cloud's public documentation, another for the customer's own runbook, and a mental merge of the two. Asking a generic AI assistant to shortcut that process often returns an answer that is right on paper and wrong for the customer in front of you.

The fix described here is a dual-engine retrieval-augmented setup. A ticket is sent in parallel to two sources, public Google Cloud docs and a private store of customer runbooks, and a Gemini model reconciles what comes back into a resolution guide that respects the customer's rules. Picture finding a well-rated recipe online and having a nutritionist rework it for your diet before you cook.

The build uses four pieces:

• Two retrieval paths that run side by side, one for generic technical advice and one for strict internal policy.
• Vertex AI Search as the private store, indexing business rules and constraints straight from runbook files.
• The Developer Knowledge API as the public source, returning grounded recommendations and troubleshooting steps from official documentation.
• Gemini as the reasoning layer, screening public solutions against private rules so the output is a guide a support engineer can safely follow.

Why generic advice goes wrong

Take a ticket like this one:

Acme reports roughly five seconds of cold-start delay on their Cloud Run frontend. What should we do?

An assistant grounded only in public docs will almost certainly say to set min-instances to 1. That is the textbook remedy for cold starts, and it is also a problem for this client, for two reasons:

• A budget cap. Acme limits the frontend service to a single instance at most, so an always-on instance doesn't fit.
• A process rule. Acme's runbook forbids changing min-instances without written sign-off from its lead architect, Sarah Connor.

What's needed is a system that fetches the public guidance on Cloud Run startup latency, checks each suggestion against Acme's private constraints, and only then tells the engineer about any conflict and proposes an alternative that complies.

Prerequisites

Have these ready before starting:

1. A Google Cloud project with billing turned on.
2. The gcloud CLI, installed and authenticated on your machine.
3. Python 3.10 or newer.

Cost and quotas

The tutorial is meant to stay inside free tiers and developer quotas for each service:

• Cloud Storage. One small runbook file sits well within the Always Free storage allowance. Charges arise only if you go past it or generate egress.
• Vertex AI Search (Gemini Agent Search). New accounts get a free trial tier. Once trial credits run out, queries are billed pay-as-you-go per 1,000 queries, so check the current pricing page.
• Developer Knowledge API. Free to enable and use within the standard developer limits. Request allocations can be watched in Google Cloud's quota pages.
• Gemini 3.5 Flash. Free within the usual developer limits in Google AI Studio. Paid billing starts only if you upgrade for higher concurrency and rate limits.

Preparing the Google Cloud environment

Before the assistant itself, the underlying infrastructure needs setting up. One bash script handles three jobs:

• Enabling APIs. It turns on Cloud Storage, Vertex AI Agent Builder (discoveryengine), Developer Knowledge and the Generative Language API, so the orchestrator can reach every endpoint it needs.
• Creating the Discovery Engine service identity. It pre-creates the Google-managed service agent (service-PROJECT_NUMBER@gcp-sa-discoveryengine.iam.gserviceaccount.com), so the background indexer can be given IAM permissions right away, with no first-time console click required.
• Setting up storage and access. It creates the Cloud Storage bucket that will hold the customer runbook and grants roles/storage.objectViewer to two principals: the background search agent, which needs it for indexing, and your own account, so you can inspect the bucket in the console.

Paste the script into a terminal:

# 1. Initialize variables from active gcloud config
export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
export BUCKET_NAME="${PROJECT_ID}-internal-kb"
export USER_EMAIL=$(gcloud config get-value account)

# 2. Enable all required GCP APIs
gcloud services enable \
  storage-api.googleapis.com \
  discoveryengine.googleapis.com \
  developerknowledge.googleapis.com \
  generativelanguage.googleapis.com \
  --project=${PROJECT_ID}

# 3. Provision the Discovery Engine Service Identity programmatically
# (This creates the background service account without requiring a manual Console click)
gcloud beta services identity create \
  --service=discoveryengine.googleapis.com \
  --project=${PROJECT_ID}

# 4. Create the GCS bucket
gcloud storage buckets create gs://${BUCKET_NAME} --project=${PROJECT_ID} --location=us-central1

# 5. Grant Storage Object Viewer to the Agent Search Service Agent (for background indexing)
gcloud storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
  --member="serviceAccount:@gcp-sa-discoveryengine.iam.gserviceaccount.com">service-${PROJECT_NUMBER}@gcp-sa-discoveryengine.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

# 6. Grant Storage Object Viewer to your active Console user (for frontend validation)
gcloud storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
  --member="user:${USER_EMAIL}" \

  --role="roles/storage.objectViewer"

echo "Setup complete! All APIs enabled and permissions configured successfully."

The values you may want to adjust:

• PROJECT_ID is your project ID. When the project is already set in gcloud, gcloud config get-value project returns it automatically.
• PROJECT_NUMBER is the project's unique number.
• BUCKET_NAME names the bucket for runbooks and defaults to ${PROJECT_ID}-internal-kb.
• USER_EMAIL is the address of the active Google Cloud account that gets viewer access in the console.

Building the private knowledge engine

The private side is a small search engine holding a sample customer runbook.

Writing and uploading the runbook

Create the runbook file locally, then copy it to the bucket:

# Create the client runbook file with .txt extension
cat << 'EOF' > client-acme-runbook.txt
# Client Acme Corporation - Cloud Run Deployment Runbook
* **Client ID:** ACME-404
* **SLA Tier:** Platinum (4-hour resolution target)
* **Infrastructure Specifications:**
  * Frontend Service Name: `acme-frontend-prod`
  * Regional Deployment: `us-central1`
  * Max Instances: `1` (Hard cap imposed due to strict budget constraints)
  * CPU Allocation: `CPU allocated only during request processing`
  * Backend Database: Cloud SQL PostgreSQL (`acme-db-prod`)
  *   Networking: Serverless VPC (Virtual Private Cloud) Access connector (`acme-vpc-connector`)
* **Deployment Constraints:**
  * Deployed strictly via GitHub Actions workflow (`.github/workflows/deploy.yml`). Manual console updates are forbidden.
  * Direct access to the production DB requires using the Bastion Host `acme-bastion-prod`.
* **Escalation Path:**
  * Primary Contact: Sarah Connor (Lead Architect, sarah@acme-corp.com)
  * Secondary Contact: PagerDuty escalation policy `acme-prod-critical`
* **Common Issues & Custom Rules:**
  * **High Cold Starts:** High cold-start latency due to CPU throttling during scale-up. Do not modify the `min-instances` count without approval from Sarah Connor, as it breaches billing thresholds.
  * **Database Connection Timeouts:** If timeouts occur, verify that the Serverless VPC Access connector is in a `READY` state before attempting database restarts.
EOF

# Upload the file to your bucket
gcloud storage cp client-acme-runbook.txt gs://${BUCKET_NAME}/

The upload command uses the bucket name from the setup step, ${PROJECT_ID}-internal-kb unless you changed it. The runbook is deliberately packed with the facts that matter: Acme's hard cap of one instance, its approval rule for min-instances, its database and VPC connector names, its deployment-only-through-GitHub-Actions policy, and the escalation contacts.

Creating an Agent Search data store

The data store is what makes the runbook searchable. In the Google Cloud console:

1. Open Agent Builder (also labelled Vertex AI Search).
2. From the left navigation, choose Data Stores, then Create data store.
3. As the source, pick Cloud Storage.
4. For the folder path, enter gs://PROJECT_ID-internal-kb/* with your real project ID in place of PROJECT_ID.
5. Pick Unstructured documents (PDF, HTML, TXT and similar) as the type of data and continue.
6. Name the data store acme-internal-kb.
7. Click Create.
8. On the Data Stores list, copy and keep the Data Store ID.

Creating a search app for serving

Queries from code go through a Search App (an Engine), which acts as the serving layer on top of the data store:

1. In Agent Builder, open Apps.
2. Click Create app.
3. Under Search on the creation page, click Create.
4. Call the app acme-search-app.
5. For the external organisation name, enter Acme Corporation.
6. Leave the location at the default, global, and continue.
7. On the Data Stores step, tick acme-internal-kb.
8. Click Create.
9. When it finishes, open the app from the Apps list and copy its Engine ID (also called App ID) from the details page. It looks something like acme-search-app_1234567890.

Connecting the public side: the Developer Knowledge API

This API gives grounded access to official Google Cloud documentation and developer guides. The steps below create an API key, restrict it, and confirm access with one curl request.

Creating an API key

1. In the console, go to Credentials.
2. Choose Create credentials, then API key.
3. Preferably restrict the key to the Developer Knowledge API alone.
4. Store the key in an environment variable:

export DEVELOPERKNOWLEDGE_API_KEY="DEVELOPERKNOWLEDGE_API_KEY

Put your own generated key in place of the placeholder.

Testing with curl

A quick query confirms the key works and that you can reach Google's developer docs:

curl -X POST "https://developerknowledge.googleapis.com/v1:answerQuery?key=$DEVELOPERKNOWLEDGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "How do I reduce Cloud Run cold starts?"}'

A structured JSON reply containing an answer drawn from official documentation means it's working.

The orchestration script

Next comes a Python script that asks both engines the same question and hands the results to Gemini for the final answer.

Installing libraries

pip install google-cloud-discoveryengine google-generativeai requests

Creating the orchestrator

The following command writes app.py. The script sends the incoming ticket text to both search paths, slots the two results into a prompt template, and passes the filled prompt to Gemini:

cat << 'EOF' > app.py
import os
import sys
import requests
import google.generativeai as genai
from google.cloud import discoveryengine_v1beta as discoveryengine
# Configuration - Replace with your details
PROJECT_ID = os.environ.get("PROJECT_ID")
ENGINE_ID = os.environ.get("ENGINE_ID")
LOCATION = "global"
MODEL_NAME = os.environ.get("GEMINI_MODEL", "gemini-3.5-flash")
# Initialize Gemini Client (Uses GEMINI_API_KEY env variable)
genai.configure(api_key=os.environ.get("GEMINI_API_KEY"))
model = genai.GenerativeModel(MODEL_NAME)

# 1. Helper to query the Private KB (Agent Search)
def query_private_kb(query_text: str):
client = discoveryengine.SearchServiceClient()
serving_config = f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/default_collection/engines/{ENGINE_ID}/servingConfigs/default_search"
request = discoveryengine.SearchRequest(
serving_config=serving_config,
query=query_text,
page_size=3,
)
response = client.search(request)
results = []
for result in response.results:
doc_data = result.document.derived_struct_data
if "snippets" in doc_data:
for s in doc_data["snippets"]:
results.append(s.get("snippet", ""))
return "\n".join(results)

# 2. Helper to query the Public KB (Developer Knowledge API)
def query_public_gcp_kb(query_text: str):
api_key = os.environ.get("DEVELOPERKNOWLEDGE_API_KEY")
url = f"https://developerknowledge.googleapis.com/v1:answerQuery?key={api_key}"
headers = {"Content-Type": "application/json"}
payload = {"query": query_text}
try:
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
data = response.json()
return data.get("answer", {}).get("answerText", "No answer found.")
except Exception as e:
return f"Error querying Developer Knowledge API: {e}"

# 3. Main Orchestration
def run_support_assistant(ticket_query: str):
print(f"\n[Ticket Query]: {ticket_query}")
# Dynamically search private client specs using the ticket query
print("Searching private database…")
private_context = query_private_kb(ticket_query)
# Dynamically search public GCP advice using the ticket query
print("Searching public Google Cloud docs…")
public_context = query_public_gcp_kb(ticket_query)
# Format system prompt template
prompt = f"""
You are a Senior Cloud Support Assistant. Your task is to resolve the support ticket.
CRITICAL INSTRUCTIONS:
1. You MUST read the [Private Runbook Context] for specific client configurations, limits, and rules.
2. You MUST cross-reference all generic GCP recommendations against the client's private constraints. If a generic recommendation (e.g., increasing min-instances) violates a client limit or rule (e.g., budget caps or requiring specific manager approvals), you MUST call this out explicitly and provide the compliant fallback option.
3. Refer to specific client contacts (e.g., Sarah Connor), services, databases, and network connectors mentioned in the runbook when explaining the steps.
[Private Runbook Context]:
{private_context}
[Official GCP Guidance]:
{public_context}
[Support Ticket Query]:
{ticket_query}
Provide a precise troubleshooting guide for the support engineer.
"""
response = model.generate_content(prompt)
print("\n[AI Assistant Resolution Guide]:")
print(response.text)
if __name__ == "__main__":
if not all([PROJECT_ID, ENGINE_ID, os.environ.get("DEVELOPERKNOWLEDGE_API_KEY"), os.environ.get("GEMINI_API_KEY")]):
print("Please set your environment variables: PROJECT_ID, ENGINE_ID, DEVELOPERKNOWLEDGE_API_KEY, and GEMINI_API_KEY.")
else:
# Allow passing the ticket query as a command line argument
default_query = "Client Acme is complaining about 5-second cold start latency on their frontend service. How can we fix it?"
query = sys.argv[1] if len(sys.argv) > 1 else default_query
run_support_assistant(ticket_query=query)
EOF

The prompt does the real work. It tells the model to read the private runbook for client-specific settings and rules, and to compare every generic recommendation with those constraints. If a public recommendation, such as raising min-instances, breaks a client limit or approval rule, the model must say so openly and offer a compliant fallback. It must also name the client's actual contacts, services, databases and network connectors when it explains the steps.

Running the test

Set the environment variables and launch the script:

export PROJECT_ID="PROJECT_ID"
export ENGINE_ID="ENGINE_ID"
export DEVELOPERKNOWLEDGE_API_KEY="DEVELOPERKNOWLEDGE_API_KEY"
export GEMINI_API_KEY="GEMINI_API_KEY"
# Run with the default query
python3 app.py
# Or pass a custom query as a command line argument
python3 app.py "Client Acme is complaining about database connection timeouts on their Cloud SQL backend."

Substitute real values:

• PROJECT_ID is your project ID.
• ENGINE_ID is the Engine/App ID of the Vertex AI Search app.
• DEVELOPERKNOWLEDGE_API_KEY is the key you generated for the Developer Knowledge API.
• GEMINI_API_KEY is a key from Google AI Studio. Google's documentation explains how to get a Gemini API key.

What the output looks like

Expect something along these lines:

[AI Assistant Resolution Guide]:
### **Acme Support Ticket: Resolution & Troubleshooting Guide**
**Ticket ID:** #ACM-98431
**Service:** `acme-frontend` (Cloud Run)
**Issue:** 5-second cold start latency
**Assigned To:** Cloud Support Engineering Team
 - -
### **CRITICAL COMPLIANCE WARNING (Read Before Acting)**
According to Acme's operational rules:
* **The Conflict:** Standard GCP documentation suggests setting `min_instances` (e.g., to 1 or more) to eliminate cold starts. However, **Acme operates under a strict budget cap policy that prohibits idle/always-on instances** for the frontend service without explicit business justification.
* **The Rule:** You **MUST NOT** increase `min_instances` on `acme-frontend` without prior written approval from **Sarah Connor** (Acme's Technical Lead & Budget Approver).
* **Compliant Fallback:** If Sarah Connor does not approve the budget increase for `min_instances`, you must resolve the latency strictly through **code-level optimizations, CPU adjustments, startup probes, and connection tuning** as detailed below.
 - -
### **Step-by-Step Troubleshooting & Resolution Guide**
#### **Phase 1: Triage and Governance Check**
1. **Contact Sarah Connor:** Reach out to Sarah Connor to request a budget exception for setting `min_instances = 1`.
2. **Fallback implementation:** If denied, proceed strictly with the budget-compliant fallback steps below to optimize the cold start duration within the 0-instance scaling limit.
#### **Phase 2: Network & Connection Optimization**
1. **Inspect VPC Connector Latency:** `acme-frontend` routes traffic to the database via the **`acme-vpc-connector`** (Serverless VPC Access). Ensure that the connector throughput is not bottlenecked.
2. **Database Connection Lazy-Loading:** The frontend connects to **`acme-production-db`** (Cloud SQL). Do not open the connection pool until the first actual handler request arrives to minimize container boot time.
…

What the result shows

The answer combines the two sources, and each contributes something different:

1. Public documentation, via the Developer Knowledge API, supplies the technical fixes: turning on cpu-boost, setting concurrency to 80, or adding a /healthz startup probe.
2. The private knowledge base, via Agent Search, adds business logic, security constraints and operational detail. It points out that the standard advice (min_instances = 1) breaks Acme's budget rules. It names Sarah Connor as the person who must approve any exception. It refers to Acme's actual database and VPC connector.

Without both engines, a generic model would just recommend raising min_instances. A junior support engineer might act on it right away, and in an enterprise that could break budget limits, trip billing alerts and skip architectural approval. Grounding the model in both public and private context yields fixes that are technically sound and also compliant with organisational policy.

Hardening it for production

A few changes would be sensible before this serves real tickets:

• Keep secrets in a secrets manager. Put API keys in Secret Manager and load them at runtime, not in plain-text environment variables.
• Use least-privilege IAM. Run the workload under a dedicated service account holding only roles/discoveryengine.viewer and roles/storage.objectViewer, and limit API keys to specific APIs and trusted network addresses.
• Automate ingestion. Link sources such as Jira, Confluence, SharePoint or Google Drive to the data store through Vertex AI Search connectors, and drop the manual uploads.
• Filter by metadata. Pass client IDs and system tags in the SearchRequest filter so results are narrowed to one customer's environment.
• Cache common lookups. Keep frequent documentation answers in Memorystore for Redis to cut latency and quota use.

Cleaning up

To avoid ongoing charges, remove what the tutorial created.

Search app and data store

A data store can't be deleted while an app is attached, so delete the app first. In the console:

1. Go to Agent Builder, then Apps.
2. Find acme-search-app, open More actions (three dots) and choose Delete.
3. Type the app's name in the confirmation dialog and confirm.
4. Open Data Stores in the navigation menu.
5. Find acme-internal-kb, open More actions, and choose Delete.
6. Confirm.

The storage bucket

Removing the bucket's contents is one command:

export BUCKET_NAME="${PROJECT_ID}-internal-kb"
gcloud storage rm - recursive gs://${BUCKET_NAME}/

API keys

1. Go to APIs & Services, then Credentials.
2. Under API Keys, find the key made for the Developer Knowledge API.
3. Click the trash icon next to it and confirm.

Local files

Delete the scripts and the sample runbook from your machine:

rm -f app.py app_test.py client-acme-runbook.txt