Playing Volleyball With AI Agents to Predict Restaurant Turnover

Picture a hackathon in which the model itself is only half the job. The other half is keeping a handful of AI agents busy on competing ideas without letting any of them make a call that belongs to a person. This write-up describes one such effort, from a restaurant turnover prediction hackathon run by Great Learning in partnership with UT Austin and its McCombs business school. It covers a lean ML workflow, strict validation and an ensemble that, by the project's own account, did well in the competition.

The orchestration layer was a framework called DontPanic. Its workflows follow the logic of a volleyball rally. An agent receives a task, sets up the next move, passes the work across specialised roles and finally hands the result over, yet the ball always lands back with a human at the decision line. Agents speed up execution. They don't own the judgment.

1. The task

Competitors had to predict the annual turnover of restaurants across India from 3,493 labelled training records and 500 unlabelled test records. There were roughly 32 raw predictors covering location, cuisine, restaurant format, amenities, social-media popularity and customer-experience ratings. Here "turnover" means yearly sales, not staff attrition. The metric was RMSE, which squares errors before averaging, so a few enormous misses can swamp the score.

The aim wasn't to fit the fanciest model. It was a forecasting system that could be defended: audit the data, understand the economics of the target, engineer signals that make business sense, validate candidates with stable cross-validation, discard improvements that failed to replicate, and finally keep the model's true generalisation separate from competition-specific tuning.

2. The tooling

Everything lived in VS Code as a modular Python package under src/, in place of a pile of unrelated notebook experiments. That choice paid off because cleaning, feature construction, validation and inference all ran through the same reusable code. A feature therefore couldn't quietly behave one way in training and another when the submission was produced.

3. Agentic orchestration

DontPanic was the harness wrapped around the project. Full autonomy was never the goal. The point was to let several agents move fast while spelling out the decision structure: who explores, who builds, who challenges, which artifact is authoritative and when work must return to a person for a go or no-go.

That matters because ML experimentation throws up one plausible next step after another. A free-running agent will keep optimising whatever signal it saw most recently, and a human can lose track of whether the hypothesis, the validation protocol or the promotion criterion has shifted underneath. Think of DontPanic as the rulebook the experiment ran under: agents were free to advance play, never to rewrite the rules without anyone noticing.

Each agent runtime was described by a manifest. One example, for a Claude CLI runtime, spells out the commands and files it requires, how readiness is verified, which parts the framework owns, what the operator must do and which actions are forbidden:

{
  "schema_version": "1.0.0",
  "id": "agent-claude-cli",
  "title": "Claude CLI agent runtime",
  "kind": "agent_cli",
  "category": "agent-cli",
  "summary": "External agent CLI used by DontPanic as an implementer, auditor, or goal-auditor runtime.",
  "setup_required": true,
  "setup_doc": "docs/GETTING_STARTED.md#setup-tracks--pick-yours",
  "default_in_profiles": ["core"],
  "use_cases": ["U1", "U3", "U4", "U5", "U6", "U7"],
  "requires": {
    "commands": ["claude"],
    "env": [],
    "files": ["~/.dontpanic/config.json"],
    "services": ["Claude account / Claude Code or CLI runtime"],
    "auth": ["Claude CLI authenticated by operator"],
    "config": [
      "roles.implementer",
      "roles.auditor",
      "roles.goal_auditor"
    ]
  },
  "verify": {
    "doctor_profiles": ["core"],
    "probes": ["cli:claude", "agent_role_config"],
    "readiness": "Ready when the claude command is present and the selected DontPanic role points at claude."
  },
  "owner_boundary": {
    "dontpanic_core": [
      "agent registry entry",
      "dispatch envelope",
      "audit/signoff capture",
      "quota and breaker handling",
      "role configuration"
    ],
    "adapter": [
      "subprocess executor for claude",
      "stream/output parsing",
      "timeout and envelope normalization"
    ],
    "operator": [
      "Claude CLI installation",
      "account authentication",
      "model/quota management",
      "local CLI updates"
    ]
  },
  "mutation_boundary": {
    "writes_dontpanic_state": true,
    "external_mutation": "agent_subprocess_execution",
    "allowed_path": "DontPanic supervisor invokes claude through the governed executor and captures artifacts through audit/signoff writers.",
    "forbidden": [
      "Bypassing supervisor gates",
      "Untracked agent writes outside declared plan scope",
      "Treating Claude account configuration as DontPanic-owned state"
    ]
  },
  "notes": [
    "Claude is an external runtime, not bundled by DontPanic.",
    "Core profile may also support other agent CLIs; this manifest records the Claude-specific path."
  ],
  "setup_steps": [
    {
      "id": "install_claude_cli",
      "what": "Install the Claude CLI so DontPanic can spawn agent runs through the governed executor.",
      "automatable": true,
      "command_template": "npm install -g @anthropic-ai/claude-cli",
      "verify_probe": "cli:claude",
      "human_required_reason": null
    },
    {
      "id": "authenticate_claude",
      "what": "Authenticate the Claude CLI against the operator's Claude account or API key.",
      "automatable": false,
      "command_template": "claude auth login --token <YOUR_CLAUDE_API_TOKEN>",
      "verify_probe": "agent_role_config",
      "human_required_reason": "Operator must approve scopes for their Claude account and treat the API token as a secret; DontPanic does not provision Claude credentials on the operator's behalf."
    }
  ]
}

3.1 The rally in five steps

1. Serve: before any experiment began, the human framed the business question, constraints, metric, acceptance criteria and the current champion.
2. Receive: a discovery agent inspected the evidence, flagged anomalies and proposed testable hypotheses instead of rewriting the model straight away.
3. Set: once a hypothesis was approved, an implementation agent wrote it up as modular code, deterministic artifacts and an experiment definition that could be rerun.
4. Volley: a reviewer agent probed for leakage, flawed cross-validation, unstable seeds and murky data provenance, and asked whether the apparent gain survived a comparison with the frozen champion.
5. Dispatch to the human: promoting, rejecting, expanding or changing strategy came back to the person. Agents could suggest the shot, but the scoreboard stayed with the human.

The real benefit was controlled parallelism rather than raw coding speed. Different agents could concentrate on different ways of failing, while the human loop protected product intent, business plausibility and accountability. The result felt closer to a small product team than to one chatbot session.

3.2 Lean ML delivery

The agent workflow was also mapped onto a lean product lifecycle, with the aim of making experiments cheap in DEV, making evidence hard to fake in UAT and making production states hard to change by accident. The DEV/UAT/PROD split grew more valuable as the experiment count climbed. A new agent could join without depending on conversational memory, because the authoritative record was kept as files, manifests, out-of-fold (OOF) predictions, a registry of champions and written-down promotion decisions. The pattern worth keeping: let agents supply speed, and let artifacts and gates supply truth.

4. Auditing the data before believing it

Before modelling, the train and test files were fingerprinted with SHA-256 and checked column by column. That wasn't bureaucracy. Several quirks could have altered the feature space at inference time and hurt predictions without raising any visible error.

• The training set had 3,493 rows and 34 columns, while the test set had 500 rows and 33 columns, since it had no target.
• The Ambience and Restaurant Tier columns had different data types in train and test, so explicit coercion was needed for the models to receive identical features.
• One column was spelled Endorsed By in train and Endoresed By in test. Without a mapping, the pipeline could have produced a test feature that was entirely missing.
• There were no duplicate rows and no duplicate predictor vectors, so the data was clean at row level.
• Registration Number acted like an identifier, not a business predictor (Pearson about -0.017, Spearman about -0.022). It was left out of the models and kept only to map rows at the end.

5. Exploring the data

5.1 A heavily right-skewed target

Annual turnover had a long right tail. The 90th percentile sat near ₹48M, the 95th near ₹57M, the 99th near ₹100M and the maximum at ₹400M, with raw skewness around 6.65. A log1p transform made the distribution nearly symmetric, and raw, log1p and Yeo-Johnson targets were compared, with a per-square-foot variant tested later. In business terms, most restaurants fall into a fairly narrow revenue band, while a tiny group of very large outlets produces outsized financial error whenever it is missed.

The tail wasn't just visually odd. It dominated the loss. Error analysis showed that a small set of worst-predicted restaurants accounted for a large share of total squared error, so tail diagnostics became a core part of evaluation.

5.2 Missing data needs more than one rule

Missing values weren't treated as a single phenomenon. Some reflected ordinary data-quality gaps. Others encoded business structure, for example an entertainment rating being blank because the venue offers no entertainment. One imputation rule for both would destroy useful information.

Ordinary missingness, such as social-media popularity, overall rating, ambience and tier, was handled with imputation fitted on training data plus explicit missingness indicators. That lets a model distinguish an estimated value from the fact that nothing was observed. Structural missingness, covering live music, comedy gigs, value deals and live sports, became availability flags like has_live_music and has_comedy. A blank rating can mean the service doesn't exist, which is a different business state from an unknown rating.

ORDINARY_MISSINGNESS_FLAGS = {
    "facebook_popularity_quotient": "facebook_missing",
    "instagram_popularity_quotient": "instagram_missing",
    "overall_restaurant_rating": "overall_rating_missing",
    "ambience": "ambience_missing",
    "restaurant_tier": "tier_missing",
}

STRUCTURAL_AVAILABILITY_FLAGS = {
    "live_music_rating": "has_live_music",
    "comedy_gigs_rating": "has_comedy",
    "value_deals_rating": "has_value_deals",
    "live_sports_rating": "has_live_sports",
}


def add_missingness_features(df: pd.DataFrame) -> pd.DataFrame:
    """Add ordinary-null and structural-availability indicators before imputation."""
    required_columns = set(ORDINARY_MISSINGNESS_FLAGS) | set(
        STRUCTURAL_AVAILABILITY_FLAGS
    )
    missing_columns = sorted(required_columns - set(df.columns))
    if missing_columns:
        raise KeyError(f"Missing required columns for missingness features: {missing_columns}")

    df = df.copy()
    for column, flag in ORDINARY_MISSINGNESS_FLAGS.items():
        df[flag] = df[column].isna().astype("int8")
    for column, flag in STRUCTURAL_AVAILABILITY_FLAGS.items():
        df[flag] = df[column].notna().astype("int8")
    return df

5.3 Categorical mess: 297 spellings of cities

The City column held 297 raw spellings for a far smaller set of real markets. Case and whitespace varied, typos and aliases appeared, and some entries were compound strings naming several cities. A reproducible clean-up combined a fuzzy-matching review using rapidfuzz, a curated alias dictionary and explicit parsing of compound entries. Rare-city flags and city-frequency features stayed in, because market size and data support can both influence turnover estimates. The matching step only proposes candidates for human review and never changes values itself:

def generate_city_candidate_matches(
    city: pd.Series,
    minimum_similarity: float = 80.0,
) -> pd.DataFrame:
    """Return nearest fuzzy matches for review; this never changes city values."""
    normalized = normalize_city_values(city, aliases={})
    counts = normalized.value_counts()
    names = [name for name in counts.index if name != "unknown"]
    rows = []
    for raw_city in names:
        candidates = [candidate for candidate in names if candidate != raw_city]
        match = process.extractOne(raw_city, candidates, scorer=fuzz.ratio)
        if match is not None and match[1] >= minimum_similarity:
            rows.append(
                {
                    "Raw City": raw_city,
                    "Candidate": match[0],
                    "Similarity": float(match[1]),
                    "Count": int(counts[raw_city]),
                }
            )
    return pd.DataFrame(
        rows,
        columns=["Raw City", "Candidate", "Similarity", "Count"],
    ).sort_values(["Similarity", "Count", "Raw City"], ascending=[False, False, True])

5.4 Pairs of variables

Restaurant type and city tier both separated turnover in a meaningful way. Format shapes average ticket, throughput and operating model, while city tier is a coarse stand-in for demand density, purchasing power and market scale. The individual ratings were useful but noisy. Food, hygiene, value for money, service and ambience aren't independent levers. They interact with format, age, market, social visibility and positioning, which motivated both non-linear learners and carefully controlled interaction features.

5.5 The wider picture

Pearson and Spearman correlations told different stories. Several predictors had a much stronger monotonic than linear link with turnover, so useful signal existed even where a straight line fit poorly. Pairwise-support checks were added so that extreme correlations resting on very few overlapping observations wouldn't be mistaken for solid structure.

5.6 Time-based features

Opening dates were split into calendar components and age features. Older restaurants tended to earn more, but that was treated as an association rather than a causal claim, since age may proxy for brand recognition, surviving locations, repeat demand or accumulated scale. A fixed reference date kept the feature reproducible.

6. An imbalance problem inside a regression task

Regression has no class imbalance in the classification sense, but this target had an analogue: the upper tail was rare and costly. Under RMSE a few giant residuals can dominate the squared-error total. In the diagnostics, the worst observations accounted for a disproportionate share, so a model could look fine on most restaurants and still score badly after missing a handful of big earners.

Validation therefore used target-stratified 5-fold cross-validation. The continuous target was cut into 10 quantile bins only to assign folds, while models still trained and scored on raw turnover. Final promotion checks were repeated on independent seeds (17, 42 and 101). That lowered the odds of a candidate becoming champion merely because one lucky split put the extreme tail into easier folds.

7. Feature engineering

Features were split into stateless transformations and fitted statistics. Stateless logic ran identically on train and test. Anything fitted, whether medians, frequency maps or encodings, was learned from training data alone, versioned and reused at inference. That kept the pipeline auditable and guarded against subtle train/test drift introduced by preprocessing.

• Missingness and availability flags, as described above.
• Calendar features from the opening date (year, month, quarter, weekday), plus age in months and years and a four-bucket "opening era" category.
• A cuisine count, obtained by parsing the comma-separated cuisine list.
• A city hierarchy from raw city to cleaned city to primary city to state, with flags for compound entries, regions and aliases, along with frequency and rare-city indicators at three thresholds.
• Categorical interactions that capture combinations with different economics from their parts: theme with restaurant type, theme with location, liquor with type, and some entertainment and format pairings.
• Numeric interactions for the regularised linear family: restaurant tier with city tier, social visibility with tier, customer experience with social visibility, and age with popularity. These turned certain non-linear business relationships into a shape Ridge could use.
• Frequency and ordinal encodings for each categorical column, learned on train and reused on test.

The main frozen pipeline used 71 predictors. Extra engineered families were tested through ablation instead of being kept automatically, and a feature earned its place through incremental validation lift, not intuitive appeal.

8. Bringing in outside reference data

A separate research branch joined cleaned city and state keys with public geographic and economic reference data. The question was whether local market capacity, urbanisation, tourism intensity and population scale added information beyond a restaurant's own attributes.

• Per city: population, household counts and size, log population, coordinates, plus yes/no markers for metros, the national capital region, coastal cities, state capitals and cities above a million people.
• State level: population, urban population and share, density, per-capita NSDP, GSDP and NSDP, and domestic and foreign tourist visits, with a derived tourism-per-capita figure.

The outcome was deliberately conservative. Several external features showed signal in isolated tests, but the family didn't improve the final ensemble under the full substitution and promotion protocol, so it never entered the clean champion. It's a useful lesson for business modelling: a variable can be plausible and predictive alone yet add little once stronger correlated signals are present.

9. Models and why

The data mixed continuous ratings, sparse and structural gaps and high-cardinality categoricals, so several model families competed as hypotheses instead of assuming one algorithm would dominate. Ridge and ElasticNet turned out unexpectedly competitive with CatBoost. That wasn't a defeat for boosting. It reflected how much structure had moved into the engineered features and how useful regularisation is when many related predictors coexist. Tree models still mattered, since they captured non-linear interactions that the linear family can't express directly.

10. Tuning

During development each family got its own Optuna study, scored by out-of-fold RMSE from repeated, target-stratified folds. Finding candidates and promoting them relied on internal validation, not the leaderboard, a distinction that matters because the later competition-optimisation phase shouldn't be confused with generalisation.

The strongest clean blend combined CatBoost, Ridge, ElasticNet and a variance-stabilised square-root Ridge variant (WR01) at about 30%, 10%, 6% and 54%. Its repeated out-of-fold RMSE was roughly ₹20.10M. The heavy WR01 weight was telling: the transformed linear model wasn't the best standalone learner, but its different error geometry made it very valuable inside the ensemble.

11. What helped and what didn't

Every experiment was logged with its feature family, model configuration, validation design, OOF predictions and promotion decision. Progress came from rejecting weak ideas as firmly as strong ones were promoted.

• A baseline came first. Mean and median predictors set an RMSE floor of about ₹21.6 to 21.7M, which made it easy to spot models that were merely complicated.
• Regularisation mattered. Ridge and ElasticNet beat plain OLS by a wide margin because the engineered space held correlated signals and interactions.
• CatBoost brought non-linear and categorical strength, but tuning wasn't assumed to help. One promising CatBoost result improved the optimisation split by roughly ₹258K, then failed confirmation on independent seeds and was dropped as selection overfit.
• Feature engineering depended on the model. Target encodings, interactions, category frequencies and tenure features were among the more useful families, while several intuitive aggregates added little once stronger predictors existed.
• External city and state enrichment was a useful negative result: some component tests improved, ensemble substitution got worse, and the branch stayed out of the clean champion.
• Ensembling gave the most dependable gain, since CatBoost and the regularised linear variants erred differently. The target was the lowest validated squared error from complementary models, not maximum complexity.
• Tail classifiers and residual-correction models showed promising diagnostic lift but not enough RMSE improvement to justify promotion. Usefulness for diagnosis and usefulness for forecasting were treated as separate questions.

12. Scores for the different champions

The project produced two kinds of champion: a clean model chosen by repeated cross-validation, and a competition champion tuned to the evaluation environment. A third checkpoint, S08, illustrates what a purposely small tail correction does, prior to the heavier competition-specific stages that followed. These scores answer different questions and aren't interchangeable measures of generalisation.

On the competition side, the LB05 submission cut public RMSE by ₹9,195,461.72 relative to S08, about 76.1%. That is a leaderboard result, not evidence that the deployable model suddenly generalises at ₹2.89M RMSE. Here is the script that fits the frozen components and writes the first blended submission:

from __future__ import annotations

from datetime import datetime, timezone
from pathlib import Path
import time
import json

import joblib
import numpy as np
import pandas as pd

from src import config, data, models, submission
from src.pipeline import FeaturePipeline
from main import _prepare_catboost_matrices, _prepare_model_matrices


S01_ID = "S01"
S01_WEIGHTS = {"catboost": 0.50, "ridge": 0.45, "elasticnet": 0.05}


def run(output_dir: Path | None = None) -> tuple[Path, Path]:
    output_dir = output_dir or config.SUBMISSIONS_DIR
    output_dir.mkdir(parents=True, exist_ok=True)
    manifest_path = config.EXPERIMENTS_DIR / "frozen_v1_manifest.json"
    if not manifest_path.exists():
        raise FileNotFoundError(f"Frozen V1 manifest is required: {manifest_path}")
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    if manifest.get("feature_pipeline") != "processed_v1" or manifest.get("predictor_count") != 71:
        raise ValueError("Frozen V1 manifest does not describe the processed_v1 71-predictor contract")

    train_raw = data.load_train()
    test_raw = data.load_test()
    pipeline = FeaturePipeline().fit(train_raw)
    train = pipeline.transform(train_raw)
    test = pipeline.transform(test_raw)
    target = config.INTERNAL_TARGET

    X_train = train.drop(columns=[target, config.INTERNAL_ID_COL], errors="ignore")
    X_test = test.drop(columns=[config.INTERNAL_ID_COL], errors="ignore")
    y = train[target].astype(float).to_numpy()
    X_numeric, X_test_numeric = _prepare_model_matrices(X_train, X_test)
    X_cat, X_test_cat = _prepare_catboost_matrices(X_train, X_test)

    predictions = {}
    timings = {}

    start = time.perf_counter()
    catboost = models.get_catboost(**{
        key: value for key, value in manifest["catboost_config"].items()
        if key != "native_categorical"
    })
    cat_features = [
        index for index, dtype in enumerate(X_cat.dtypes)
        if str(dtype) in {"object", "string", "category"}
    ]
    catboost.fit(X_cat, y, cat_features=cat_features, verbose=False)
    predictions["catboost"] = catboost.predict(X_test_cat)
    timings["catboost_seconds"] = time.perf_counter() - start

    start = time.perf_counter()
    ridge = models.get_ridge(**manifest["ridge_config"])
    ridge.fit(X_numeric, y)
    predictions["ridge"] = ridge.predict(X_test_numeric)
    timings["ridge_seconds"] = time.perf_counter() - start

    start = time.perf_counter()
    elasticnet = models.get_elasticnet(**manifest["elasticnet_config"])
    elasticnet.fit(X_numeric, y)
    predictions["elasticnet"] = elasticnet.predict(X_test_numeric)
    timings["elasticnet_seconds"] = time.perf_counter() - start

    weights = manifest["ensemble_weights"]
    blend_prediction = sum(weights[name] * predictions[name] for name in weights)
    submission_df = submission.build_submission(test_raw, blend_prediction)
    submission_path = output_dir / "S01_frozen_v1_blend.csv"
    submission.write_submission(submission_df, submission_path, expected_rows=len(test_raw))

    component_paths = {}
    for name, prediction in predictions.items():
        component_paths[name] = output_dir / f"S01_{name}_predictions.csv"
        pd.DataFrame({config.ID_COL: test_raw[config.ID_COL].to_numpy(), config.TARGET: prediction}).to_csv(
            component_paths[name], index=False
        )

    metadata = {
        "submission_id": S01_ID,
        "model": "Frozen V1 Blend",
        "feature_version": "frozen production",
        "oof_rmse": 20097389.205713525,
        "catboost_weight": S01_WEIGHTS["catboost"],
        "ridge_weight": S01_WEIGHTS["ridge"],
        "elasticnet_weight": S01_WEIGHTS["elasticnet"],
        "public_leaderboard_rmse": None,
        "leaderboard_rank": None,
        "timestamp_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "submission_file": str(submission_path),
        "component_files": {name: str(path) for name, path in component_paths.items()},
        "fit_method": "fit_each_frozen_component_on_all_3493_processed_v1_training_rows",
        "catboost_production_iterations_rule": "fixed_manifest_value_no_early_stopping",
        "manifest_path": str(manifest_path),
        **timings,
    }
    metadata_path = output_dir / "S01_metadata.json"
    metadata_path.write_text(json.dumps(metadata, indent=2), encoding="utf-8")
    return submission_path, metadata_path


if __name__ == "__main__":
    for path in run():
        print(path)

13. Competition performance versus generalisation

Once the clean model and its validation protocol were frozen, a separate competition-optimisation stage improved predictions for the specific held-out evaluation set. The mechanics are out of scope here. What matters methodologically is that the stage drew on information from the competition environment, not on newly labelled training data.

So the final leaderboard RMSE of roughly ₹2.893M should be read as specific to the competition and potentially overfit to that evaluation set. The fair estimate of how the underlying model would do on new restaurants remains the repeated-CV result of about ₹20.10M. Keeping the two numbers apart is essential for both scientific and business interpretation.

In a real forecasting deployment, the competition-optimised layer would be dropped unless an equivalent calibration signal were available prospectively. The deployable system would use the frozen feature pipeline and ensemble, watch residuals and distribution drift, and retrain as actual turnover arrives.

14. Assumptions

• Train and test were assumed to come from sufficiently similar business processes for cross-validation to mean something. No formal covariate-shift correction went beyond schema and distribution checks.
• The registration number was strictly an identifier and left out of the clean model, since it had no defensible link to turnover.
• Missing amenity ratings were assumed to carry structural information when the service was absent, which drove the availability-flag design.
• All fitted preprocessing statistics had to come from training data and be reused unchanged at inference, a rule the pipeline itself enforces, so it never depended on an analyst remembering.
• External market data counted as admissible only if it improved repeated validation and ensemble performance. Plausibility alone wasn't enough.

15. Recommendations

• Treat data quality as model performance. Schema mismatches, fragmented city names and inconsistent categoricals can wipe out signal before an algorithm sees it.
• Separate "missing because unknown" from "missing because the capability doesn't exist". The latter is often a feature, not a nuisance.
• For skewed revenue targets, track tail RMSE and the concentration of squared error as well as the aggregate figure. A few large accounts can decide whether a forecast is financially useful.
• Use explicit promotion gates: candidate discovery, confirmation on independent seeds and ensemble substitution. This stops a gain seen on one split from being crowned champion by mistake.
• Treat agentic AI as an operating model, not a licence for autonomous promotion. Let agents propose, build and challenge, while high-impact model and product decisions stay at human gates.
• Store the record of truth somewhere other than the chat window. Datasets, manifests, OOF predictions, experiment ledgers, frozen champions and the decisions to promote them should all be persisted, letting a fresh agent pick up from the evidence and not from a scroll of earlier messages.
• Separate DEV, UAT and PROD when experimenting with ML. Free creativity goes in DEV, while replication and hostile review happen in UAT, and PROD should be deterministic, frozen and reversible.

16. Lessons

The main lesson isn't that one algorithm won. Disciplined problem framing, data auditing, tail-aware validation, feature engineering, diverse residuals and a controlled agentic workflow produced a better forecasting system than model complexity alone. With DontPanic the agents worked as a team, speeding up discovery and implementation, while explicit human gates kept speed from replacing judgment.

The final competition champion reached about ₹2.893M RMSE after an optimisation stage tailored to the evaluation environment. That score settled the competition, yet it is reported separately because it may be overfit and shouldn't be read as the expected error on future restaurants.

The wider lesson is a practical one. Set up validation ahead of any tuning, hold on to failed experiments, promote gains only after they replicate, and never confuse a model that tops a benchmark with one you would trust in production. And if several AI agents are involved, make them play volleyball: let them volley work aggressively, but make sure the important decisions return to a human before the point is scored.