Gemini Agentic Video Mode: Schema Design and Timestamp Drift

Google recently released an "agentic" video understanding feature for Gemini, and it makes a fine excuse to run some hands-on experiments with a hard test case: a feature-length Chinese-language movie, Secret (2007) by Jay Chou. The experiments concentrate on one question. If you ask the model for the moments in a video where key ideas appear, how accurate are the timestamps, and how much does the way you design the output schema change the result?

You'll need a GOOGLE_API_KEY, which AI Studio hands out with a decent free tier. Setup details sit in the appendix at the end.

Why a Chinese movie makes a good test

The idea began with a study tool, built to help students work through assigned material, though curious adults could get something from it as well, by running into unknown unknowns. After noticing that a Gemini Flash Lite model could both summarize Chinese and reply in Chinese, widening the scope from education to entertainment looked natural. The film was a good candidate, since it's a piano-heavy romance with a memorable soundtrack and a lead actress who carries the quiet moments.

A spoiler warning applies: what follows reveals the film's central secret. The source video for the tests is a YouTube movie analysis that retells the story up to roughly the 16:23 mark, after which the narrator moves into analysis of how the plot was inspired by Jay Chou's own life. The movie itself is still worth watching for its music and the lead actress's subtle expressions, and there are at least two versions of it on Bilibili.

What makes this a genuinely difficult test is the language. Chinese has far less auto-generated transcript coverage than English video, so the model is handicapped from the beginning and has to show what it can do without that scaffolding. You don't need to read Chinese to follow along, though it helps in appreciating the outputs.

Why timestamp drift matters

Across study and entertainment alike, a common need is localizing timestamps: jumping directly to the important moments. Students do it to revisit a concept's explanation. Viewers do it to find easter eggs and revisit good storytelling.

Progressive timestamp drift in Gemini's audio transcription has been widely reported on Google's developer forum. A dependable remedy is an older, non-LLM model, but that's hardly an interesting experiment, so the hope was that agentic mode would do better.

The short version: it still drifts. About 95% of the time the error stays within one second, but some points show wild drifts of 10 to 20 seconds. Interestingly, the drifts cluster at the same few places, which is good enough that casual use wouldn't annoy anyone. Developers can also filter out bad timestamps afterward by string-matching against a time-labeled transcript within a time window, assuming such a transcript is available.

Schema design beats prompt design

LLM schemas are usually written in JSON Schema. Libraries such as Pydantic in Python or Zod in TypeScript let you define them at application level and compile to JSON Schema automatically before they're injected into the prompt. From the variable names and types alone you can usually predict what the model will return.

The experimenter's view is blunt: it's odd that prompt engineering gets so much hype while schema engineering is where the real power sits. A schema encodes nearly the whole behavior of the model. Even with a mediocre prompt, a model can infer what you want from field structure and names, whereas with a poor schema you're on your own and hoping a brilliant prompt rescues you.

There's a parallel with data modeling in data engineering, with the difference that LLM output schemas resemble NoSQL design, such as MongoDB, more than relational modeling, such as Postgres. One interesting decision concerns recursive structures: they can be encoded as flat, self-referencing tables, with a child_id and parent_id allowing unlimited chaining. Providers differ on whether they permit recursive JSON Schema at all, so that choice may be forced on you.

Reading the schema

As in data modeling, cardinality matters. One movie has many concepts, expressed as an array. In education a topic has prerequisites, applications, extensions, assumptions and limitations. In a film there are events, character development, a climax and a resolution.

One concept has many timestamps. In teaching, that happens when several concepts are compared to build higher-order thinking after each has been introduced. In film, it happens when directors use flashbacks or tell the story out of order.

The timestamp entries carry a handful of properties:

• seconds says where the moment occurs.
• description explains what the quote means, which can help with jokes or with reinterpreting the script.
• quote is the script segment extracted verbatim through speech-to-text.
• visual flags whether the moment can be depicted.
• visualDescription is a caption of what's on screen at that point.

There are also checkpoints, each with a conceptLabel, a question and an answer. In education they test the learner's understanding. In entertainment they prompt the audience to consider things missed on a first watch, or to reread events from another point of view. The model was clever enough to reuse exactly the same text for conceptLabel and the matching label under concepts, a small but telling sign of internal consistency.

What was varied

Typical runtime was about 20 seconds. Some runs finished in 9 seconds, probably because Google cached the prompt and reused results. One took 50 seconds right after a prompt change, likely before any cache existed.

Processing mode, agentic against static, made little difference on its own, and most timestamps landed within a second either way. Agentic runs sometimes took up to twice as long (about 40 seconds) and sometimes less. Script runtime did affect quality: longer runs found more timestamps per concept and produced longer descriptions and quotes.

The prompt included the instruction to find at least three timestamps per concept label. Without it the model often found only one per concept. With it, timestamps seemed slightly more accurate, and the working hypothesis is that raising search density had a side effect on precision. The number three is worth varying to see what happens.

An appended request asked the model to find the moment a specific Chinese line was spoken, the film's climax, where a huge iron ball comes crashing toward the protagonist. That forces the model to match audio against an exact phrase and to know when it occurred.

Round one: static volunteers extra fields, agentic leaves them out

In the first attempt, static mode tended to add visual and visualDescription keys that the schema never required. It did so inconsistently, including them for some timestamps and omitting them for others within one run. visualDescription is useful for accessibility and for filling search databases, since key moments could be searched through the generated caption, and it was accurate in every example examined.

Only quote was reliably in the original Chinese. Every other key could come out in Chinese or in English translation, unpredictably across runs, agentic or static. In both modes, timestamps were neatly ordered within and across concepts, which made hand verification easy.

Agentic mode produced longer quotes, a pattern that returned in the second attempt. It also gave the first appearance of a scene that keeps resurfacing: agentic mode labeled one moment 10:41, but it actually occurs at 10:50 in the video. Had visualDescription been present in that output, it would have shown whether the model was truly looking at 10:41 or 10:50.

The static run's description for one entry merely echoed its label ("Time Travel Mechanics"), adding little, though later runs showed more meaningful differences. On the positive side, static did a great job spotting the old piano book and the heroine closing her eyes in visualDescription, and even identified the character correctly. Agentic simply ignored the optional visual keys.

Second run: static gets the 10:50 moment right

Static again added the optional visual keys and agentic again didn't. Static also correctly found the 10:50 moment that the first agentic attempt had mislabeled, even though that attempt had the right text. Static labeled the moment's visualDescription as a "flashback scene", raising a question: how did Gemini know it was a flashback and not the present timeline?

Timestamps stayed ordered within each label in both modes. Across labels, static brought the narrator's analysis forward, while agentic interleaved times such as 10:41, 17:00, 15:00 and 17:51.

Static's first quote, at 16:31, used the Chinese book-title brackets that also appeared on screen, which suggests the model captured that image instead of relying only on audio. Static kept to shorter quotes, as before. Agentic stretched its quotes until the middle truncated with an ellipsis; its quote for 15:00 really spans 15:01 to 15:38, and its quote for 17:51 spans 17:54 to 19:56.

That last span is striking, because 17:54 to 19:56 covers the full description of the film's four endings. It prompts a question: what if the prompt asked for an end timestamp as well? That question motivates the next schema change.

Third run: adding time ranges

The seconds field was originally a string, so it had to be reworked. There are two options. The first is an array:

"seconds": {
    "type": "array",
    "description": "The start and end time range in HH:MM:SS or ISO format, structured as [start_time, end_time].",
    "prefixItems": [
        {
        "type": "string",
        "description": "Start time (e.g., '00:01:15' or ISO 8601 string)"
        },
        {
        "type": "string",
        "description": "End time (e.g., '00:02:30' or ISO 8601 string)"
        }
    ],
    "minItems": 2,
    "maxItems": 2
    }

The second is an object:

"seconds": {
      "type": "object",
      "properties": {
        "start_time": {"type": "string"},
        "end_time": {"type": "string"}
      },
      "required": ["start_time", "end_time"]
    }

The array uses fewer tokens, but it needs clumsy min and max bounds to keep start-end logic from breaking, and it was used for the experiments below. The object gives clearer keys and takes a bit more code to read, but tolerates extra keys, so outside experiments it's the recommended choice.

The quote isn't only audio

In agentic mode, one range ran 6:20 to 6:42, though the quote actually began at 6:30. The description matched the audio word for word starting at 6:20, for eight words, before switching to its own paraphrase. It's as though the job of quoting had been split between quote and description.

In another agentic range, 8:13 to 9:00, the quote (roughly "eyes locked") began at 8:21, while the description said "show up again", and the female lead does appear around 8:13. That supports the notion that time ranges are shaped by more than audio. The quote ended at 8:55, five seconds before the range did, for no obvious reason.

The implication is that audio should line up closely with what's on screen when video is parsed, or timestamp localization gets confused. Yet nothing obvious controls which modality a multimodal model weights more heavily when it emits a timestamp. That's a genuine research opportunity, and a cleaner ablation (full audio and video against blacked-out video against muted video) would isolate it.

Two new failure modes

A third appearance of the recurring scene exposed a new failure. In static mode, a range at 14:49 to 15:02 contained a quote that actually comes from 10:49, a four-minute error. It's the same 10:50 moment that static found accurately in the second run and that agentic mislabeled as 10:41 in the first.

So one real moment, tested three times across two schema versions, was first off by nine seconds, then exact, then off by four minutes. If drift were just a property of an inherently hard scene, you'd expect consistent failure, not failure, then success, then worse failure.

The second failure: a static quote for 15:07 to 15:17 (roughly "he's going to find Xiaoyu") never occurs in the audio. The model seems to have been summarizing on its own instead of transcribing.

Static wins the needle-in-a-haystack test

To see whether a specific quote can be located, the prompt was extended with the iron-ball line. Before the time-range schema, static found 15:22 and agentic found 15:00. After it, static hit the exact moment, 15:21, and even ran on to 15:26, just as the ball hits the building and the noise dies away. Agentic did worse, with a range of 15:00 to 15:37 that was far too wide. In practical use, being even three seconds off would test a viewer's patience.

What the tests suggest

Static is better at finding moments, and agentic is better at summarizing ideas that unfold over longer stretches. The sensible conclusion isn't to pick a winner. Route by task and media length: static for pinpoint retrieval or short clips where cost differences hardly matter, agentic for synthesis across something like a 90-minute lecture, where its token savings begin to count.

Moving from a timestamp to a time range didn't change the starting point, judging by the search experiment, so it's wise to always fetch the end time too, which gives an application much more room for UX design. It's rather like doing vector embeddings in a search system without embedding anything.

Time ranges may be influenced by several modalities, and even within text by several keys, as the quote-offloading pattern shows. Perhaps the schema should be simpler so the model isn't confused, or the relationship between derived and sibling fields should be made explicit, for example by instructing that description must derive from quote, or by deciding whether concepts are independent or must explain causality between themselves.

One more pattern deserves a mention. After a time range was requested, static stopped producing visual and visualDescription altogether. One hypothesis is that the model focused on the required fields and dropped optional keys under load. Making them required might fix it, possibly at the cost of quality elsewhere, since less thinking budget would remain for them. It resembles the way a slide-generation tool can produce excellent slides on one run and regress to stick-figure design on the next, as though it needed a rest.

Where this could go

Language learning. Prompting for English and Chinese side by side on the same line worked in some unshown experiments even without being requested.

Video creation. Timestamps and ranges let downstream tools such as ffmpeg cut and trim so the main concepts come first. LLMs are lossy here, so to keep all content and only remove pause fillers, more specialized speech-to-text tools such as Descript, or Google's newer Gemini transcription model, are a better fit.

What accurate localization makes possible:

• Jump-to-moment navigation. Timestamps work as anchors that turn linear video into an indexed, searchable resource, replacing manual scrubbing.
• Automated highlight reels. With specific moments findable without manual scanning, creators can assemble scenes across videos, or educators can cut a timestamped interactive lesson.
• Natural-language moment search across lectures and meetings. The "find this exact line" test on the film's climax is the same trick a student could use on a recorded lecture: type the one sentence half-remembered and jump to where it was said.

Thinking about edges as well as nodes

In the current design, most information is packed densely into nodes. How might the edges be modeled to improve the experience further? Should some information move from nodes to edges, or are there knowledge representations beyond a graph of nodes and edges? Techniques from the book Learn Like a Polymath could apply: part-whole, cause-effect, thing-function, problem-solution, fact-opinion and example-type-of. LLMs can likely produce such analogies already, and the trick is to ask for the right thing at the right time and present it in the right way.

One answer already exists in a text-only study app that builds a live concept map during chat. There the agent generates the edges itself. Ask about singing, then later about vocal technique, and the agent clusters them into linked nodes without being told the relationship, with color coding for main topic, abstraction, sibling and detail serving as the edge information. The checkpoints property in this schema has a live analogue as well, flagging unresolved questions and possible misconceptions during a chat that then resolve once understanding is shown.

Experiments still to run

These are specific enough for anyone to try, along with what each might reveal.

Ablation on modality conflict. Run the same video four ways: full audio, video and subtitles; full audio and video without subtitles, including those already burned into the picture; blacked-out video, so audio only; and muted video, so visual only. Should drift disappear for the audio-only version, sound dominates and the burned-in subtitles were muddying the result. Should it persist for audio-only yet change character for visual-only, that is stronger evidence that timestamps are truly multimodal and not simply audio with visual noise.

Caching and temperature against reproducibility. Will pairing prompt caching with temperature zero return identical moments on every run? Should drift vanish at zero, the model holds one stable, if imperfect, belief about where a moment sits, and across-run drift is mostly sampling noise. If drift persists at zero, there's no stable internal timestamp for that moment at all.

Does drift follow content or position? If it hinges on traits of the moment itself, such as rapid cuts, talking over each other or swelling music, wherever it falls, that's a translation-invariant failure that travels with the content. If it worsens further into a long video, it's something else, possibly related to "lost in the middle", the drop in LLM performance on information buried mid-context.

Record linkage across schema segments. Can the model recognize that two concept labels, produced in separate calls or separated by long context, name the same real-world concept? That's the classic record-linkage or entity-resolution problem, close to coreference resolution when it happens within a single generation. It worked once, when the model reused identical text for conceptLabel and label, but the open question is whether consistency survives across independent calls.

Cross-lingual grounding. Measure how faithfully Gemini turns spoken Chinese into English schema fields while keeping the timing accurate.

Measure, don't eyeball. A modest table listing timestamps checked, median drift, percentage within one second and worst outlier across all six attempts would upgrade the observations from "noticed" to "measured", and the raw outputs already exist.

Test whether required visualDescription changes quality. The model seemed to drop optional keys as the schema grew heavier.

Lectures, films and memory

A lecture and a movie ask different things of a viewer. A lecture wants attention, a movie wants feeling. Characters in a story aren't solved problems like a concept in a textbook; they're written like real people whose motivations are open to interpretation. Often there's no correct answer or even a clean solution, only trade-offs, plus the effort of salvaging what you can from a regrettable situation, a life lesson that extends well beyond one subject.

The neurologist Oliver Sacks described a patient, Clive Wearing, whose amnesia left him unable to retain a memory beyond a few seconds, except for music, which he could still perform, read and recall completely. Sacks put it this way: music's power must steal on one unawares and come spontaneously as a blessing.

That's close to what an LLM might become for us: no replacement for memory, but a help with how imperfect our memories already are, bringing us back to moments that mattered and reordering them into something coherent when we're confused and overwhelmed.

The collaboration may also run both ways. Schemas can be refined to draw out the best from LLMs, and in turn LLMs can reshape our worldviews and abilities, helping us think critically about a subject and also build empathy and social skill by understanding why a character behaves as they do, exploring what-ifs and learning how to tell a great story. The heroine of the film arguably had four possible endings, and an LLM could likewise offer multiple lifetimes of knowledge within reach, with the ability to speed up, slow down, rewind and choose a path.

A related direction is a study agent that watches a lecture alongside you, pauses on request and answers grounded in the exact moment of the pause. That's the same timestamp-localization problem tested here, except live and harder. If agentic mode still drifts by minutes on a finished, controlled video, a live-streaming version would face a strictly harder form of the same problem. These experiments are early groundwork for that idea, not a description of a shipped product.

Appendix: setup and reproducing the runs

Get a GOOGLE_API_KEY from AI Studio, then export it in your shell:

export GOOGLE_API_KEY=<your key>

To avoid exporting it after every terminal restart, put the key in a .env file and add this to the script:

from dotenv import load_dotenv
load_dotenv()

Frustrations with the API

Unfamiliarity with client.models.generate_content() was one source of trouble, and Google's product introduction page showed code with the newer Interactions API (client.interactions.create()), which suggested agentic video wasn't supported in the older generate_content API until the documentation showed that it is. It also wasn't clear how, or whether, certain parameters such as thinking summaries could be specified. On top of that came the breaking changes to the Interactions API in May 2026, plus looping answers from Gemini's own web chat, and getting started cost hours of guesswork.

Empty thought summaries

Whether the parsing code was wrong or the task was simply too easy is unclear, but there was no thought summary at all: the output file for thoughts consistently showed an empty list.