Why we render video from JSON, not HTML

Language models write HTML, CSS and animation code well. Ask a frontier model for an animated title card and you get a working page in one shot. Our renderer is HTML too: every Clipwise frame is a DOM tree, styled with CSS, animated with GSAP and captured from headless Chrome.

Our models still do not write that HTML. They write a scene graph in JSON, and our code compiles it to DOM. Four reasons:

Same JSON, same frames. Time is data, not code, so the preview and the export match frame for frame.
Checks before render. A JSON scene passes or fails a check before anything renders. HTML cannot fail: a browser renders any input.
Edits that name one node. /children/1/style/color addresses exactly one property. A patch touches nothing else.
One document for the user and the model. A drag in the editor and an edit from the model write to the same paths.

What the model writes

A video is a list of scenes. A scene is a tree of nodes. Each node says when (startTime, endTime, locked to the narration), where (positioning on a 720×1280 canvas), how it looks (style, a subset of CSS) and how it moves (animations). There are 7 node types and 38 shapes.

A real 5-second scene: “FIXED” rises in, pops on the beat at 1.6 s and leaves; a bench slab slides in and lands with a squash. ... marks fields cut for space:

{
  "id": "native-346-scene-1", "type": "scene",
  "startTime": 0, "endTime": 5,
  "style": { "width": 720, "height": 1280, "background": "#0A0A0A" },
  "children": [
    {
      "id": "s1-fixed-word", "type": "text",
      "startTime": 0, "endTime": 5,                // when
      "positioning": { "x": 64, "y": 168, ... },   // where
      "style": { "fontSize": 112, "color": "#FFD400", ... },
      "properties": { "content": "FIXED", "font": "Anton" },
      "animations": [{
        "phase": "main",
        "timeline": [                              // how it moves
          { "to": { "opacity": 0.3, "y": 36, "duration": 0 } },  // start
          { "to": { "opacity": 1, "y": 0, "duration": 0.5 } },   // rise
          { "to": { "opacity": 1, "duration": 1.1 } },           // hold
          { "to": { "scaleX": 1.15, "scaleY": 1.15, ... } },     // pop
          ...                                                    // exit
        ]
      }]
    },
    { "id": "s1-bench-top", "type": "shape", ... }  // slab slides in
  ]
}

Steps run in sequence: each starts when the previous one ends. A zero-duration step sets the start state; a hold changes nothing for its duration. The model never writes a clock, a callback or a loop.

1. Same JSON, same frames

Capturing an HTML page frame by frame repeats only if every moving part reads the same clock. Free HTML breaks that in many legal ways:

CSS @keyframes and transitions run on the compositor’s clock, not on a timeline you can seek.
setTimeout, setInterval and requestAnimationFrame loops move state per callback, so the frame depends on how many callbacks ran.
Math.random() and Date.now() change every render. Models reach for them first for particles and “organic” motion.
onComplete chains make frame 120 depend on frames 0–119 having played in order. Seek straight to 120 and you get another picture.
Runtime fetches of images and fonts render differently when the network is slow.

A virtual-time library catches some of these, not all. You find the leak when a customer’s export flickers.

In the grammar, none of these can be written. Our compiler turns every animation in a scene into one paused GSAP timeline that moves only when the player seeks it. Frame n is time n / fps; saved timings stay in seconds, so trims and non-integer frame rates never accumulate rounding error. Randomness is seeded: image motion uses a CRC32 of the scene ID, and transitions take an explicit seed.

Export opens the editor’s own preview in headless Chrome and steps it:

Chrome runs with --deterministic-mode, --enable-begin-frame-control and --run-all-compositor-stages-before-draw, so a frame is drawn only when the renderer asks.
A virtual-time library replaces requestAnimationFrame, performance.now and setTimeout.
Per frame: set the time, seek the timeline, wait for paint and media, capture. FFmpeg encodes (x264, CRF 16).

Inside one pinned render image (browser, fonts, assets), what the user approves in the editor is what the MP4 contains.

2. Checks before render

Checking JSON is cheap: parse it, walk the dict, compare against allow-lists. Checking HTML is not. The HTML standard defines a recovery for every malformed input, so a parser never rejects a page, and what the CSS and JavaScript do is known only after render.

Every edited scene goes through a sanitizer, our v2 render fragments go through a strict validator, and the renderer clamps what is left:

Rule	What the code does
Node types	Allow-list of 7 types and 38 shapes. Anything else is dropped.
Keys	Each node type has a key allow-list. Unknown keys are dropped.
Numbers	Formula strings such as `"720/2"` or `"SCENE_WIDTH"` are rejected.
Position	The renderer clamps `x` and `y` to the canvas.
Time	Child times are clamped into the scene window. A missing start, or an end before its start, is fixed.
Identity	On composed nodes, an edit cannot change a media ID, so it cannot swap the user’s upload.
Text	On text nodes an edit adds or replaces: contrast below 2.0 is repaired when both colours are known; an unknown font becomes the brand kit’s font.
Size (v2 fragments)	At most 64 nodes, 16 animations per node, 2,000 characters of text, 600 s. Colours must be `#RRGGBB`, URLs `http(s)`.

Each rule is a few lines of Python over a dict. Visual fit still needs a render in both formats.

The checks also shrink the model’s job. A model that cannot express setInterval cannot get it wrong.

3. Edits that name one node

Most requests after the first draft are edits: “make BENCHES yellow”, “move the subline down”, “slower”. Our revise path asks the model for an RFC 6902 JSON Patch:

[{"op": "replace", "path": "/children/1/style/color", "value": "#FFD400"}]

The path names one node by its place in the tree. HTML has no such address. In this scene color:#FFFFFF appears in three elements, so a search-and-replace edit must quote enough nearby markup to match once. That match is text, not structure: after a later edit changes the nearby markup, the same request fails or hits another element.

A patch also touches only what it names. A full rewrite of a scene can silently undo a font size the user set by hand. Only our schema and timing repairs touch anything outside the path, and the patched scene goes through the checks in section 2.

Tokens are not the reason. Measured with the Qwen3.5 tokenizer on the scene above, against the same scene as one HTML page with a GSAP timeline:

Output	JSON	HTML
Full scene	1,404	1,358
Change one colour	24 (patch)	37 (search-and-replace)
Move one line and change its text	44 (patch)	42 (search-and-replace)

The formats cost about the same. The gap is against a full rewrite, which is what a model does without an edit format: 1,404 output tokens against 24, or 47 seconds against under one at 30 tokens/s (why output tokens are the expensive ones).

The input still carries the whole scene, because the model needs context to edit correctly. Patches cut output tokens, not input tokens.

4. One document for the user and the model

When a user drags a headline or retypes a word, the editor sends the changed fields for that one node ({"style.color": "#FFD400"}), with no model call. The model’s next edit sees that change at the same path. Model-written HTML has no stable place for a drag to write to.

Two choices in the compiler make this safe:

Position and motion live on different elements. An outer wrapper holds the node’s position; GSAP animates only an inner element. A drag never fights an animation.
Our code builds the DOM, not the model. Model-written JavaScript previewed in the editor runs on your app’s origin, with the user’s session. Sandbox it in an isolated iframe and the editor can no longer reach in for drag, resize and inline text edits.

What it costs

No model knows the grammar. A general model sees it fresh on every call, so the schema, rules and examples go into every prompt.
Expressiveness has a ceiling. If the grammar has no node for an effect, the model cannot produce it. A new effect means renderer code, a validator rule and training data.
You own a compiler. Every node type needs a renderer, an editor interaction and a validator. It pays back every time an export matches its preview.