Translating 300 Subtitle Segments in 7 Seconds Instead of 35

Translation is the slowest thing our API does. Fetching a transcript is a network hop and some parsing; translating it means a language model has to write every line again. The output is the bottleneck, not the input, and that distinction turns out to decide the whole design.

When you call GET /transcript with translate_to, we hand the segment list to a model and get a translated segment list back. The shape of that handoff — one call or many — is invisible from outside. It is worth about 28 seconds.

The naive version, and why it is slow

The obvious implementation numbers every segment, joins them into one block of index|text lines, and asks the model to return the same format. It is short, it is easy to parse, and it is correct.

# 286 segments in, 286 segments out, one request
def translate(segments, target):
    numbered = "\n".join(
        f"{i}|{s['text']}" for i, s in enumerate(segments)
    )
    out = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": PROMPT.format(lang=target)},
            {"role": "user",   "content": numbered},
        ],
    )
    return parse(out.choices[0].message.content)

For the 19-minute video we test against — 286 segments — this took 35.0 seconds. Not because the model is slow in any interesting sense, but because it produced roughly 5,000 output tokens and it produces them one after another. There is no parallelism to find inside a single completion. You are watching a single decode loop run to the end.

Thirty-five seconds is past the default timeout of many HTTP clients and well past what anyone calling a synchronous API will sit through. We could have made the endpoint asynchronous and handed back a job ID, but that pushes complexity onto every caller for a problem that is structurally parallel. The transcript is already a list. Nothing about segment 200 depends on segment 12.

Chunk it, then fan out

The fix is to split the segment list into fixed-size chunks and send them at the same time, bounded by a semaphore so we do not open an unbounded number of connections on a long video. We settled on 40 segments per chunk and concurrency 8; the reasoning for both numbers is further down, and neither is arbitrary.

The one thing that must survive chunking is the numbering. Each chunk is labelled with its absolute position in the transcript, not with a local 0..n. That single decision is what lets the chunks come back in any order.

# CHUNK is a latency knob. It gets retuned. See the pricing section.
CHUNK = 40
CONCURRENCY = 8

async def _one(client, chunk, offset, target, sem):
    # global indices: offset + i, never 0..n
    body = "\n".join(
        f"{offset + i}|{s['text']}" for i, s in enumerate(chunk)
    )
    async with sem:
        r = await client.chat.completions.create(
            model=MODEL,
            messages=[
                {"role": "system", "content": PROMPT.format(lang=target)},
                {"role": "user",   "content": body},
            ],
        )
    return parse(r.choices[0].message.content)  # -> {global_index: text}

The driver builds one task per chunk, awaits them all with asyncio.gather, and merges the results. Because every chunk returns a dict keyed by global index, the merge is a plain dict.update in whatever order the results happen to arrive.

async def translate(segments, target):
    sem = asyncio.Semaphore(CONCURRENCY)
    tasks = [
        _one(client, segments[o:o + CHUNK], o, target, sem)
        for o in range(0, len(segments), CHUNK)
    ]
    parts = await asyncio.gather(*tasks)

    merged: dict[int, str] = {}
    for p in parts:
        merged.update(p)  # order-independent: keys are absolute

    return [
        {**s, "text": merged.get(i, s["text"])}  # missing -> source line
        for i, s in enumerate(segments)
    ]

Same video, same model, same prompt: 6.6 seconds over 8 calls.

Global indices do more than reassembly

It is tempting to number each chunk from zero and stitch by list position afterwards. Do not. Global indices buy two separate things, and the second one is the important one.

The first is order independence. asyncio.gather preserves task order in its return value, so you could get away with positional stitching there — but only until someone adds a retry, a per-chunk timeout, or a fallback that skips a chunk. Keying on absolute position means the merge cannot be wrong, regardless of what the concurrency layer does later.

The second is failure containment. Models drop lines. Occasionally a chunk of 40 comes back with 39, because two short consecutive captions got merged into one. With local indices, a single dropped line shifts every subsequent subtitle in that chunk by one — the text is fine and the timings are all wrong, which is the worst possible failure for subtitles because it looks plausible. With global indices, a dropped line simply has no key in merged, and merged.get(i, s["text"]) falls back to the original source text for that one segment. You get one untranslated line in the right place instead of an entire chunk sliding out of sync.

A missing translation is a cosmetic defect. A shifted timeline is a broken file.

That fallback is also why we can be relaxed about the occasional imperfect chunk rather than retrying it and paying the latency back. If you are building SRT or VTT output on top of this, the cookbook recipe assumes exactly this invariant: the segment count and the timings out always match the segment count and timings in.

The cost: about 8% more input tokens

Chunking is not free. The system prompt — the instruction block that explains the target language and the output format — has to be sent with every chunk. Eight calls means eight copies of it.

For the 19:13 video, the single-call baseline used 4,505 input tokens and 5,019 output tokens. Chunked at 40, the same video used 4,871 in and 5,460 out.

At $0.20 per million input tokens and $1.20 per million output, the absolute difference is fractions of a cent. We still track it, because the overhead scales with the number of chunks, not the length of the video — halve the chunk size and you double the prompt tax. It is a real constraint on how small chunks can usefully get, independent of everything else.

Latency stays flat until the semaphore saturates

With eight concurrent slots, any transcript that fits in eight chunks — up to 320 segments — costs roughly one model call in wall clock, plus scheduling overhead. Past that, chunks start queueing.

This is the shape you want, and it is worth being precise about why the last bar is tall. The 61:00 video is not 1.9× slower because it has more text — it has 329 segments against 286, only 15% more. It is slower because 329 segments produce 9 chunks, and there are 8 slots. Eight go out immediately; the ninth waits for whichever one finishes first and then adds most of another round-trip on top. Cross a multiple of CHUNK × CONCURRENCY and you pay a step, not a slope.

Costs include both the model tokens and the proxy bandwidth of fetching the transcript itself, so they are what the request actually costs us end to end rather than a token line item. Note the 61:00 row: fewer input tokens than the 19:13 row but more output, because it is a sparser transcript with longer individual captions. Segment count, not duration, is what drives everything here — which becomes the pricing question below.

Smaller chunks are faster and worse

The obvious next move is to shrink the chunk and raise the concurrency. More calls, more parallelism, shorter critical path. We tried it. It works, and we did not ship it.

We do not have a tidy theory for the exact mechanism, and we are not going to invent one. The empirical result is that fidelity degrades as chunks shrink, and 40 was the largest setting where the latency curve had already flattened and the drop rate had not yet started climbing. Both properties held there; that is the whole justification. If you are tuning this yourself, measure returned-segment count alongside wall clock, because it is the metric that silently gets worse while the one you are watching gets better.

The fallback described earlier is what makes 285-of-286 acceptable at all. Without it, a 99.7% success rate per file would be a correctness bug rather than a cosmetic one.

Pricing has to follow length, not calls

Once translation is chunked, the billing question answers itself if you look at the token numbers. Across the four videos, spend works out to roughly 33 input tokens and 18 output tokens per segment. It is close to linear in segment count and essentially unrelated to anything else.

A flat fee per translation would overcharge the 6-segment video by an absurd margin and lose money on every hour-long one, and the people translating hour-long videos are exactly the ones who use the endpoint most. So translation is billed per 40 segments, which is what the credits page describes: the price of a translation moves with the size of the transcript, the same way the work does.

There is one implementation detail here that looks like duplication and is not. The billing granularity and the chunk size are both 40, and they live in two separate constants:

# latency knob. Retuned whenever the model changes.
CHUNK = 40

# billing granularity. Frozen. Changing it changes what people pay.
BILLING_UNIT = 40

def translation_credits(segments):
    return math.ceil(len(segments) / BILLING_UNIT)

If those were one constant, then the next time someone tunes chunk size to 32 because a new model handles smaller batches better, every customer's bill for the same video would change by 25% as a side effect of a performance experiment. Nobody would intend it and it would probably ship. Performance constants and price constants have different change rates, different reviewers, and different blast radii, and they should not be the same symbol just because they currently hold the same number. Anything that moves BILLING_UNIT is documented on the credits page; latency tuning never shows up there, which is exactly the point of keeping them apart.

What we would do next

The current setup has an obvious ceiling: past 320 segments you pay a full extra round in latency for the 321st. A work-stealing pool with variable chunk sizes would smooth that, splitting the tail into smaller pieces so the last round is short rather than full. We have not built it because the step is only visible on videos over an hour and 12.3 seconds is still inside what callers tolerate.

The more interesting open question is whether chunk boundaries should follow the content instead of a fixed count — breaking on pauses rather than every 40 lines, so a chunk never splits a sentence across two calls. That might raise the 285-of-286 figure. It would also make the chunk count unpredictable, which is fine for latency and awkward for everything else.

Until then: 40 segments, 8 slots, absolute indices, and a fallback that keeps the timeline intact when the model drops a line. Roughly seven seconds for a twenty-minute video, and honest about the 8% of extra input tokens that buys it.