Four chunking strategies left recall near 0.2. Reading the rulebooks took it to 0.89

Context

Finance influencers publish investment content on YouTube that can break US marketing and endorsement rules: implied guaranteed returns, performance claims with no risk disclosure, sponsorships nobody mentions, buy and sell calls from people with no credentials. Checking one video means watching it, finding the offending sentence, and matching it to a specific sub-paragraph of a rule. The videos are long, the rulebooks are dense and nested, and that combination is the problem I wanted to work on. Nobody asked me to. VeriStream AI was a personal project, built to apply what I had learnt to something real, and I finished it in March 2026.

The bar I set for the output was a citation, not a hunch. Not "probably non-compliant", but which rule was broken, down to the sub-paragraph, with the timestamp and the speaker's exact words. A video URL goes in. A list of findings comes out, each with a severity, a timestamp, a verbatim excerpt and a rule citation, in a dashboard where clicking a timestamp seeks the player to that moment.

The project began as a brand-compliance checker for YouTube ads and turned into this early on. The constraints were one developer, a laptop, and Azure OpenAI on its S0 tier, whose rate limits shaped one of the decisions below. Nothing was deployed; it runs locally. The knowledge base is small on purpose: five PDFs, 52 pages of primary rule text. They are the FTC Endorsement Guides (16 CFR Part 255), the SEC Marketing Rule (17 CFR 275.206(4)-1) and FINRA Rules 2210, 2220 and 2010. The pipeline screens for 19 categories of violation, and everything was scored against 18 test cases I wrote by hand: 7 SEC, 6 FINRA, 5 FTC.

Approach

I tried the obvious thing first: give the model a transcript and ask for verdicts with no retrieved text. The answers were fluent and sure of themselves, and the citations were wrong. They were real-sounding rule numbers, stated with full confidence, that did not govern the behaviour in question. That is harder to catch than an invented rule, and it settled the design. The model reasons, the retrieved rule text is the source of truth, and every citation has to come from a retrieved chunk.

Before tuning anything I set up RAGAS, so that every later change had a number attached. The test set is 18 cases. Each is a question of the form "Does the transcript phrase violate this rule by doing that?", a one-line transcript phrase, and a ground-truth sentence quoting the relevant sub-paragraph. The set took its final form partway through, after the corpus swap in decision 2, and stayed fixed from then on. A second harness came later: an LLM judge with structured output that gives a 0 to 5 legal-accuracy score and a 0 or 1 citation match, because RAGAS has no notion of the right rule with the wrong sub-paragraph. Both run against the live Azure services, not stand-ins.

The system is in Figure 1. A FastAPI server starts a LangGraph run and streams one progress event per node to the dashboard over a WebSocket. The indexer downloads the video with yt-dlp and sends it to Azure Video Indexer, which returns a transcript in timestamped segments plus any on-screen text. Query expansion is one GPT-5.2 call that pre-screens the transcript against the 19 categories and, for each candidate, writes three retrieval queries that name a regulator, a rule and the behaviour observed. Retrieval runs those against Azure AI Search, hybrid BM25 plus vector at k=4 per query, de-duplicates the chunks, and passes up to ten on through a semantic rerank step that I come back to at the end. The audit agent gets the excerpts, the candidates and the full transcript, and can call the search index itself, filtered by regulator, up to ten more times before it returns findings as JSON. Its prompt tells it to take citations from the chunk headers and not to fabricate or paraphrase rule numbers.

The index is built offline. Each PDF is loaded, page headers and citation-tool debris are stripped, rule IDs and the rule title are extracted, and the text is split. The piece that matters is the header. Every chunk gets a line of the form [REGULATOR rule_ids - rule_title | Type: chunk_type] prepended to its text before embedding, so the rule's identity is inside what gets embedded and retrieved, and not only in metadata. A chunk of rule text often does not name its own rule. The header fixes that, and the audit prompt and both eval prompts depend on it.

Decisions

[01]Design decisionSplit on each regulator's numbering, not on character counts

Decision

A recursive splitter at 1400 characters with 250 overlap, with a different separator ladder for each regulator that follows that rulebook's own paragraph markers, and the rule header prepended to every chunk.

Why

I ran five configurations back to back on the same 52 pages and the same 18 cases. In my build log, fixed-size chunks at 512, 1024 and 1400 characters scored context recall of 0.21, 0.24 and 0.22, and semantic chunking scored 0.25. All four were counting characters or similarity, and none knew what a clause was. The fifth has the same size and overlap as the 1400 run, so the separators are the only difference between them, and it scored 0.89.

Tradeoff

Rule IDs are extracted once per document and copied to every chunk, so a header can list every rule its PDF mentions. In the RAGAS run the generated rule line ran past 400 characters in 16 of 18 answers. Three separator maps are more to maintain than one number, and the index is wiped and rebuilt on every run.

The separator ladders came from reading the PDFs. The FTC guides use § sections, then (a)(1)(i), with Example blocks that work as natural breaks. The SEC rule nests Rule 206, then (a), (1), (i), (A). FINRA goes from 2210. to (a), (1), (A), (i), and one lettered level below that. The failure that made the point was FINRA 2210(d)(2), "Comparisons", which straddles the bottom of page 9 and the top of page 10. One chunk held the rule identifier with no requirement, the other held the requirement with no identifier, and keyword matching retrieved the empty one.

Semantic chunking lost on cost as well as recall. I set it up with some care: the breakpoint percentile lowered from the default 95 to 85, because legal text changes sub-topic quickly, and a 300-character minimum to avoid one-sentence orphans. It still embeds every sentence to find its breakpoints, and on the S0 tier it hit a 429 on a single PDF. The structural splitter makes no API calls at all.

Each bar in Figure 2 is a single run, and none of them left a result file, for a reason I come back to at the end. A gap of 0.22 to 0.89 is roughly 4 of 18 cases against 16 of 18, far more than a few cases flipping would produce. The spread among the first four is not. I wrote up the three separator maps↗ and the five-way comparison↗ at the time. The line I would keep from those posts: the hardest part of building a compliance AI wasn't the AI, it was learning to read like a lawyer.

[02]Design decisionIndex the rule, not the commentary about the rule

Decision

Replace three broad source PDFs with five narrow primary texts totalling 52 pages, and rewrite retrieval queries from keyword strings into full sentences that name the regulator, the rule and the behaviour.

Why

A diagnostic on the first setup ran 18 keyword queries at k=5 and checked the results for key terms from the ground truth. 5 of 18 queries retrieved none of them, and the 90 retrieved chunks split 55 SEC, 28 FTC, 7 FINRA, 15 of them with no rule ID at all. The SEC file was the Marketing Rule's adopting release, hundreds of pages of commentary and footnotes about the rule. The test cases quoted the rule itself. Full-sentence queries also give both the BM25 half and the vector half of hybrid search something to match.

Tradeoff

A much narrower knowledge base. The commentary and worked examples in the longer documents are gone, and coverage outside these five rules is zero. It is also not a controlled comparison: corpus, query style and ground truths all changed within a day of each other.

The retriever had been returning text like "259 See final rule 206(4)-1(a)(6)... As proposed, the final rule will prohibit...", while the test cases asked about "it is unlawful for any investment adviser to disseminate any advertisement that...". Same regulation, different document, completely different vocabulary. The swap was necessary and not sufficient. The four flat runs in Figure 2 were already on the clean corpus.

[03]Design decisionFour nodes and an agent, not one audit call

Decision

Replace the first version, an indexer followed by a single audit prompt with one retrieval, with four nodes: indexer, query expansion, retrieval, and an audit agent that can search again on its own.

Why

The retrieval query has to come from what the speaker said. A pre-screen pass turns the transcript into candidate phrases and rule-targeted queries, and the agent can fetch more text, filtered by regulator, when the pre-fetched excerpts are not enough. Along the way the taxonomy widened from 5 categories to 19, because 5 made the model force-fit some violations and miss others. Figures 3 and 4 are a day apart.

Tradeoff

At least two more model calls per audit plus up to ten tool rounds, all with the full transcript in context, and more places to fail quietly. I never measured latency or cost, and I never scored the two designs against each other.

These screenshots show an automated tool's output on public videos. They are not a legal finding about anyone.

[04]Design decisionEmbed the corpus with OpenAI, keep Azure for everything live

Decision

The indexing script embeds chunks through the OpenAI API directly. The pipeline and the evals embed queries through Azure OpenAI. Both sides use text-embedding-3-small.

Why

Rate limits. A semantic-chunking run died mid-batch on the FINRA PDF with a 429. In my one timed test, embedding 450 sentences took 11 minutes on Azure's S0 tier and 14 seconds on OpenAI directly, about 47 times faster. Azure also enforces its token limits across the whole resource, so a batch indexing job could starve the live audit's chat model. The embedding model only ever sees public regulations, so it can run anywhere. The chat model, which reads transcripts, stays on Azure.

Tradeoff

Documents and queries are embedded by two providers, which works only while both serve the same model. When recall was near zero, this split was on my own list of suspects. The isolation is also partial, since query-time embeddings still go through Azure.

[05]Design decisionBuild the scoreboard first, then debug the scoreboard

Decision

RAGAS went in before any tuning, and I treated the evaluation harness as code that could be wrong.

Why

It was wrong four times, and none of the four raised an exception. The LangChain models were passed to RAGAS unwrapped. The prompt never explained the injected header, so the model ignored it and the citations came out unmatchable. The question was never passed to the generator, only the transcript phrase, which held answer relevancy down. And a conversational preamble in each answer was being scored as an unsupported claim, which capped faithfulness. Each one silently bent a metric.

Tradeoff

Once the ruler is fixed, the early measurements stop being comparable with the late ones. Part of the rise in faithfulness and relevancy is the pipeline improving and part is the measurement being repaired, and I cannot separate the two.

One change after the bug fixes was to the context itself. Retrieved chunks had been concatenated into a wall of text, and the model blended sources. I changed that so each chunk is wrapped in a numbered block with its source, regulator, rule ID and page. In my build log that took faithfulness from 0.66 to 0.70 and answer relevancy from 0.56 to 0.61, and those end values match my final runs. With 18 cases and a model doing the scoring, moves of that size are within what one or two cases flipping would produce, and I never repeated a run to check. How context is presented to the model matters about as much as what is retrieved.

The same change took context precision to 1.00, and at the time I posted that as a high. It is an artifact. My evaluation passes the single formatted block to RAGAS as the only retrieved context, and precision over a list of one relevant item is 1.0 by construction. I do not count it as a result.

Results

Those numbers come from my final evaluation runs, and they describe the retrieval and citation layer. There is no measured end-to-end result: no labelled set of real videos, no precision or recall on violations found, no latency or cost figures.

The per-case rows say more than the means. The single retrieval miss was FINRA 2220, a six-page options rule. Three of the four citation misses were SEC cases, where every sub-paragraph sits under the one rule, 206(4)-1, so the rule is right and the paragraph is not. Answer relevancy averaged 0.615, pulled down by three scores of exactly zero, all FTC cases where the generator answered "Cannot Determine". The other fifteen answers were all "Yes", and the dataset has no compliant examples, so false positives are untested. Faithfulness of 0.71 means roughly 3 in 10 generated claims were not supported by the retrieved text, with a floor of 0.25 on one case. For a tool whose whole point is grounded citations, that is the number I would work on next.

The two harnesses also used different generator prompts. The RAGAS run asked for the bracketed header verbatim, which is where the 400-character rule lines in decision 1 came from. The judge run asked for the header's rule combined with the sub-paragraph found in the text, and its citations read like FTC 16 CFR 255.5(a). The improvement is visible side by side, and I never scored it as a delta.

I wrote this build up in public as it happened, and a few things I posted do not survive a second look. I said an ungrounded model had cited a rule that did not exist. The rule exists. It simply governs something else. I described recall as reaching a high of 0.94 after the context-formatting change, when my own chart showed it dropping from 1.00 to 0.94, which is one case of 18. I presented the 1.00 precision above as a gain. And I concluded in one post that splitting on page boundaries had been the real problem, when the splitter I shipped still does it, as the next section says. The series also stopped at post 11 of a planned 18, before it reached the pipeline, the agent or the judge. This page is the corrected account, and the first account of those three.

Looking back

I kept almost no record of the experiments. Both evaluations saved their results to the same file every time, so each run overwrote the last. The five-way chunking comparison is the best thing in this project, and it survives only as numbers in my build log. A timestamped results file per run would have made it verifiable. I would not skip that again.

Two defects I found while writing this up are still in the code. The rerank step does not rerank. After multi-query retrieval it joins the first three queries into one string, runs a fresh semantic search, and replaces the multi-query results with that, so chunks fetched for later candidates can be dropped before the audit. The agent's own tool partly covers for it, and the evals use a single query, so they never exposed it. And the structural splitter still splits page by page. Only the abandoned semantic path joined pages first. FINRA 2210(d)(2) from decision 1 would still be cut at the page break. The injected header now gives the orphaned half its identity, which softens the failure without removing it.

Failures degrade quietly. Three of the four nodes catch every exception and return empty output, so an audit can run with no transcript or no regulations and still produce a verdict. Nothing checks that a verbatim excerpt exists in the transcript, or that a timestamp falls inside the video. On-screen text comes back from Video Indexer without timestamps, so catching a disclaimer that flashes for two seconds was a goal and never a capability.

There are smaller things of the same kind. The download goes to one fixed filename, so two concurrent audits would overwrite each other's video. The Video Indexer polling loop has no deadline. If the agent is still calling tools at round ten, nothing forces a final answer and the JSON parse fails. And the dashboard was left behind by the four-node rewrite: demo mode still emits the old two steps, and a YouTube Shorts URL gets audited by the backend while the player reports an invalid URL.

The harder gap is scope. The top of Figure 4 shows the SEC investment-adviser marketing rule cited against a claim that a tool will help a video rank on YouTube. Whether a rule applies to the speaker at all is a question the system never asks, and neither do my test cases. No one with a legal or compliance background has reviewed the categories, the ground truths or any output yet.

What I would change. A saved results file for every experiment. Freeze the test set before tuning, add compliant cases, and use a different model as the judge. Score the real pipeline on a handful of hand-labelled videos. Rerank the retrieved set, not re-search it. Join pages before the structural split. Validate excerpts and timestamps in code, use schema-enforced output for the pipeline prompts as the judge already does, and make failures fail. Give each audit its own temp directory and a polling deadline.