Implementing an AI RAG for Bylaw Retrieval

I was building an AI chatbot for the bylaw search in Strata Aura , and the job sounded straightforward: someone asks a question, and the chatbot finds the clauses that apply.

The important word there is find. The project is supposed to retrieve the bylaws verbatim, not reinterpret them into a new rule. Once an AI model sits between the question and the source, however, “just find the clause” turns into a pipeline with enough moving parts to hide a mistake almost anywhere.

That sent me down the Retrieval Augmented Generation (RAG) rabbit hole. I learned how to split the bylaws, retrieve possible matches, rerank them, compose an answer, and then test whether the answer still pointed back to the sections it was supposed to find.

The pipeline was more than vector search

My first useful correction was understanding that RAG is not one search call. For this project, it became a sequence:

1. Chunk the bylaws into clear sections.

2. Turn those sections into semantic vectors so a question can match related language.

3. Rerank the retrieved chunks to put the strongest candidates first.

4. Use the prompt to interpret the request and present the selected clauses in a consistent format.

Each layer has a different responsibility. Chunking decides the unit I can retrieve. Semantic search finds language related to the question, even when the exact keywords differ. Reranking decides which candidates deserve to survive. The prompt shapes what the reader sees.

That separation is useful, but it also creates a debugging problem. If the final answer misses a clause, where did it disappear? It may never have been retrieved, it may have lost during reranking, or it may have reached the model and then been omitted from the response.

Tracing where a clause disappeared

With that many handoffs, reading the final response was not enough. I added Langfuse traces around the RAG pipeline so I could inspect what entered and left each stage.

That made the pipeline less of a black box. I could look at the original question, the chunks returned by retrieval, the set kept after filtering, and the citations returned with the composed answer. A polished answer could no longer hide a weak retrieval step, and a good retrieval set could not excuse an answer that dropped the evidence.

The trace did not decide whether a result was correct, though. For that, I needed known questions with known answers—a gold set—and metrics that measured retrieval separately from output.

Testing the retrieval

I split evaluation into two broad questions: did the pipeline find the intended sections, and did the answer use them correctly?

At the retrieval and filtering stage, I used three metrics:

• RetrievalCoverage is the fraction of gold sections present in the kept RAG set sent to Claude. It answers, “Did the evidence make it through retrieval?”

• ContextRecall uses an AI judge to score how well the kept passages support the reference answer.

• ContextPrecision uses an AI judge to decide whether the kept context was useful for reaching that reference answer.

RetrievalCoverage is deterministic because I can compare section identifiers directly. The context metrics need judgment because a passage can support an answer without matching a reference string word for word.

This distinction mattered. A response can sound right while the intended clause never reached the model. Measuring the kept set catches that problem before the prose has a chance to smooth it over.

Testing the answer and its citations

The output needed its own checks. Since the chatbot is meant to return source-backed bylaw information, citations are not decoration; they are the trail back to the actual rules.

I used three deterministic metrics at this stage:

• CitationGrounding is the fraction of {{cite}} markers that resolve to chunks retrieved for that turn. A score of 1 means the answer did not invent a citation.

• OutputCoverage is the fraction of gold sections represented by the returned citations.

• OutputPrecision is the fraction of unique cited sections that belong to the gold set. A score of 1 means there are no extra sections.

Coverage and precision pull in different directions. Citing one correct clause can be precise while still missing the rest of the answer. Citing every nearby section may improve coverage while adding clauses that do not belong. I needed both numbers to see that tradeoff instead of rolling it into one friendly-looking score.

Autoevals and a separate AI judge helped run these checks against the known questions and answers. The deterministic metrics handled facts I could calculate from section IDs and citation markers. The judge handled whether the retrieved context actually supported the reference answer.

Correct once was not correct enough

The next problem only showed up when I ran the same questions more than once. An answer could pass a single evaluation and then cite a different set of sections on the next run.

So I added a stability test that repeated the compose step while bypassing the cache. For those trials, I measured:

• Mean OutputCoverage across the repeated answers.

• Mean OutputPrecision across the repeated answers.

• Consistency, calculated as the mean pairwise Jaccard similarity between cited section sets.

For two trials, that last comparison is:

|cited sections A ∩ cited sections B| / |cited sections A ∪ cited sections B|

A value of 1 means the trials cited identical section sets. Lower values show that the answer is moving around, even if individual runs still look plausible.

This was the missing dimension in my first evaluation pass. Accuracy asks whether the answer found the right material. Stability asks whether it will keep finding that material when the same question comes back.

Final Thoughts

The biggest lesson from this build is that a RAG answer cannot be evaluated as one blob. Retrieval, filtering, composition, citation, and repeatability can each fail on their own, so each one needs evidence of its own.

Langfuse gave me the path through the pipeline. The gold questions and deterministic metrics told me whether the right bylaw sections survived that path. The AI judge covered the places where a section-ID comparison was not enough. Finally, the repeated uncached trials showed whether a passing answer was dependable or just a good roll of the dice.

For a chatbot that is supposed to find clauses rather than invent an interpretation, that separation is what I will keep. A fluent answer is nice, but the real result is knowing which source sections reached it, which ones came back out, and whether the pipeline can do the same thing again.