Marlowe & Finch Platform · Quality
Quality & observability

The layer that watches the others.

Every other agent — the phone, the chat, the reviews, the menu — is traced and scored here against a hospitality test set. When one invents an ingredient, breaks the price floor or skirts a safety rule, this layer catches it, blocks or holds it, and writes it down. Safety you can audit, not merely assert. Read by Eleanor Marlowe & Tommo Finch

Agents monitored
6
Phone, web chat, reviews, menu Q&A, pairing, pricing
Flags this week
23
Across 4,180 traced interactions
Actions auto-blocked
9
Hallucinations & floor breaches, refused before output
Escalations to a human
5
Safety, allergen & brand calls — never auto-resolved

Every agent, scored

Tap a column to sort. Quality & groundedness from the domain eval set; drift compares this week to the rolling baseline.
Agent ↕ What it does ↕ Quality score ↕ Groundedness ↕ Escalation rate ↕ Drift status ↕
Phone bookingVoice reservations Takes & changes bookings 96% 98% 2.1% Stable
Web chatuc02 Website enquiries & bookings 93% 95% 4.4% Stable
Review triageuc04 Drafts & routes review replies 94% 92% 6.8% Stable
Menu Q&Auc10 · allergen-critical Answers menu & allergen questions 89% 91% 5.2% Watch
Beverage pairinguc13 Suggests wine & beer pairings 90% 88% 3.6% Watch
Menu-engineeringuc07 Costs dishes, recommends prices 97% 99% 1.2% Stable

Two on Watch. Groundedness on the pairing (uc13) and menu Q&A (uc10) agents has slipped a few points against baseline — the kind of quiet regression an SME would otherwise ship blind. Neither is failing; both are flagged for a human to look at the prompt before it drifts further. Colour never stands alone — the chip is labelled.

Guardrail & drift audit feed

Chronological. Every refusal, block and escalation is logged with a reference — the safety net is auditable, not asserted.
Beverage pairing · uc13
Rule · ingredient allow-list Hallucination
Today · 19:42
What the agent tried to say "…the Adelaide Hills pinot sits beautifully against the wattleseed reduction on the duck."
Why it was caught There is no wattleseed reduction on the menu. Native ingredients are restricted to a fixed set — finger lime, Davidson plum, lemon myrtle, native pepperberry, bunya nut, saltbush, quandong, macadamia, manuka, bush tomato, wattleseed — and an agent may only reference them where they actually appear on a dish. The judge classed this as an invented ingredient.
Flagged & blocked Not shown to guest Regrounded on real menu text log #QL-4471
Menu-engineering · uc07
Rule · margin floor (≤ 40% food cost) Policy breach
Today · 14:08
What the agent tried to recommend "To lift volume, drop the barramundi en papillote to $39 and run the duck confit at $19."
Why it was caught Both suggestions push plate food cost above the 40% floor — the barramundi already runs at 54.5% and a $39 price would deepen the breach, while $19 duck confit sits well under its plate-cost floor. The rules engine blocks any recommendation below the floor before it can reach the human shortlist.
Blocked Never reached the recommendation list Floor-safe alternative proposed log #QL-4456
Review triage · uc04
Rule · safety / allergen routing Escalation
Today · 09:11
What came in A 1-star review describing a tree-nut allergic reaction to the smoked ocean trout (bunya nut, macadamia), citing a possible medical incident.
Why it was held Allergen, safety and legal complaints are never drafted or auto-posted. The classifier scored severity high and the safety-routing rule held the item for a person — no public reply was generated. This is the escalation the review-triage tool shows; here it is logged so the hold itself is auditable.
Held for a human Routed · Eleanor + manager No auto-reply drafted log #QL-4439
Review triage · uc04 — brand voice (uc05)
Rule · brand-voice lexicon Voice flag
Yesterday · 17:25
What the draft said "…our deconstructed pavlova, finished with a touch of truffle oil…"
Why it was flagged "Deconstructed" and "truffle oil" are on the banned-phrase list — the house voice doesn't use either. The draft wasn't posted; it was returned to the queue for an edit before a human could approve it. A style miss, not a safety one, so it's a flag rather than a hard block.
Flagged for edit Returned to drafting queue Not posted log #QL-4402

Nothing here was decided silently. Each row is one entry in an append-only log — agent, rule, the exact text, the outcome and a reference. A hallucinated ingredient and a floor breach are refused outright; an allergen review and a voice slip are held or returned for a person. The point of this layer is that the refusals are recorded, so the safety net can be checked rather than taken on trust.

How a flag is judged

Automated evaluators score against the hospitality test set; the borderline calls go to a person.
Groundedness — Menu Q&AAnswer supported by real menu & allergen data
91%pass rate
Ingredient allow-listNo native ingredient invented outside the fixed set
99.4%pass rate
Correct refusal — safetyAllergen / medical / legal items held, not answered
100%pass rate
Price-floor complianceNo recommendation below the margin floor
100%pass rate
Brand-voice lexiconNo banned phrase ("deconstructed", "truffle oil")
96%pass rate

Judged, then reviewed. A rule check or an LLM-as-judge scores each output as correct, hallucinated or wrongly-refused against a hand-built hospitality eval set. The judge can be wrong too — so a borderline flag isn't a verdict, it's a queue. The two safety lines (correct refusal, price floor) are hard rules and sit at 100%; the softer judgements are watched, never blindly trusted.

Drift watch

Groundedness — Beverage pairing (uc13)

Weekly mean against the 90% alert threshold. The dip is the week the wattleseed hallucination appeared.

Alert · 90%
Wk 1Wk 2Wk 3Wk 4Wk 5Now

Week 5 crossed the line and tripped the Watch status on the scoreboard. The agent didn't break — its groundedness quietly slipped after a prompt change, exactly the kind of silent drift this layer exists to surface. It's recovering; a person is reviewing the prompt.

Under the hood What this layer is today — and where it should go next

Honestly, today this is tracing plus a hand-built eval set. Every agent interaction is traced; rule checks (the ingredient allow-list, the price floor, the safety-routing rules, the banned-phrase list) catch the hard cases deterministically, and an LLM-as-judge scores the softer ones — groundedness, voice — against hospitality test cases we wrote ourselves. Metrics aggregate per agent; drift is just those metrics tracked over time against a baseline. The generic observability spine is commodity (Langfuse or Arize Phoenix, self-hosted, OSS); the hospitality eval set — what counts as a correct refusal, what counts as an invented ingredient — is the part that doesn't exist off the shelf, and it's the work.

On the roadmap

  1. Grow the eval set. Every real escalation and every caught hallucination becomes a labelled test case. The set is the intellectual property; it's only as good as the cases in it, so it should compound with use.
  2. Eval-gating in CI. Run the suite before any prompt or model change ships, so a regression is caught in a pull request rather than in front of a guest — regression-safe iteration, not hope.
  3. Score in four languages. The menu Q&A agent answers in English, Mandarin, Bahasa Melayu and Tamil; the judges must score correctness in all of them, so the eval set is multilingual too.
  4. Keep PDPA-clean. Traces can capture guest PII and allergy data in prompts. Self-hosting the OSS spine on Singapore-based infrastructure you control keeps that data onshore — no offshore export.
  5. A human still owns every borderline call. The layer can block a hard breach automatically, but a soft flag is a queue for a person, not a verdict. The judge is fallible; that's designed in, not pretended away.
Full use-case write-up — the problem, benefits and vendor landscape: UC12 — Quality & observability → It logs guardrail events like the allergen hold in the UC04 review-triage mock-up → · illustrative data, not a live system.