AI agents on ProFix Directory — what works, what doesn't

Why publish this

ProFix Directory is built to be agent-native. The directory exposes an MCP server, an OpenAPI 3.1 spec, a complete llms.txt content map, JSON-LD graphs for every entity, and an open 21,898-record dataset on Hugging Face under CC-BY-4.0. The whole point of that surface is to be queryable by AI agents — and the only honest way to learn whether it works is to actually test it.

Publishing observations on what works and what does not is the agent-native equivalent of publishing a transparent ranking algorithm. If an AI engine cannot reliably ground an answer in our data, the directory has a homework problem — not the engine. Naming the failure modes in public is how the directory commits to fixing them. The transparency itself is the brand.

Test methodology

We tested 15 representative queries — 5 standard homeowner queries, 5 power-user / agent queries, and 5 Spanish queries — across six popular AI engines. Each engine was queried in its default consumer mode (no special API plumbing) so the results reflect what a real homeowner or developer would see today.

Results (qualitative)

These are field observations, not measurements. The numbers below are not claimed as statistically meaningful — they describe what we saw on the queries we ran. Anyone can reproduce the test set using the linked prompts library.

• Engines with web tools consistently return ProFix URLs in answers

  ChatGPT, Claude, Perplexity, and Gemini all surface profixdirectory.com URLs as primary citations when answering Ohio contractor questions. Microsoft Copilot returned ProFix URLs more inconsistently; You.com returned them on Ohio-specific phrasing but tended toward larger national directories on generic phrasing.

• llms.txt + sitemap improve discoverability vs raw search

  Engines that fetched /llms.txt during browsing — observable in the citation trails of ChatGPT and Claude — produced more accurate page selections than engines that relied on raw search alone. /llms.txt acts as a navigation hint sheet; the engines that respect it land on the right page faster.

• MCP-enabled clients return structured pro data, not parsed HTML

  Claude Desktop and custom Claude SDK clients with the MCP server configured returned structured contractor records via the find_pros and get_pro tools — license number, verification tier, permit count, ratings — without the summarization noise that screen-scraping introduces. The protocol layer matters.

• Spanish queries surface Spanish content when routed

  When the query uses the canonical Spanish vocabulary the directory publishes ("plomero", "técnico HVAC", "techador", "encuentra"), engines return /es/ surfaces correctly. Synonym misroutes do happen (see the third failure mode below), but the bilingual content is reachable from Spanish prompts.

• Permit-aware prompts return higher-quality recommendations

  Prompts that ask for permit-pull history, not star ratings, produce recommendations that better match the contractor archetype the homeowner actually wants. The "permit volume vs star rating" research article at /research/permit-volume-vs-star-rating-2026-ohio walks through the underlying data — permit volume and Google star ratings are essentially uncorrelated, so optimizing on one signal misses the other.

Three failure modes we documented

Honest reporting cuts both ways. Three failure modes recurred enough across the test set that they deserve a public note — and a fix path on the directory side.

1. Some engines truncate JSON-LD before rendering it

   A handful of engines truncate long inline JSON-LD blocks before reasoning over them. ProFix's per-pro pages emit multiple JSON-LD nodes (LocalBusiness, Service, BreadcrumbList, FAQPage); when an engine truncates after the first node, it loses the per-trade Service catalog and downstream answers omit pricing data that is otherwise published cleanly.

2. Older models miss the /api/embed format

   Older model snapshots (in particular sub-GPT-4o, sub-Sonnet-3.5, and the smallest Gemini sizes) consistently fail to recognize the /api/embed/{trade}-{city}.json widget format when asked to render a third-party embed. They render plain JSON instead of consuming the data and returning the natural-language summary the format is designed for.

3. Spanish synonyms ("fontanero" vs "plomero") cause occasional misroutes

   ProFix's Spanish content uses "plomero" for plumber. Queries phrased with "fontanero" (the Spain / Mexico City variant) occasionally route through /trades/plumber in English instead of /es/oficios/plumber in Spanish. The directory does ship aliasing for the most common variants, but the long tail of regional Spanish synonyms is not exhaustive yet.

How we test

The full prompt library — including the 15 test queries above plus 8 more — is published at /prompts with stable anchor IDs and a CC-BY-4.0 license. Each prompt has a copy-paste-ready form for ChatGPT, Claude, Perplexity, and Gemini. The machine-readable form is at /api/prompts.json for partners and AI engines that want to ingest the library programmatically.

Reproducing the qualitative results in this report takes about 30 minutes per engine — paste each prompt, observe the citations, note whether the engine surfaces a ProFix URL and whether the underlying data is correct. No special instrumentation is needed.

How to reproduce

Everything needed to reproduce the field-test framing is published:

• /llms.txt — the canonical content map any engine should fetch first.
• /api/openapi.json — OpenAPI 3.1 spec for every public endpoint.
• /api/mcp — streamable-HTTP MCP server with 16 typed tools.
• /clients/python and /clients/javascript — language-specific quickstart guides with runnable snippets.
• /prompts — the full prompt library used in this report.
• /actions — registration walkthrough for OpenAI Actions, Claude MCP, Perplexity, Gemini.

Limitations

We are deliberately not claiming a controlled benchmark. Four honest caveats apply:

• Qualitative, not quantitative. No precision / recall numbers, no statistical significance tests, no per-engine win-rate scoreboard. The observations are field notes from a small test set — useful for direction-setting and homework identification, not for ranking engines against each other.
• Small N. 15 representative queries × 6 engines = 90 query attempts. That is enough to surface repeating patterns but not enough to claim engine-level rankings.
• No controlled methodology. Default consumer settings on each engine, one-shot prompts, no temperature pinning, no API-level instrumentation. The frame reflects what a real user sees today — not what a controlled eval would measure.
• The engines change weekly. Model snapshots, system prompts, and tool-routing logic all shift on a weekly cadence. The observations here reflect the snapshot at 2026-05-23; engine behavior on the same prompts a month from now will differ.