How Batch built the AI architecture behind AI assist

AI integrations in most CRM tools tend to disappoint: a free-text field, an API call to a model, the raw answer displayed on screen. A wrapper around a single LLM, with no orchestration and no quality control. It shows in the output. That's exactly what Batch's stack avoids.

What you actually see in Batch AI Assist, and why this generative AI is different

In Batch AI Assist, agents don't chat. They execute. Writing content variants, adjusting tone and emojis, translating a campaign into 30 languages, generating A/B/N tests, auto-naming the steps of an automation scenario.

The product's real value comes down to one principle: intelligence lives where the work happens. Not in a chat window next to your tool. Not in an external assistant you have to re-explain your context to every time. Right inside the interface, at the exact moment of action, with context already in place.

Assist comes in two forms. One-off agents, triggered with zero or one click, embedded exactly at the point of work. A public example: the Step Name Generator in the Automation Builder, which auto-generates a descriptive label for each step of a scenario based on its actual content. And an operator agent on the Intelligent Home page, which proactively pushes recommendations and alerts, with a chat interface for back-and-forth.

An external assistant or an MCP connection forces you to manually re-inject context an embedded agent already has. That's exactly what Assist's architecture avoids.

Why Batch built its own AI gateway

Here's why it matters: AI features ship fast across the whole product, and stay stable while the model landscape shifts every two weeks. That's the whole point of our gateway: the middle layer that connects every product agent to AI models, so no team has to talk directly to Anthropic, OpenAI, or Google.

Batch evaluated the gateways available on the market, then decided to build its own. The verdict: too expensive, and still needing integration with our own user and company system either way. Might as well start from a blank slate.

The in-house gateway rests on three building blocks.

A unified abstraction for product squads. One internal interface, modeled on a widely adopted industry standard (OpenAI's). Every squad builds against a single endpoint instead of learning each provider's API. When Anthropic, OpenAI, or Google ships a new model, teams have nothing new to learn.

Model name aliasing. Agents reference readable aliases, decoupled from versioned model IDs. We update the underlying model without breaking a single agent or touching application code. When a provider deprecates a model, we just change the alias in configuration. No migration project required.

Native governance. Usage limits set per user and per company, request logging, token-level cost tracking. This is infrastructure built to be governed from day one, designed to operate at the scale of hundreds of customers.

The right AI model for the right marketing use case

This is where the difference between "an AI button" and an agent that produces a publish-ready result without heavy review actually gets decided.

Every agent runs on a frontier model chosen for its specific use case. Anthropic, OpenAI, Google's Gemini: a mix runs in production. And each prompt is built around that specific model.

Swapping models isn't just a config change. A prompt optimized for one model loses performance the moment you switch it to another. Put the wrong model on the wrong use case, and the marketer feels the quality drop immediately.

Let's be direct about this: a vendor selling "bring your own API key, use any model" is implicitly admitting its agents are model-agnostic. Which usually means under-optimized. Choosing a model per use case is a quality commitment, stated plainly.

Measuring AI agent quality: how Batch tests and safeguards its results

An AI can impress in a single demo and disappoint in daily use. Systematic evaluation protects the second, not just the first.

LLMs are non-deterministic. Same input, different output. And that output shifts the moment you touch the model, its internal settings that steer creativity and precision, or the wording of the instructions you give it. Without a way to measure quality, there's no telling whether an update is improving the product or quietly degrading it.

Batch evaluates at three points.

1. At selection. We decide between models and prompts using a rigorous dataset, not by eyeballing two or three examples.

2. At every change over time. A provider ships a new model, deprecates an old one, or the team improves a prompt. We run a comparative evaluation to confirm it's an improvement, not a regression.

3. In production. We sample real execution traces and monitor live with alerts. Real users' inputs are never fully predictable.

This whole mechanism runs on a dedicated tool, Langfuse, which centralizes prompts, test sets, quality-check execution, response scoring, and tracking for every production call.

The method rests on four pillars.

A reference dataset of 100 to 1,000 representative, varied examples, pairing inputs with expected outputs. AI-generated, then human-checked, which cuts annotation time from months down to days. Every revision is tracked and versioned like code, with the ability to compare two versions or roll back.

Prompt versioning. Every prompt is versioned and tied to a model alias. Before promoting a change, we re-run an evaluation on both the old and new version. Full history stays available for an instant rollback if a regression makes it to production.

Deterministic code evaluators. Hard, computable checks: format compliance, required fields present, token parity.

Automated evaluators, run by a second model acting as reviewer. It scores each response in a sample against precise criteria and explains its reasoning, much like a grader justifying a mark. One important detail: this automated reviewer can run on a more powerful model than the one serving the feature. We serve on a fast, cost-efficient model, and review with a more capable one.

Generative AI in the CRM, a concrete example: email translation

Email translation is a good case study, because the stakes here are both qualitative and operational.

It's validated by a full battery of evaluators covering both.

On the automated review side: translation fidelity, target-language accuracy, subject line and preheader fidelity, entity preservation (brand names, products, personalization variables). Each dimension is scored 1 to 5 with reasoning: fluency and naturalness, terminology accuracy, register and tone match.

On the deterministic code side: subject line present, preheader present, token count, token parity. That last point is critical. Every personalization token in the source has to show up in the translation too. A translation that drops a token means a missing first name, repeated across thousands of emails sent to your contacts.

The automated reviewers cover meaning, the kind of thing a human proofreader would check. The code evaluators cover structure, the thing a campaign send actually depends on. A competitor's "translate with AI" button has none of these safeguards.

Your customer data never leaves your control, even with AI

You get frontier-model quality without handing your customers' data to a frontier-model provider.

Assist's generative layer never receives your customers' personal data. Agents work on technical metadata: campaign names, platform content. Never on end-customer personal data.

Two facts anchor this principle. Customer A's data never serves customer B. For Assist, Batch trains no model on customer data. Batch feeds context to the model to generate a response, for that same customer, and nothing else. For the technical metadata that does flow through, the providers are frontier players (Anthropic, OpenAI, Google), and Batch stays the data processor for that limited, non-personal data.

For contrast: Predict runs on a different foundation, machine learning trained per customer, operated by Batch on infrastructure Batch controls in Europe.

This principle answers the "no US model, full stop" objection on the merits. When no personal data ever transits, sovereignty concerns over the generative layer mostly dissolve.

What's next for AI in the CRM: sovereign inference

For organizations most sensitive to sovereignty, the goal is an agentic CRM where none of your activity depends on a US cloud.

Batch's philosophy: radical data privacy and sovereignty, on infrastructure Batch controls (bare metal). Today, no personal data leaves for the generative layer, so US frontier models are safe to use for the limited technical metadata involved.

The direction Batch is exploring: running open-source models on its own leased GPUs, in the same data centers where Batch already hosts its servers. In practice, that means using a European cloud infrastructure like OVHcloud to run the generative layer with zero dependency on US providers. So that even limited technical metadata stays on infrastructure Batch controls.

Data minimization is already in place today. Self-hosted sovereign inference is the next step, the one Batch is building toward.

Coming next: where does your CRM's AI context actually come from?

This article deliberately left one central question aside: how do agents actually get their context? They're grounded in years of CRM expertise, in each customer's platform content, in the structure of their CRM. But how does that context reach them?

That's a topic for a future article: retrieval, RAG, semantic engines, and how we keep that context grounded, precise, and private.

Capabilities we built ourselves. Rigor we measure. Trust by design. Sovereignty on the horizon. Anyone can bolt on an AI button. Few can stand behind what comes out of it.