How to test ads in your AI app without hurting retention
Test ads in an AI app the way you would test any product change: put them behind a feature flag, keep a no-ad holdout, and compare revenue per thousand prompts against 7-day retention. Fill rate tells you whether the integration works, and on its own it says nothing about whether ads belong in your product.
Most builders who hold off on monetization are worried about losing the users they spent months acquiring. A controlled test answers that with your own data in a few weeks. This guide assumes the Adgentek Web SDK is already in your app. If it is not, point your AI coding tool at docs.adgentek.ai and the quickstart gets an ad rendering; the full page index for coding agents is at docs.adgentek.ai/llms.txt. Everything below uses documented features of the Adgentek Ad Server plus code you already own.
What does test mode do in an AI app?
Test mode confirms that your integration renders and tracks correctly on a real placement, and it earns nothing while it runs. You switch it on per placement in the dashboard, and from then on that placement serves only Adgentek test ads. They come from Adgentek's own test brand, with copy that says so, and they travel the same path as live ads: same request, same response shape, same impression pixel, same click URL.
Delivery metrics report as usual. Revenue and eCPM read zero on purpose. Allow about five minutes for the switch to reach the ad server, and know that test creatives carry no badge or watermark, so the copy and the zero revenue are how you recognize them.
That makes test mode the right tool for two checks: the impression pixel fires from the user's browser, and the click URL opens correctly from your surface. It cannot tell you what real demand pays or how your users react to real advertisers. If you are still building, ask Adgentek for a dedicated test placement so you never pull an earning one out of service. When the checks pass, ask Adgentek to take the placement live. Your code does not change.
How do I A/B test ads in an AI app?
You A/B test ads in an AI app by assigning each user to an ad group or a no-ad holdout behind a feature flag, and calling the SDK only for users in the ad group. The holdout never sends an ad request, so Adgentek does not need to know it exists. The comparison happens in your analytics.
Assign at the user level. Assigning per session or per request mixes the groups, because the same person sees ads on Tuesday and none on Wednesday, and their retention then belongs to neither group. Hash a stable user ID into the flag and record the group as a user property in your product analytics, since that is where retention gets measured.
New users make the cleanest read. Someone who has used your app daily for six months will tolerate an ad that a person on their second day will not, and new-user cohorts give you a like-for-like 7-day retention comparison. An even split reads fastest. A smaller ad group limits your exposure if ads do hurt retention, and the test takes longer to reach a clear answer.
// Group assignment lives in your own feature flag tool
const inAdGroup = flags.isEnabled('ads_pilot', user.id);
async function afterResponse(userMessage, aiResponse) {
if (!inAdGroup) return; // holdout: no ad request at all
await AdgentekAds.showAd(
[
{ role: 'user', content: userMessage },
{ role: 'assistant', content: aiResponse }
],
{ custom: { ad_test: 'every_response' } }
);
}Run the test until every enrolled cohort has finished its full 7-day window. In practice that means two weeks of enrollment and one more week of follow-up, so weekday and weekend usage both land in each group.
What metrics matter for AI app ads?
Two metrics decide an AI app ad test: revenue per thousand prompts in the ad group, and the 7-day retention gap between the ad group and the holdout. Everything else is a health check.
Revenue per thousand prompts is the ad group's revenue divided by the total prompts its users sent, multiplied by 1,000. It differs from eCPM, which divides by impressions and climbs when you serve fewer, more selective ads. Dividing by prompts captures the real tradeoff, which is how much each unit of usage earns once you account for every turn that carried no ad.
| Metric | Where it comes from | What it answers |
|---|---|---|
| Revenue per thousand prompts | Adgentek revenue, your prompt count | Does the ad group earn enough per unit of usage? |
| 7-day retention by group | Your product analytics | Do ads cost you users? |
| Prompts per active user | Your product analytics | Are people using the app less while ads run? |
| Fill rate | Adgentek reporting | Is the integration requesting and rendering ads? |
| eCPM and CTR | Adgentek reporting | What does each served ad earn, and do people engage with it? |
| IVT Rate | Adgentek reporting | Is any of the ad traffic invalid? |
Expect a low fill rate and do not chase it. The Ad Server classifies each turn by intent and returns no ad on support questions and low-confidence turns, so a support-heavy app should see many empty responses. Prompts per active user is worth watching daily because it can move before retention does.
Two reporting details matter when you read results. Revenue in Adgentek reporting is net publisher revenue, which is the number to set against your inference bill. Reporting times are UTC, and the dashboard publishes no freshness SLA, so let the last few days reconcile before you call the result.
How often should an AI app show an ad?
An AI app should show an ad only on turns where the person has expressed commercial intent, and the right frequency is something you test rather than guess. The Adgentek Ad Server sorts every turn into one of nine intent buckets, and that bucket decides which demand is eligible and which format renders. Read how conversational ads work for what the person sees when one does.
You have controls on both sides of the integration. In the dashboard, under Inventory, Ad Policies, a policy can set a Minimum Relevance Threshold, a Minimum Conversation Turn, per-intent rules that block or down-price low-scoring matches, and brand safety filters. In your code, you decide which turns call showAd() at all. Pass a stable sessionId in the options if you want frequency capping to follow a conversation across reloads; otherwise the SDK generates one per browser tab. Image and video generators use a wait-state surface type with no conversation turns, so their minimum turn stays at 0.
There are two documented ways to build frequency variants:
- Code variants: one placement, different calling rules per group (every response against every third response, say), each tagged with a custom parameter value.
- Policy variants: two AI Surfaces with different ad policies attached, each with its own Placement ID. Your flag decides which Placement ID a user's SDK is initialized with, and reporting splits by Surface Name with no extra setup.
Policy variants are also how to test brand safety settings. The Strict preset blocks every category in the sensitive-category registry and can shrink eligible demand a lot, so test it against Standard on your own traffic before committing. Brand safety in AI advertising covers what each level is for.
How to tag ad test variants in Adgentek reporting
Tag every ad request in a test with a custom parameter, and Adgentek stores that value on every event the request produces, so reporting breaks down by variant. Pass custom once at init when the value is fixed for the page, or per showAd() call when it varies. A per-call value overrides the init value. getAd() takes the same option.
Register the key before launch. Sending a value stores it, but a key only becomes a dimension you can group by in the Custom Report Builder after you register it on the Custom Parameters screen in publisher settings. Each organization gets up to 10 indexed keys. The builder takes up to three breakdown keys and a daily time bucket, which is enough to read variant against date.
The rules are short and strict:
- Custom parameters are reporting only. They never change which ad wins or what it pays, so tagging cannot bias your test.
- Pick the key name once. Renaming it later splits your history into two dimensions that cannot be combined.
- Keep values bounded. A handful of variant names works, and a per-user or per-request value produces an unusable dimension.
- Never send personal data. No emails, phone numbers, or user IDs, because the values are stored on every event and shown in reporting.
Use lowercase snake_case names, such as ad_test, so they stay consistent across surfaces.
How to decide whether to keep ads in your AI app
Keep ads on when revenue per thousand prompts covers enough of your inference cost to matter and 7-day retention stays within the margin you set before the test began. Write both thresholds down at the start. A team that picks its bar after seeing the data will always find a reason to ship.
| Result | What to do next |
|---|---|
| Revenue worth it, retention flat | Roll out to everyone and keep a small permanent holdout |
| Revenue worth it, retention down | Raise the relevance threshold or minimum turn, then rerun |
| Revenue thin, retention flat | Check fill by intent bucket and the formats enabled on the surface |
| Revenue thin, retention down | Turn ads off on that surface and revisit placement |
A permanent holdout is cheap insurance. Demand, formats, and your own product all change, and a small group that never sees ads keeps the retention comparison honest months after launch.
If revenue is thin because text units draw little engagement, test a richer format. Spark, the conversational ad format, is available now on AI surfaces running the Adgentek Ad Server. It is an interactive Q&A unit inside the conversation, and Spark sessions average 3.2 interactions. Spark shows how the unit works, and the industry examples show finished placements. The wider case for ad revenue covering inference cost is on the publisher monetization page.
Getting started
Adgentek gives AI app builders the controls to run this test on live demand: intent-based serving rules you set per surface, and custom parameters that split reporting by variant. Point your AI coding tool at docs.adgentek.ai, put a placement through test mode, then start the holdout. To take a placement live or add Spark to your surface, email hello@adgentek.ai.
