How we measure what AI agents actually read

Key takeaways

This page covers: What each number means; The pipeline; Worked example: this site; What this challenges; Reproduce it; Limitations; About the author; What to do with this.

• Every SiftServe draft ships with capture stats: page weight in KB, token cost, readable-content yield, and a 1:1 coverage check, measured on both the original and the sifted page.
• This post is for anyone evaluating those claims: a pilot customer reading their audit report, or a skeptic who wants to run the measurement on their own site.
• Agents budget tokens, not kilobytes. A page that is cheap in KB can still be expensive in tokens, and the yield decides how much of your story an agent can hold.
• On our own homepage, about 97% of what an agent downloads is delivery, not content. That ratio is where the ~90% fewer tokens per page figure comes from.
• Coverage below 100% fails review. Sifting is translation, not summarization; a draft that drops a fact does not ship.
• You can reproduce the human-view measurement yourself with curl and ten lines of Python, shown below.

What each number means

Every draft in the SiftServe review desk ships with two things: an audit score and a set of capture stats. The audit score comes from the 80-item check covered in the CORE-EEAT scoring methodology post [F11]: "Every page is checked twice using the same 80-item CORE-EEAT standard." This post is about the other half: the capture stats (page weight, token cost, and readable-content yield) and how they're measured, so customers can check the math instead of taking it on faith. These are also the numbers quoted about SiftServe itself: ~90% fewer tokens per page, +37% more readable content, 114 KB → 29 KB on the homepage. It covers what each number means, the measurement pipeline, a worked example, and the limitations; it does not cover the audit rubric itself, which has its own post.

This page is for pilot customers checking their audit report and for skeptics who want to run the measurement themselves. It covers the four capture-stats measurements and the pipeline that produces them — it does not cover the CORE-EEAT audit rubric, which lives in its own post.

Capture stats are the four measurements SiftServe reports for every page: page weight, token cost, readable-content yield, and coverage. Together they describe how much an AI agent can actually read of a page, and at what cost.

• Page weight (KB): the raw HTML response an AI agent downloads on one fetch, measured before and after sifting. No images and no JavaScript execution, because no major AI crawler executes JavaScript (Vercel × MERJ analyzed 500M+ GPTBot fetches and found zero JS execution). This is also why 57% of top sites show AI crawlers a nearly empty page (ModPageSpeed, May 2026): what renders client-side never reaches the measurement.
• Token cost: what that document costs against an agent's context budget once tokenized. Agents budget tokens, not kilobytes; a page that is cheap in KB can still be expensive in tokens.
• Readable-content yield: the share of the document that is content an agent can use (body copy, facts, headings, structured data) rather than markup, class names, scripts, and chrome.
• Coverage: the 1:1 trace check. Every claim in the original must be present in the sifted copy; 100% coverage means nothing was summarized away, and the audit flags a draft that drops a fact. [F28] "Coverage: the 1:1 trace check. Every claim in the original must be present in the sifted copy; 100% coverage means nothing was summarized away, and the audit fails a draft that drops a fact."

The pipeline

We sift our own site, so it is the example we can show end to end. We tested the homepage measurement across multiple versions and found what the limitations below describe: capture stats are per-version snapshots, not one fixed number — the homepage weighed 114 KB at draft v6, 368 KB at the version measured for the v10 audit, and about 121 KB at a later live fetch. Each new sift re-measures.

1. Fetch like an agent. One raw HTTP GET with a crawler user-agent. No JavaScript, no rendering: the document GPTBot or ClaudeBot receives.
2. Extract what's readable. Strip <head>, scripts, styles, and inline SVG; collapse the markup; keep text and structured data. This mirrors what a non-rendering reader can use.
3. Count. Document bytes, extracted-text bytes, and tokens for both.
4. Sift, then measure again. The same pipeline runs on the sifted document: semantic HTML, FAQs, structured data, and facts that were locked inside images restated as text.
5. Audit both. The original and the sifted copy each get the full 80-item CORE-EEAT check. That pair of scores (like the 33 → 44 on our own homepage) is what the review desk shows.

Problems encountered

Problem 1: client-rendered content never reached the measurement. No major AI crawler executes JavaScript, so what renders client-side never reaches the measurement; this is also why 57% of top sites show AI crawlers a nearly empty page. The fix built into the pipeline: fetch like an agent, with one raw HTTP GET and a crawler user-agent, no JavaScript and no rendering, so the measured document is the document GPTBot or ClaudeBot actually receives.

Problem 2: facts locked inside images were missed. The fetch measures the HTML document, which is all a text crawler receives — facts that exist only inside images are invisible to it. The fix: sifting restates them as text, which is part of where the +37% readable-content gain comes from. The result on our own homepage: the sifted copy carries more readable text than the original (16,787 vs 11,957 bytes, +40% on this measurement) because sifting restates facts locked inside images and adds FAQs and structured data.

Worked example: this site

We sift our own site, so it is the example we can show end to end. The numbers below are the homepage's published capture stats — version 10, the sift currently serving AI traffic — plus a live measurement from 2026-08-26 that you can re-run yourself with the snippet in the next section.

Page weight: 368 KB on the original page, 32 KB on the sifted page (v10), a 91% reduction. Readable text, measured live on 2026-08-26: 11,957 bytes and 1,885 words on the original, 16,787 bytes and 2,700 words on the sifted page. Readable-content yield: 3.2% on the original, ~51% on the sifted page. Coverage: 96% (3 items flagged) on the sifted page. CORE-EEAT audit score: 65 on the original, 76 on the sifted page.

And this is an already-clean, statically rendered page; it weighed 114 KB at draft v6 and grew to 368 KB after a mobile-performance pass inlined its CSS, which speeds up the page for human visitors and is pure overhead for a text crawler. The sifted copy hands the same agent about 8,300 tokens, roughly half of it readable content, and it carries more readable text than the original (16,787 vs 11,957 bytes, +40% on this measurement) because sifting restates facts locked inside images and adds FAQs and structured data.

What 96% coverage looks like in practice: three source items were not carried into v10 — a section heading and two near-duplicate FAQ entries that review deduplicated. Each one is listed by name in the review desk, and publishing meant signing off on exactly that list. Coverage below 100% fails review. Sifting is translation, not summarization; a draft that drops a fact does not ship — the v10 page published at 96% because the three flagged items were named individually and signed off on, not because the 100% standard was waived generally.

That ratio is the mechanism behind the headline numbers. Serving an agent just the content, instead of the whole delivery vehicle, is where ~90% fewer tokens per page comes from (−91% on this page). Restoring what the original hid from non-rendering readers (facts inside images, FAQs, structured data) is where +37% more readable content comes from; today's measurement reads +40%, and we quote the conservative figure. The per-page numbers behind both ship in every pilot's audit report.

What this challenges

The assumption worth challenging: if a page renders fine in a browser, an AI crawler reads it the same way. It does not — no major AI crawler executes JavaScript (Vercel × MERJ analyzed 500M+ GPTBot fetches and found zero JS execution). This is also why 57% of top sites show AI crawlers a nearly empty page (ModPageSpeed, May 2026): what renders client-side never reaches the measurement. On our own homepage — an already-clean, statically rendered page — about 97% of what an agent downloads is delivery, not content.

Reproduce it

The human-view measurement needs nothing but curl and Python:

curl -s https://siftserve.com/ | wc -c

curl -s https://siftserve.com/ | python3 -c "import re, sys; h = sys.stdin.read(); h = re.sub(r'<head[\s\S]*?</head>|<script[\s\S]*?</script>|<style[\s\S]*?</style>|<svg[\s\S]*?</svg>', ' ', h); t = re.sub(r'<[^>]+>', ' ', h); t = re.sub(r'\s+', ' ', t).strip(); print(len(t.encode()), 'bytes of readable text,', len(t.split()), 'words')"

Point it at your own homepage and you have your own content yield in under a minute.

Limitations

We only tested this pipeline — a raw, non-rendering HTTP GET with a crawler user-agent — run against our own homepage across draft versions v6 and v10 plus one later live fetch; we did not test browser rendering or every page type on the site.

• Kilobytes are not tokens, and tokenizers differ by model. We publish which tokenizer we count with; the ≈4-characters-per-token figure above is an estimate, not an exact count.
• Readable-content extraction is heuristic. The rules are simple and shown above so you can argue with them.
• Capture stats are per-version snapshots. Pages change; our own homepage weighed 114 KB at draft v6 and about 121 KB today. Each new sift re-measures.
• The audit score is rubric-based (80 items, weighted by content type), not a model's overall vibe.

About the author

This post is written by Rafi, a SiftServe founder with more than 17 years of experience architecting large-scale data and AI-driven systems, with a strong foundation in machine learning, personalization systems, and data platform engineering. Questions about this post can be sent to info@siftserve.com.

What to do with this

The capture stats turn "AI-readable" into numbers you can check. If you run the snippet on your own homepage and the yield comes back under 10%, most of what an agent downloads from you is markup. That is the gap sifting closes, and the what-is-SiftServe guide explains how. To see the full measurement run on your site, become a research partner.