How we measure what AI agents actually read
By Rafi ·
Key takeaways
This page covers: What each number means; The pipeline; Worked example: this site; Reproduce it; Limitations; What to do with this; About the author; Experience & expertise.
- Every SiftServe draft ships with capture stats: page weight in KB, token cost, readable-content yield, and a 1:1 coverage check, measured on both the original and the sifted page.
- Agents budget tokens, not kilobytes — a page that is cheap in KB can still be expensive in tokens, and the yield decides how much of your story an agent can hold.
- On SiftServe's own homepage, about 97% of what an agent downloads is delivery, not content; that ratio is where the ~90% fewer tokens per page figure comes from.
- Coverage below 100% fails review — sifting is translation, not summarization; a draft that drops a fact does not ship.
- You can reproduce the human-view measurement yourself with curl and ten lines of Python, shown below.
This post is for anyone evaluating those claims: a pilot customer reading their audit report, or a skeptic who wants to run the measurement on their own site. It covers what each number means, the measurement pipeline, a worked example, and the limitations; it does not cover the audit rubric itself, which has its own post.
Every draft in the SiftServe review desk ships with two things: an audit score and a set of capture stats. The audit score comes from the 80-item check we covered in the CORE-EEAT scoring methodology post. This post is about the other half: the capture stats (page weight, token cost, and readable-content yield) and how we measure them, so you can check our math instead of taking it on faith. These are also the numbers we quote about ourselves: ~90% fewer tokens per page, +37% more readable content, 114 KB → 29 KB on our own homepage.
The common assumption is that AI crawlers work like browsers — rendering JavaScript, loading images, seeing what a human visitor sees. The evidence says otherwise. No major AI crawler executes JavaScript (Vercel × MERJ analyzed 500M+ GPTBot fetches and found zero JS execution). Because most modern sites rely on client-side rendering, 57% of top sites show AI crawlers a nearly empty page (ModPageSpeed, May 2026). Measuring what an agent actually reads starts with fetching the way an agent fetches. Two common problems follow from this: facts locked inside images are invisible to a text crawler (the sifting step restates them as text), and content that renders client-side never reaches the measurement at all.
What each number means
Capture stats are the four measurements SiftServe reports for every page: page weight, token cost, readable-content yield, and coverage. Together they describe how much an AI agent can actually read of a page, and at what cost.
- Page weight (KB)
- The raw HTML response an AI agent downloads on one fetch, measured before and after sifting. No images and no JavaScript execution — what renders client-side never reaches this measurement.
- Token cost
- What that document costs against an agent's context budget once tokenized. Agents budget tokens, not kilobytes; a page that is cheap in KB can still be expensive in tokens.
- Readable-content yield
- The share of the document that is content an agent can use (body copy, facts, headings, structured data) rather than markup, class names, scripts, and chrome.
- Coverage
- The 1:1 trace check. Every claim in the original must be present in the sifted copy; 100% coverage means nothing was summarized away, and the audit fails a draft that drops a fact.
The pipeline
- Fetch like an agent. One raw HTTP GET with a crawler user-agent. No JavaScript, no rendering: the document GPTBot or ClaudeBot receives.
- Extract what's readable. Strip <head>, scripts, styles, and inline SVG; collapse the markup; keep text and structured data. This mirrors what a non-rendering reader can use.
- Count. Document bytes, extracted-text bytes, and tokens for both.
- Sift, then measure again. The same pipeline runs on the sifted document: semantic HTML, FAQs, structured data, and facts that were locked inside images restated as text.
- Audit both. The original and the sifted copy each get the full 80-item CORE-EEAT check. That pair of scores (like the 33 → 44 on our own homepage) is what the review desk shows.
Worked example: this site
| Measurement | Original page | Sifted page (v10) |
|---|---|---|
| Page weight | 368 KB | 32 KB (−91%) |
| Readable text (live, 2026-08-26) | 11,957 bytes · 1,885 words | 16,787 bytes · 2,700 words |
| Readable-content yield | 3.2% | ~51% |
| Coverage | — | 96% (3 items flagged) |
| CORE-EEAT audit score | 65 | 76 |
We sift our own site, so it is the example we can show end to end. The numbers below are the homepage's published capture stats — version 10, the sift currently serving AI traffic — plus a live measurement from 2026-08-26 that you can re-run yourself with the snippet in the next section.
On this measurement: the original page weighs 368 KB; the sifted page (v10) is 32 KB (−91%). Readable text (live, 2026-08-26) runs to 11,957 bytes · 1,885 words on the original and 16,787 bytes · 2,700 words on the sifted page. Readable-content yield is 3.2% on the original and ~51% on the sifted page. The CORE-EEAT audit score moves from 65 (original) to 76 (sifted, v10).
The read, in tokens: the original document is 376,286 bytes. At the ≈4-characters-per-token estimate (OpenAI's published rule of thumb for English text), that is roughly 94,000 tokens of document for roughly 3,000 tokens of readable text. We measured and found that about 97% of what an AI agent downloads here is delivery, not content. And this is an already-clean, statically rendered page; it weighed 114 KB at draft v6 and grew to 368 KB after a mobile-performance pass inlined its CSS, which speeds up the page for human visitors and is pure overhead for a text crawler. The sifted copy hands the same agent about 8,300 tokens, roughly half of it readable content, and it carries more readable text than the original (16,787 vs 11,957 bytes, +40% on this measurement) because sifting restates facts locked inside images and adds FAQs and structured data.
What 96% coverage looks like in practice: three source items were not carried into v10 — a section heading and two near-duplicate FAQ entries that review deduplicated. Each one is listed by name in the review desk, and publishing meant signing off on exactly that list.
That ratio is the mechanism behind the headline numbers. Serving an agent just the content, instead of the whole delivery vehicle, is where ~90% fewer tokens per page comes from (−91% on this page). Restoring what the original hid from non-rendering readers (facts inside images, FAQs, structured data) is where +37% more readable content comes from; today's measurement reads +40%, and we quote the conservative figure. The per-page numbers behind both ship in every pilot's audit report.
Reproduce it
The human-view measurement needs nothing but curl and Python:
curl -s https://siftserve.com/ | wc -ccurl -s https://siftserve.com/ | python3 -c "
import re, sys
h = sys.stdin.read()
h = re.sub(r'<head[\s\S]*?</head>|<script[\s\S]*?</script>|<style[\s\S]*?</style>|<svg[\s\S]*?</svg>', ' ', h)
t = re.sub(r'<[^>]+>', ' ', h)
t = re.sub(r'\s+', ' ', t).strip()
print(len(t.encode()), 'bytes of readable text,', len(t.split()), 'words')"Point it at your own homepage and you have your own content yield in under a minute.
Limitations
- Kilobytes are not tokens, and tokenizers differ by model. We publish which tokenizer we count with; the ≈4-characters-per-token figure above is an estimate, not an exact count.
- Readable-content extraction is heuristic. The rules are simple and shown above so you can argue with them.
- Capture stats are per-version snapshots. Pages change; our own homepage weighed 114 KB at draft v6 and about 121 KB today. Each new sift re-measures. (Note: the 368 KB figure in the worked example reflects the homepage after a mobile-performance pass inlined its CSS — a change that occurred between the article's writing date and the 2026-08-26 live measurement. Different versions, different weights.)
- The audit score is rubric-based (80 items, weighted by content type), not a model's overall vibe.
- The measurement method documents what a text-crawling agent receives via a raw HTTP GET — pages that generate content entirely through client-side JavaScript will appear nearly empty under this methodology, just as they do to AI crawlers.
What to do with this
The capture stats turn "AI-readable" into numbers you can check. If you run the snippet on your own homepage and the yield comes back under 10%, most of what an agent downloads from you is markup. That is the gap sifting closes, and the what-is-SiftServe guide explains how. To see the full measurement run on your site, become a research partner.
About the author
Rafi is an Engineering Leader turned founder with 17+ years of experience architecting large-scale data and AI-driven systems, with a strong foundation in machine learning, personalization systems, and data platform engineering.