How we measure what AI agents actually read

Every draft in the SiftServe review desk ships with two things: an audit score and a set of capture stats. The audit score comes from the 80-item check we covered in the CORE-EEAT scoring methodology post. This post is about the other half: the capture stats (page weight, token cost, and readable-content yield) and how we measure them, so you can check our math instead of taking it on faith.

These are also the numbers we quote about ourselves: ~90% fewer tokens per page, +37% more readable content, 114 KB → 29 KB on our own homepage.

What each number means

Page weight (KB): the raw HTML response an agent downloads on one fetch, measured before and after sifting. No images and no JavaScript execution, because no major AI crawler executes JavaScript (Vercel × MERJ analyzed 500M+ GPTBot fetches and found zero JS execution).
Token cost: what that document costs against an agent’s context budget once tokenized. Agents budget tokens, not kilobytes; a page that is cheap in KB can still be expensive in tokens.
Readable-content yield: the share of the document that is content an agent can use (body copy, facts, headings, structured data) rather than markup, class names, scripts, and chrome.
Coverage: the 1:1 trace check. Every claim in the original must be present in the sifted copy; 100% coverage means nothing was summarized away, and the audit fails a draft that drops a fact.

The pipeline

Fetch like an agent. One raw HTTP GET with a crawler user-agent. No JavaScript, no rendering: the document GPTBot or ClaudeBot receives.
Extract what’s readable. Strip <head>, scripts, styles, and inline SVG; collapse the markup; keep text and structured data. This mirrors what a non-rendering reader can use.
Count. Document bytes, extracted-text bytes, and tokens for both.
Sift, then measure again. The same pipeline runs on the sifted document: semantic HTML, FAQs, structured data, and facts that were locked inside images restated as text.
Audit both. The original and the sifted copy each get the full 80-item CORE-EEAT check described above. That pair of scores (like the 33 → 44 on our own homepage) is what the review desk shows.

Worked example: this site

We sift our own site, so it is the example we can show end to end.

Review-desk capture (draft v6 of the homepage): 114 KB original → 29 KB sifted at 100% coverage, a 75% cut in page weight with every claim retained. The audit score moved 33 → 44.
Live measurement (2026-08-04): the homepage document is 123,875 bytes (about 121 KB; the page has grown since v6). Readable text extracted by the pipeline above: 7,991 bytes and 1,223 words, a 6.5% content yield. At the standard ≈4-characters-per-token estimate, that is roughly 31,000 tokens of document for roughly 2,000 tokens of text. About 93% of what an agent downloads here is delivery, not content. And this is an already-clean, statically rendered page with no client-side rendering to lose.

That ratio is the mechanism behind the headline numbers. Serving an agent just the content, instead of the whole delivery vehicle, is where ~90% fewer tokens per page comes from. Restoring what the original hid from non-rendering readers (facts inside images, FAQs, structured data) is where +37% more readable content comes from. The per-page numbers behind both figures ship in every pilot’s audit report.

Reproduce it

The human-view measurement needs nothing but curl and Python:

			
curl -s https://siftserve.com/ | wc -c
curl -s https://siftserve.com/ | python3 -c "
import re, sys
h = sys.stdin.read()
h = re.sub(r'<head[\s\S]*?</head>|<script[\s\S]*?</script>|<style[\s\S]*?</style>|<svg[\s\S]*?</svg>', ' ', h)
t = re.sub(r'<[^>]+>', ' ', h)
t = re.sub(r'\s+', ' ', t).strip()
print(len(t.encode()), 'bytes of readable text,', len(t.split()), 'words')"

		

Limitations

Kilobytes are not tokens, and tokenizers differ by model. We publish which tokenizer we count with; the ≈4-characters-per-token figure above is an estimate.
Readable-content extraction is heuristic. The rules are simple and shown above so you can argue with them.
Capture stats are per-version snapshots. Pages change; our own homepage weighed 114 KB at draft v6 and about 121 KB today. Each new sift re-measures.
The audit score is rubric-based (80 items, weighted by content type as described above), not a model’s overall vibe.

Quick answers

What does 100% coverage mean? Every claim in the original page exists in the sifted copy, traced 1:1. Coverage below 100% fails review. Sifting is translation, not summarization.

Why count tokens instead of kilobytes? Agents work inside context budgets denominated in tokens. Two pages of equal weight can cost very different token amounts depending on how much of the document is markup. The yield decides how much of your story an agent can hold.

Why doesn’t the live page match the screenshot’s 114 KB? The capture stats belong to a page version. The homepage has been revised since draft v6; every revision gets sifted, measured, and audited again.