Serve Markdown to AI Agents with Accept Headers

HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.

We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.

I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.

It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now?

There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.

A cherry.jpg image/jpeg file is shown as an image by the browser

A logo in svg like https://news.ycombinator.com/y18.svg too

So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki

Browsers should implement that asap and we will follow

The open internet deserves a real solution instead of gatekeepers in the form of brightdata, firecrawl, cloudflare and the likes.

Maybe tarpit them into reading gigabytes of Markov Chain nonsense, really slowly?