FreeWeb

What it does

• Searches the public web without API keys, using several engines in turn.
• Opens search results and extracts readable page content.
• Reads a site’s llms.txt and Markdown variants when they exist.
• Falls back to the Internet Archive for dead or blocked pages.

How it works

• Every URL goes through six fetchers in order and the first success wins: llms.txt and Markdown, GitHub raw files, RSS and Atom feeds, native fetch with jsdom, the Archive.org cache, and finally Playwright for script-heavy or protected pages.
• Because most pages resolve in the fast layers, the heavy browser is only started when a page needs it.

Who it is for

Developers who want their AI assistant to read the web without paying for a search API or managing keys.

Limits

• The Playwright layer needs Playwright’s Chromium to be installed locally; without it, pages that only render with JavaScript cannot be read.

Built with

• TypeScript
• MCP
• Playwright