Crawler.sh – Local AEO & SEO Spider with Markdown content extraction

Summary: Crawler.sh is a fast, local web crawler and SEO analysis tool that extracts site content as clean Markdown and exports data in JSON, CSV, or Sitemap XML formats. It operates from the terminal or a native desktop app, combining crawling, SEO checks, and content extraction in one tool.

What it does

It crawls entire websites quickly using a Rust-based engine, performs automated SEO and AEO audits, and extracts content for export in multiple standard formats.

Who it's for

Developers and SEO professionals needing fast, local site audits and content extraction without relying on cloud services or multiple tools.

Why it matters

It solves the problem of using slow, complex, or cloud-dependent tools by providing a fast, private, all-in-one local solution.