Sosse

Sosse is an open-source search engine and web crawler designed to index, archive, and monitor web pages — including JavaScript-heavy sites — using browser-based rendering. It combines full-page archiving with flexible crawling policies and search capabilities for private or organizational use.

Key Features

• Index and search web page content, including dynamically rendered pages via browser automation
• Recurring and scheduled crawling with adaptive policies and queue management
• Pixel-perfect archiving: preserve HTML and assets, rewrite links for local/offline viewing
• Tagging and metadata support for organizing and filtering archived content
• Batch file downloads and content deduplication for large-scale collection
• Webhooks and RESTful API for integrations, automated processing, and AI-driven workflows
• Atom feed generation and change detection for pages without feeds
• Authentication and permission controls for accessing and searching private resources

Use Cases

• Institutional web archiving and long-term preservation of web pages and assets
• Internal site and document indexing for enterprise search and knowledge discovery
• Continuous monitoring and competitive analysis with automated alerts and exports

Limitations and Considerations

• Browser-based crawling (Selenium + headless browsers) increases resource usage and operational complexity compared to pure HTTP crawlers
• Requires browser binaries and drivers plus a production database (PostgreSQL) for scalable deployments
• Designed as a general-purpose crawler/search stack; very large-scale deployments may require additional tuning, infrastructure, and queue scaling strategies

Sosse is well suited for teams needing accurate rendering and archival fidelity for dynamic sites, combined with search and automation capabilities. It is distributed under a strong copyleft license and is commonly deployed using containerized images for evaluation and production.