🛡️ Compliant Site Content Fetcher — Ethical Scraping Engine

Compliant Site Content Fetcher is a masterclass in responsible automation. Built on n8n, this system is designed for developers who need to aggregate web data without compromising ethical standards or technical compliance. It treats robots.txt files as hard boundaries, automatically validating every request before execution to preserve site integrity and prevent IP blacklisting.

By combining intelligent content detection with resilient backoff algorithms, this project sets the standard for production-grade, low-impact data extraction.

🛠️ Technology Stack

✨ Features at a Glance

🛡️ Automated Compliance Guard

Robots.txt Validator: A custom-coded JavaScript engine that fetches and parses a site’s robots.txt in real-time.
Auto-Kill Directive: If the parser detects a Disallow: / or specific bot exclusions, the workflow terminates immediately with a detailed compliance log.
User-Agent Management: Transparently declares its identity to site owners, adhering to “Politeness” protocols.

🧠 Polymorphic Content Extraction

Header-Aware Routing: Automatically identifies the input format (HTML, JSON, or XML) and routes the payload to the specific parser node.
Advanced Scraping Modes:
- API Mode: Directly parses structured JSON for high-fidelity data.
- Feed Mode: Converts RSS/Atom XML into refined, object-based article lists.
- Fallback DOM Mode: Uses specialized selectors to extract meta-descriptions and titles from unstructured standard webpages.

⏳ Industrial-Grade Resilience

Exponential Backoff: Implements a sophisticated retry algorithm (base 1000ms, doubling per attempt) to handle 5xx server errors or rate limiting (429s).
Rate-Limit Awareness: Includes built-in Wait nodes that respect site-specific “crawl-delay” directives found in robots.txt files.

📁 Workflow Architecture

site-fetcher-workflow/
├── Webhook Trigger      # Entry point with Target URL
├── Compliance Node      # JavaScript-based robots.txt check
├── HTTP Fetcher         # Secure data retrieval with custom Headers
├── Logic Switch         # Determines Content-Type (JSON/RSS/HTML)
├── Payload Parser       # Regex & Logic nodes for data cleaning
└── Success Output       # Formatted JSON storage

🚀 Installation & Setup

1. Prerequisites

n8n Instance (v1.0+ recommended)
Node.js capability enabled in n8n settings (for custom JavaScript nodes)

2. Workflow Import

Download the compliant-fetcher.json from this repository.
In n8n, click “Add Workflow” -> “Import from File”.
Select the JSON file.

3. Usage

Send a POST request to the Webhook URL with a targetUrl parameter:
```
{
  "targetUrl": "https://example.com/blog"
}
```
Monitor the n8n execution log to see the compliance verification and data extraction result.

📄 License

This template is part of an automation framework for ethical AI and is available under the MIT License.

Designed & Developed by Aditya (aka aditya-2129)