Authorized crawling
Use the crawler for websites and content you own, manage, are authorized to test, or are otherwise permitted to collect.
Rules for safe, bounded, permission-respecting crawler use and responsible source-data processing.
Use the crawler for websites and content you own, manage, are authorized to test, or are otherwise permitted to collect.
Use bounded page limits, delay controls, and conservative settings when crawling sources that could be sensitive to traffic.
Exports should be used for review, automation, internal knowledge workflows, or AI/RAG preparation consistent with the source rights.
Report suspicious activity, abuse, or crawler misuse through the security contact channel.