Applying rate limits and avoiding unnecessary load

Respecting robots.txt and site terms where applicable

Complying with privacy laws and minimizing personal data collection

Collecting sensitive personal data by default for completeness

Maximizing request rate regardless of server impact