Good Practices For Crawling Web Data

1. Do not circunvent any technological measures designed to prevent or restrict unauthorised acts. Particularly, respect any technological denial or restriction of access imposed by subscription models or paywalls.

2. Do not crawl websites that make content available to the public and are known to persistently and repeatedly infringe copyright and related rights.

3. Employ web crawlers that read and follow instructions expressed in accordance with Robots Exclusion Protocol(robots.txt).

4. Identify and comply with other appropriate machine-readable protocols to express rights reservations.

5. It's highly encouraged to engage on a voluntary basis in bona fide discussions with rightsholders with the aim to develop machine-readable standards and protocols to express rights reservations.