What Is Web Scraping?

To scrape something is to scratch it off a surface, and Google's spam policies have a section under exactly that name. The term describes an automated process that copies data from a page to use it somewhere else, for example a price or the list of results for a search.

A crawler follows links to discover pages, while a scraper goes to specific pages for specific data; the entry on crawlers explains the difference. An API is the route a site itself offers for handing over its data under terms of use; scraping reads what the page shows to people, without that agreement, and depends on a layout it does not control.

The process has three steps. The first is fetching: the program requests the URL over HTTP and receives the HTML. If the page loads its content with JavaScript (client-side rendering), that HTML can arrive without the data, and a headless browser has to run the code before anything is read.

The second is parsing. The program turns the HTML into a tree of elements, the DOM, and finds the data by position or attributes: the element holding the price, the CSS class of the title. If the page also publishes product structured data, the scraper can read it from that block, which does not depend on the layout.

The third is storage in a spreadsheet or a database, with the source URL and the date of extraction for every value. Without a date, a price series cannot be compared over time.

Before the first request, a well-built scraper reads the domain's robots.txt. The protocol was published as RFC 9309 in September 2022, and it says so explicitly: its rules are not a form of access authorization. robots.txt states which paths the owner asks crawlers not to visit. It says nothing about what may be done with the data on a page that is visited.

In marketing, scraping is used to track competitors' prices and product ranges and to follow positions in search results (SERP). Google's spam policies, last updated on 28 August 2026, list under machine-generated traffic "scraping results for rank-checking purposes or other types of automated access to Google Search conducted without express permission", and state that such activities violate its spam policies and the Google Terms of Service. The same page calls it scraping when content is taken from other sites and hosted to manipulate search rankings, a case of duplicate content. AI crawlers that download text to train models also extract content automatically.

The rules below do not make scraping legal or illegal across the board: it depends on what data is extracted, from which source and for what purpose. If the data relates to people, such as names or email addresses on a profile, the GDPR applies. Its definition of processing covers collection and retrieval, "whether or not by automated means" (Art. 4(2)), and its definition of personal data does not exclude data published on a website (Art. 4(1)). A legal basis under Article 6 is needed, such as legitimate interests weighed against the rights of the person concerned, and Article 14 requires informing that person, with exceptions such as disproportionate effort.

Directive 96/9/EC of 11 March 1996 gives the maker of a database a sui generis right where obtaining, verifying or presenting the contents involved a substantial investment. The maker can prevent extraction and re-utilisation of the whole or a substantial part (Art. 7(1)). The repeated and systematic extraction of insubstantial parts is not permitted either where it conflicts with normal exploitation of the database or unreasonably prejudices the maker's legitimate interests (Art. 7(5)). A lawful user may extract insubstantial parts, and a contract cannot take that away (Art. 8(1) and Art. 15). The right expires fifteen years from 1 January of the year after completion, and a substantial change gives the resulting database its own term (Art. 10).

The Court of Justice of the EU has applied the directive in two judgments. In Case C-762/19, CV-Online Latvia, of 3 June 2021, it ruled that a specialised search engine which copies and indexes the whole or a substantial part of a database freely accessible on the internet, and then lets its users search that database on its own website, is extracting and re-utilising its content, which the maker may prohibit where this puts the return on its investment at risk, a point for the national court to verify. In Case C-30/14, Ryanair, of 15 January 2015, it ruled that the directive does not apply to a database protected neither by copyright nor by the sui generis right, so it does not preclude contractual limits on its use, without prejudice to national law.

For automated analysis of content, Directive (EU) 2019/790 of 17 April 2019 applies. It defines text and data mining as "any automated analytical technique aimed at analysing text and data in digital form in order to generate information which includes but is not limited to patterns, trends and correlations" (Art. 2(2)). Article 4 allows reproductions and extractions of lawfully accessible works for that purpose unless the rightholder has reserved this use "in an appropriate manner, such as machine-readable means in the case of content made publicly available online". The directive does not mention robots.txt; recital 18 refers to "metadata and terms and conditions of a website or a service". How the reservation is expressed is covered in the entry on TDM rights reservation.

• Check first whether the site offers an API or a data export.
• Read robots.txt and the terms of use before starting, and keep the version you read with its date.
• Space out requests and use a user agent of your own.
• Extract only the fields you need and no personal data the purpose does not require.
• Check the results whenever the source site changes its layout.

• Treating robots.txt as access authorization or as a technical block.
• Assuming that data published on a website is not personal data.
• Believing that extracting a little every day never touches a substantial part.
• Republishing scraped content without adding anything of your own.