Skip to content

Glossary Big data

What Is Big Data?

Manuel Riveiro Rodriguez
Manuel Riveiro Rodriguez

CEO & Digital Strategist

6 min reviewed on 2026-10-22

  • Analytics
Definition

Big data refers to datasets so large, so fast or so varied that ordinary tools can no longer cope, and a scalable architecture is needed to store, process and analyse them.

On this page 5
  1. What big data means
  2. How it works
  3. Why it matters
  4. Best practices
  5. Common mistakes
In brief

Big data is data so large, fast or varied that it needs a new architecture. What it is, how it works and why Search Console never shows it all.

What big data means

The definition most cited in technical circles comes from NIST, the US standards institute. In its 2019 interoperability framework, it describes big data as extensive datasets, characterised mainly by volume, variety, velocity or variability, that require a scalable architecture for efficient storage, manipulation and analysis.

The second half of that definition is the important part. Big data is more than "a lot of data": the size, the pace at which it arrives or the mix of formats forces a change of tool. A spreadsheet or a database on a single server stops being enough, and the work is spread across many machines.

The four characteristics are easier to see with examples. Volume: the log files of a large web server, with millions of lines a day. Velocity: the clicks in an online shop that have to be processed as they happen. Variety: tables alongside text, images or machine logs. Variability: data whose meaning or rhythm shifts over time, such as a website's traffic before and after a Google update.

How it works

The technical principle is distribution. Instead of one ever more powerful server, NIST describes the big data paradigm as distributing data systems across horizontally coupled, independent resources: many ordinary machines, each storing and processing one part. When the data grows, more machines are added.

Two typical ways of working sit on top of that. In batch processing, data is collected and analysed at intervals, for instance once a day. In stream processing, it is analysed as it arrives, which makes it possible to react within seconds. Cloud data warehouses such as BigQuery let you query huge tables with SQL without managing the machines behind them.

For people in marketing or SEO, contact with big data is usually indirect: data from Search Console, Google Analytics or server logs arrives already summarised by a tool. And every summary leaves something out.

Why it matters

Search Console is the clearest example. Google explains that its performance data has two limits. One is privacy: queries that have not been made by more than a few dozen users over two to three months are anonymised and do not appear in the tables, although they do count in the chart totals. The other is the number of rows: according to Google in 2022, the interface exports at most 1,000, and according to its current documentation the API returns at most 50,000 rows per day and per search type. Since February 2023, the bulk export to BigQuery delivers all the data without that row limit. There too, the wording of anonymised queries stays hidden: their rows carry is_anonymized_query = true and an empty query field.

Google illustrates this on its blog with a worked example: the itemised queries add up to 450 clicks, but the chart total shows 550, because it also includes the clicks from anonymised queries. Anyone who adds up the rows of the table or the API will therefore never reach the total.

For AI search, the scale is larger still. Language models are trained on text collections that can only be handled with big data techniques, and AI search systems rely on indexes that cover a large share of the web. Whatever a website publishes becomes part of that data, with the same logic of volume, velocity and variety.

Work out shares against the overall figure, never a table alone.

Best practices

  • Frame the question before collecting data: volume without a concrete question only produces more storage.
  • Know the limits of each source: sampling, anonymisation, row limits and delays before data is final.
  • On large sites, use the Search Console bulk export to BigQuery instead of the API if you need every row.
  • Store totals and breakdowns separately, and never compare a sum of rows with a total without knowing what is missing.
  • Record when the data is from and what state it is in, for instance provisional or final.
  • Handle personal data in line with the GDPR from the design stage, not as an afterthought.

Common mistakes

  • Adding up Search Console rows by query and expecting them to match total clicks.
  • Confusing volume with insight: more data does not answer a badly framed question on its own.
  • Mixing provisional and final data in the same time series.
  • Building big data infrastructure for an amount of data that fits in a spreadsheet.
  • Keeping personal data "just in case", with no purpose and no legal basis.
Manuel Riveiro Rodriguez CEO & Digital Strategist

A technical audit covers this and everything else in one pass.

Request an audit

Frequently asked

How much data does it take to count as big data?

There is no fixed threshold. NIST defines big data by the need for a scalable architecture rather than by a number of gigabytes: data counts as big data when its volume, velocity, variety or variability can no longer be handled by a single conventional system. What is large today may be ordinary in a few years.

What is the difference between big data and data analysis?

Big data describes the kind of data and the infrastructure it needs. Data analysis is what you do with it: questions, methods and conclusions. You can analyse a small table without big data, and you can have big data without analysing it well. The value comes from the question.

Why don't the clicks in Search Console add up?

Because the tables by query leave out anonymised queries, those not made by more than a few dozen users over two to three months, while the chart total includes them. On top of that, the API returns at most 50,000 rows per day and per search type. The sum of the rows therefore usually stays below the total.

Does the BigQuery export show every query?

It shows every row, without the daily limit of the API and the interface, including those of anonymised queries with their clicks and impressions. Only their wording is missing: for privacy, those rows carry is_anonymized_query = true and an empty query field. For sites with many pages and queries, it is the most complete way to get the performance data.

How is big data related to AI?

Language models are trained on huge text collections, and handling them takes the same distributed storage and processing techniques as big data. In the other direction, AI is used to analyse large volumes of data. For a brand, it means that what it publishes flows into those datasets and into the indexes that assistants query.

Sources

  1. NIST, Special Publication 1500-1r2, "NIST Big Data Interoperability Framework: Volume 1, Definitions", version 3, October 2019. Source for the definition of big data by volume, variety, velocity and variability, the need for a scalable architecture and the horizontal distribution across independent resources.
  2. Google Search Central Blog, "A deep dive into Search Console performance data filtering and limits", October 2022. Source for anonymised queries, the worked example of 450 against 550 clicks and the 1,000-row limit in the interface.
  3. Google for Developers, "Getting all your data", Search Console API documentation, last updated 28 August 2025. Source for the limit of 50,000 rows per day and per search type in the API.
  4. Google Search Central Blog, "Bulk data export: a new and powerful way to access your Search Console data", 21 February 2023. Source for the daily export to BigQuery without a row limit.
  5. Search Console Help, "Table guidelines and reference", no update date shown. Source for rows of anonymised queries in the bulk export, flagged with is_anonymized_query and with an empty query field.