Wiki-Attention is a reproducible ETL pipeline and D3.js dashboard for retrospective analysis of Wikipedia attention dynamics. The project combines Wikimedia pageview dumps with Wikipedia structural dumps to aggregate daily attention into category-derived macrothemes, inspect temporal patterns, and drill down into seed/page-level explanations.
The current implementation focuses on en.wikipedia and covers the historical period from 2016-01-01 to 2025-12-31 for macrotheme-level and seed-level analysis.
Wikipedia pageviews provide a large-scale signal of information seeking, but page-level traffic is difficult to interpret when the goal is to understand broad thematic dynamics. This project addresses that problem by mapping Wikipedia pages and categories into 12 analytical macrothemes and by providing an interactive dashboard for exploring attention patterns over time.
The final dashboard supports four coordinated views:
-
Macrotheme overview A weekly 100% stacked streamgraph showing the relative share of 12 macrothemes over time.
-
Year-over-year macrotheme comparison A vertical streamgraph aligned by day/month of the year for the selected macrotheme.
-
Daily macrotheme series A daily time series with raw, baseline, and residual modes.
-
Seed/page-level explanation A Top-15 seed ranking for the selected macrotheme and day, with tooltips showing Top-5 weighted explanatory pages when available.
wiki-attention/
├── config/
│ └── config.yaml
├── dashboard/
│ ├── index.html
│ ├── package.json
│ ├── public/
│ │ └── data/
│ │ ├── macro_views_wide.csv
│ │ ├── seeds/
│ │ └── seed_pages/
│ └── src/
│ ├── main.js
│ └── style.css
├── data/
│ ├── raw/
│ │ └── wikidumps/
│ ├── intermediate/
│ └── processed/
├── etl/
│ ├── 00_sanity_check.py
│ ├── 01_download_wikidumps.py
│ ├── 02_build_snapshot_db.py
│ ├── 03_aggregate_category_views.py
│ ├── 04_build_macrothemes.py
│ ├── 05_export_macro_views.py
│ ├── 06_export_macro_views_wide.py
│ ├── 07_export_seed_top15_daily.py
│ ├── 08_build_seed_page_weights.py
│ ├── 09_export_seed_top_pages.py
│ └── common.py
├── .env.example
├── .gitignore
├── requirements.txt
├── validate_snapshot.py
├── check_snapshot_counts.py
├── count_parquet.py
├── inspect_day.py
└── README.md
The data/ directory contains large generated artifacts and is not intended to be committed to Git. It contains raw Wikimedia dumps, intermediate DuckDB databases, processed Parquet files, and CSV exports.
The ETL pipeline requires Python and the packages listed in requirements.txt:
duckdb>=1.0.0
polars>=1.0.0
pyarrow>=16.0.0
tqdm>=4.66.0
requests>=2.32.0
PyYAML>=6.0.1
python-dotenv>=1.0.1
The dashboard uses:
D3.js 7.9.0
Vite 7.3.1
Node.js and npm are required to run the dashboard locally.
On Windows PowerShell:
python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements.txtcd dashboard
npm install
cd ..The main configuration file is:
config/config.yaml
Example:
project: "en.wikipedia"
pageviews:
agent: "user"
start_date: "2025-10-28"
end_date: "2025-12-31"
snapshot:
dump_date: "latest"
wiki: "enwiki"
hierarchy:
depth: 1
paths:
raw: "data/raw"
wikidumps: "data/raw/wikidumps"
intermediate: "data/intermediate"
processed: "data/processed"The dates in config.yaml define the default processing interval for pageview-related stages. For the final 2016–2025 dataset, the pipeline was executed in multiple smaller date windows due to the large processing time, network interruptions, and local hardware limitations. Therefore, the current dates in config.yaml may correspond to the last processed window rather than the full historical period represented in the final exports.
For long runs, update start_date and end_date in config.yaml or use command-line arguments where supported.
The ETL pipeline is organized into numbered stages.
Checks whether the Wikimedia pageview_complete format contains the expected page_id field.
python -m etl.00_sanity_checkDownloads the structural Wikimedia dumps required to build the snapshot:
page.sql.gzcategorylinks.sql.gzlinktarget.sql.gz
python -m etl.01_download_wikidumpsBuilds the DuckDB structural snapshot from Wikimedia SQL dumps.
python -m etl.02_build_snapshot_dbThe snapshot contains page dimensions, article-to-category links, category-child-to-parent links, and auxiliary count tables used for fractional allocation.
The implementation uses DuckDB and may require a large temporary directory. On the original development machine, a local temporary directory such as D:\duckdb_tmp was used.
Aggregates daily pageviews into category-level daily Parquet files using fractional allocation.
python -m etl.03_aggregate_category_viewsThis stage processes Wikimedia pageview_complete data day by day and may take a long time for large date ranges. For the final historical dataset, this stage was executed in multiple smaller windows.
Builds the mapping between Wikipedia categories and the 12 analytical macrothemes using manually defined seeds and BFS traversal over the category hierarchy.
python -m etl.04_build_macrothemesThis stage produces category-to-macrotheme and category-to-macrotheme-seed mappings.
Exports daily macrotheme time series in LONG format.
python -m etl.05_export_macro_views --start 2016-01-01 --end 2025-12-31 --format csv --drop-otherExports daily macrotheme time series in WIDE format for the dashboard.
python -m etl.06_export_macro_views_wide --start 2016-01-01 --end 2025-12-31 --format csv --drop-otherThe resulting file is used by the dashboard as:
dashboard/public/data/macro_views_wide.csv
Exports Top-15 seed rankings for each macrotheme and day.
python -m etl.07_export_seed_top15_daily --start 2016-01-01 --end 2025-12-31 --k 15 --drop-otherThe resulting files are copied/materialized to:
dashboard/public/data/seeds/
There is one seed file per macrotheme.
Builds the DuckDB table connecting seeds to pages through weighted category relationships.
python -m etl.08_build_seed_page_weights --threads 8 --memory 10GB --tempdir D:/duckdb_tmp --rebuildThis stage creates the intermediate seed_weights.duckdb database.
Exports Top-5 weighted explanatory pages for selected macrothemes and date windows.
Example:
python -m etl.09_export_seed_top_pages ^
--macro Human_activities ^
--start 2024-06-01 ^
--end 2024-12-01 ^
--top-pages 5 ^
--max-seeds-per-day 15 ^
--threads 8 ^
--memory 10GB ^
--tempdir D:/duckdb_tmp ^
--overwriteThe resulting files are stored in:
dashboard/public/data/seed_pages/
Important: page-level explanation files are generated on demand for selected macrothemes and date windows. Generating seed-page explanations for the full 2016–2025 period and all macrothemes is computationally expensive. On the original development machine, a window of approximately six months for a macrotheme could take roughly 1 to 3 hours to process. For this reason, the final implementation precomputes page-level explanations only for analytically relevant windows.
The dashboard is located in:
dashboard/
To run it locally:
cd dashboard
npm run devThen open the local URL shown by Vite in the browser.
To build the dashboard:
npm run buildTo preview a production build:
npm run previewThe dashboard expects the following data structure:
dashboard/public/data/
├── macro_views_wide.csv
├── seeds/
│ ├── Culture_and_the_arts.csv
│ ├── Geography_and_places.csv
│ ├── Health_and_fitness.csv
│ ├── History_and_events.csv
│ ├── Human_activities.csv
│ ├── Mathematics_and_logic.csv
│ ├── Natural_and_physical_sciences.csv
│ ├── People_and_self.csv
│ ├── Philosophy_and_thinking.csv
│ ├── Religion_and_belief_systems.csv
│ ├── Society_and_social_sciences.csv
│ └── Technology_and_applied_sciences.csv
└── seed_pages/
├── Culture_and_the_arts.csv
├── Geography_and_places.csv
├── Health_and_fitness.csv
├── History_and_events.csv
├── Human_activities.csv
├── Mathematics_and_logic.csv
├── Natural_and_physical_sciences.csv
├── People_and_self.csv
├── Philosophy_and_thinking.csv
├── Religion_and_belief_systems.csv
├── Society_and_social_sciences.csv
└── Technology_and_applied_sciences.csv
The macro_views_wide.csv file is required for the main macrotheme and time-series visualizations.
The files in seeds/ are required for the Top-15 seed ranking.
The files in seed_pages/ are required for the Top-5 weighted explanatory page tooltips. These files may cover only selected date windows, depending on which macrothemes and periods were precomputed.
The complete local data directory contains large files and is excluded from the repository.
Observed local structure:
data/
├── raw/
│ └── wikidumps/
│ ├── enwiki-latest-categorylinks.sql.gz
│ ├── enwiki-latest-linktarget.sql.gz
│ └── enwiki-latest-page.sql.gz
├── intermediate/
│ ├── enwiki-latest-snapshot.duckdb
│ ├── seed_weights.duckdb
│ └── seed_weights.duckdb.tmp/
└── processed/
├── category_views_daily/
├── seed_top15_daily/
├── category_edges_depth1.parquet
├── category_to_macro.parquet
├── category_to_macro_seed.parquet
├── macro_views_daily_20160101_20251231.csv
└── macro_views_wide_20160101_20251231.csv
Some large local artifacts observed during development:
enwiki-latest-snapshot.duckdb: approximately 3.56 GBseed_weights.duckdb: approximately 7.57 GBcategory_edges_depth1.parquet: approximately 146 MBcategory_to_macro.parquet: approximately 46 MBcategory_to_macro_seed.parquet: approximately 53 MBmacro_views_daily_20160101_20251231.csv: approximately 2.5 MBmacro_views_wide_20160101_20251231.csv: approximately 404 KB- daily category Parquet files: approximately 68 MB per day in early 2016
The project uses 12 analytical macrothemes:
- Culture and the arts
- Geography and places
- Health and fitness
- History and events
- Human activities
- Mathematics and logic
- Natural and physical sciences
- People and self
- Philosophy and thinking
- Religion and belief systems
- Society and social sciences
- Technology and applied sciences
These macrothemes are analytical categories, not official Wikipedia taxonomy classes. They are generated through seed-based traversal of Wikipedia categories.
The project was developed and executed locally on a laptop with the following environment:
Operating system: Windows 11, version 25H2
CPU: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
RAM: 16 GB
Storage: 1 TB
Execution mode: local
Due to local hardware constraints and the size of the 2016–2025 pageview period, the ETL pipeline was executed incrementally across smaller time windows. Network interruptions during long processing runs were also a practical limitation.
The pipeline is modular and can be rerun stage by stage. However, complete reproduction of the full 2016–2025 processing requires substantial time, disk space, and stable network access.
The macrotheme-level and seed-level datasets were generated for the full 2016–2025 period. The page-level explanatory datasets were generated on demand for selected macrothemes and time windows, due to the computational cost of producing them for all macrothemes and all dates.
- The structural snapshot uses
enwiki-latestdumps and is static, while pageviews cover 2016–2025. - Wikipedia categories are noisy and not designed as a strict taxonomy.
- Macrotheme mapping depends on manually selected seed categories and BFS traversal depth.
- Pageviews are attention signals, not direct measures of social importance or causality.
- Top-K seed/page explanations improve readability but hide the long tail of smaller contributors.
- Page-level explanation files may only cover selected date windows.
- Full processing of all macrothemes and all page-level explanations is costly on local hardware.
Leonardo Yundi Aikawa Master's Dissertation Project Mestrado em Engenharia da Computação e Ciência de Dados Instituto Superior Técnico, Universidade de Lisboa