Methods · CAFE Dataverse Dashboard

The one thing to know before reading any chart

“Published” is not the same as “visible”. Harvested datasets are indexed from other repositories and are fully visible to users, but they carry no local publication date. Filtering on publication date is not null silently drops of the live collection. Everything on this site uses published locally OR harvested as the test for whether a dataset is live. The consequence shows up wherever a panel needs a publication date rather than a publication fact: deposit timelines cover 794 datasets, not all 1,307.

Scope

The extract covers the CAFE collection and everything linked into it. Drafts are excluded from this dashboard, as SETUP.md requires.

What is counted

Three ways a dataset reaches CAFE

All three count as part of the collection, and the site's “subcollection” stratifier resolves each to a collection inside the CAFE tree:

Owned — the dataset lives in a CAFE-subtree collection.
Linked in — it lives elsewhere and is linked into a CAFE collection individually. Its own collection alias is outside CAFE, so the subcollection shown is the CAFE collection it is linked into.
In a linked collection — its owning collection, or an ancestor, is linked into CAFE, so the whole collection browses under CAFE. Resolved by walking its ancestry against the collection-link table.

Collection links are followed one hop only. Following them transitively would make “under CAFE” effectively unbounded.

Where the data comes from

Two sources, for a reason worth stating plainly.

1. A SQL extract of the Harvard Dataverse database

The primary source: 44 tab-delimited extracts pulled on 2026-09-08 from the Dataverse application database, scoped to CAFE by a query pack whose own scope was reconciled against Harvard Dataverse's public search index exactly — 794 published plus 513 harvested equals the 1,307 the index reports.

The set on disk is the no-personal-data subset. Nine of the 53 produced extracts contained email addresses and were withheld: 1.4, 1.5a, 1.5b, 1.7, 2.3, 2.5a, 2.13b, 2.13c, 3.1 and 4.1d. Most of those addresses were not in columns whose names suggested them; five were found only by scanning the files.

2. Harvard Dataverse's public Search API

Withholding 1.4 and 1.7 also removed the only copy of the dataset descriptions, which the required description-similarity analysis cannot be done without. They were re-acquired from the public, unauthenticated, read-only Search API, which also supplies parsed author lists, author affiliations and time-period-covered.

Email-typed and contact fields are dropped before anything is written, and any address a depositor typed into free text is redacted by pattern. The fetch refuses to finish if one survives.

What the API could and could not supply

Requesting the citation metadata block returns IndexOutOfBoundsException for individual records, and the failures are not random: every harvested record fails. The fetch bisects around them, so the affected datasets keep everything the plain search result gives and simply lack the citation-block fields. The dataset export endpoint is not a fallback: it returns 403 to anonymous clients.

The site prints the denominator wherever it shows a partially covered field. A 42%-covered field is never charted as if it described the collection.

The pipeline

Every number shown here is produced by a script that runs on its own, in R or in Python, without the site running. The site's JavaScript filters, sorts and draws; it computes no statistics. R does the descriptive layer and Python does the semantic-similarity layer, so each half is reviewable by whoever reads that language.

Decisions that change a number

Authors

Parsed from the API's author list, one entry per author, rather than by splitting a joined string. Descriptor labels — Federal Agency, N/A, Various and others listed in config/dashboard-config.yml — are removed before counting.

Matching is on the whole trimmed string, case-insensitively, never as a substring. A substring rule would delete Industry Canada along with Industry.

Per-author totals count a dataset once for each of its authors, so they sum to more than the collection totals. That is what a per-author view means, and the tables say so rather than presenting it as a partition.

Keywords

Taken from the extract that stores one row per value, not from the joined string, so a keyword containing a semicolon survives. Whitespace is collapsed and case folded; placeholders such as n/a are dropped.

The display spelling is the most common original, except that an all-capital spelling longer than five characters loses to any mixed-case variant — otherwise one depositor who shouts every keyword sets the label for everyone. Short all-caps strings are left alone, because they are usually acronyms.

Counts are cross-checked against the extract's own keyword census; the pipeline warns if the two parsings disagree.

Subjects

N/A is not a subject. Dataverse assigns it to records arriving without one, which is nearly every harvested record, and it accounts for 39% of the distribution. It is reported separately and left out of the subject chart. A chart built without this adjustment puts a non-category second from the top and misleads.

Files, sizes and usage

File counts and sizes measure the latest released version only. Superseded versions are not counted and draft-only datasets are excluded, so this is what is published rather than what is stored. It is the wrong number to size a migration with.

Downloads are guestbook rows collapsed to dataset-level events: a zip download of N files writes N rows, deduplicated per actor per minute. Views come from Make Data Count as monthly aggregates. They are different mechanisms and are never added into a single “usage” number.

Three questions the database cannot answer

From the original request sheet. Each is documented at the point of use rather than quietly approximated.

Search-term frequency

Dataverse stores user searches nowhere. Its action log records commands, not searches, and search queries never reach the database at all. Real answers need web server access logs, the analytics tag, or search-engine request logging.

The keyword and title-term frequencies on this site are content proxies: they describe what the collection contains, not what anyone looked for.

Views by individual user

View counts come from Make Data Count as monthly aggregates by country and agent type. Unique-view counts exist; viewer identities are not recorded anywhere in the schema.

The nearest per-user records are explore and preview events, which only fire for external tools and cover a small fraction of activity.

Location of the contributor

There is no such field. Linking a contributor to a place would need the depositor's account record, and the user table has no link to a collection, so “users of CAFE” is not a question it can answer.

The site shows the three things that do exist, labelled distinctly and never combined: where the data is about, where viewers are, and what institution an author gave. The last is the nearest proxy and it is an institution, not a place.

Privacy

This repository is public, so two rules apply to everything committed.

No personal data. The extract set on disk already excludes the nine outputs carrying email addresses. The API fetch drops contact and email-typed fields and redacts free-text addresses before writing. Depositor usernames, which the master extract carries, are dropped in the first analysis script rather than merely left unselected later. Contributor counts are published; contributor names are not.

Small cells are suppressed where they could identify. Views by country are published at the collection level, where they aggregate over every dataset. The per-dataset breakdown is suppressed below five datasets per cell: a single view from one country on a niche dataset is potentially identifying. This is a policy judgement, and it is recorded here so it can be revisited rather than rediscovered.

No bulk data. The extract set and the API response cache are both excluded from version control. What is committed is the aggregated payload the site draws, plus one row per published dataset carrying public metadata and a truncated description snippet.

Reproducing this

From dashboard-site/, with the extract set present at ../dv-data/:

just setup     # restore the R and Python environments
just fetch     # re-acquire public metadata from Harvard Dataverse
just build     # run the full pipeline, R then Python
just serve     # preview the site at http://localhost:8000
just check     # lint, tests, and the privacy scan

Any single step runs on its own. Rscript analysis/R/04-keywords.R rebuilds just the keyword summaries; uv run python analysis/python/11_keyword_similarity.py rebuilds just the keyword similarity. Each script's header says what it reads and what it writes.

The pipeline is deterministic. Every projection and factorisation is seeded from config/dashboard-config.yml, no method downloads a model, and rerunning without changing the inputs reproduces the same files.

Methods and provenance