Automated thematic research-overview pages for the MaRDI portal.
The pipeline searches recent arXiv papers using queries stored on research-theme items in the MaRDI Wikibase, classifies the results into registered themes, imports matched papers as canonical publication items, links them from the matching theme item, and ensures each theme has a wiki page that renders live from the KG.
Research themes are Wikibase items with P31 -> Q7266523. Each theme can carry
an arXiv search query (P1965) and/or an OpenAlex query string (P1967),
configured via TOPIC_OVERVIEWS_ARXIV_QUERY_PROPERTY and
TOPIC_OVERVIEWS_OPENALEX_QUERY_PROPERTY. Paper membership is stored on the
theme, not on the paper:
Q_theme --P265 has part(s)--> Q_paper
Imported papers use the local MaRDI IDs in topic_overviews/kg/model.py,
including P21 arXiv ID, P159 title, P28 publication date, P43 author name
string, P1963 TL;DR, P1964 keywords, and optional P1642 generated-by model
provenance.
Each theme page is created with only:
{{ResearchTheme}}
Template:ResearchTheme and Module:ResearchThemePublications render the paper
cards live from SPARQL. Theme overview/index pages are handled on the wiki side
with SPARQL, not generated by this repo.
Configuration is environment-variable first.
| Variable | Default | Purpose |
|---|---|---|
TOPIC_OVERVIEWS_ARXIV_QUERY |
empty | Fallback arXiv search query for themes without a KG query |
TOPIC_OVERVIEWS_ARXIV_QUERY_PROPERTY |
P1965 |
Theme-item property containing an arXiv search query |
TOPIC_OVERVIEWS_OPENALEX_QUERY_PROPERTY |
empty | Theme-item property containing an OpenAlex query string (P1967) |
TOPIC_OVERVIEWS_OPENALEX_EMAIL |
empty | Email passed to OpenAlex polite pool (mailto param) |
TOPIC_OVERVIEWS_SINCE_DAYS_PROPERTY |
empty | Theme-item property for per-theme harvest window in days (P1968) |
TOPIC_OVERVIEWS_SINCE_DAYS |
10 |
Default search window in days (overridden per theme via P1968) |
TOPIC_OVERVIEWS_RESEARCH_THEME_QID |
Q0 |
Research-theme class, usually Q7266523 |
TOPIC_OVERVIEWS_LLM_PROVIDER |
anthropic |
LLM provider: anthropic or openai |
TOPIC_OVERVIEWS_MODEL_QID |
empty | KG model item used for P1642 provenance and runtime model lookup via P1966 |
TOPIC_OVERVIEWS_OPENAI_BASE_URL |
https://ollama.zib.de/api |
OpenAI-compatible chat API base URL |
TOPIC_OVERVIEWS_OPENAI_API_KEY |
empty | Bearer token for the OpenAI-compatible API |
TOPIC_OVERVIEWS_HARVEST_LIMIT |
0 |
Max new papers inserted into the KG; 0 means unlimited |
TOPIC_OVERVIEWS_DRY_RUN |
false |
Classify without writing |
ANTHROPIC_API_KEY |
empty | LLM API key |
MEDIAWIKI_API_URL |
empty | Portal MediaWiki API URL |
MEDIAWIKI_BOT_USER / MEDIAWIKI_BOT_PASSWORD |
empty | Bot credentials |
WIKIBASE_URL |
empty | MaRDI Wikibase URL |
SPARQL_ENDPOINT_URL |
empty | MaRDI SPARQL endpoint |
pip install -e '.[dev]'
python -m pytest -q
TOPIC_OVERVIEWS_DRY_RUN=true python -m topic_overviews --dry-run
python -m topic_overviews --themes-only
python -m topic_overviewsThe default harvest path uses the arXiv search API sorted by submission date,
running one search per distinct theme query. harvest/arxiv_oai.py remains as
tested OAI-PMH support and shared record model code, but it is not the current
CLI harvest mode.
To limit a run to a single new paper inserted into the KG, set:
TOPIC_OVERVIEWS_HARVEST_LIMIT=1