nsepit

Point-in-time NSE universes, and survivorship bias you can measure.

Most retail backtests on Indian equities run on today's index members. Every historical portfolio inside them quietly excludes the companies that died. Everyone knows this is bad. Fewer people check how bad, and the answer turns out to depend heavily on the strategy: the bias is not a constant you can subtract.

NSE's terms don't allow redistributing its data, so nobody can hand you a survivorship-free dataset. What can be shared is the machine that rebuilds one. This package downloads the exchange's own daily files to your disk, reconstructs universes as they actually were (dead companies included, with their prices), refuses to build when its own checks fail, and then measures the bias for whatever strategy you hand it.

Methodology background: the write-up this code was extracted from.

What it does

Point-in-time index membership (nsepit.membership). Start from the current NIFTY 50 constituents, walk the dated replacement history backwards, undo each change. Renames and mergers are resolved through an alias table before any set arithmetic, so Sesa Goa and Vedanta count as one company, not two. The build refuses to emit output unless its invariants hold: a changes table carrying the columns it claims to, exactly 50 members at every month-end, no duplicate symbols, every name mapped to a traded symbol. Twenty hard-coded spot checks against known index events sit on top. Any failure raises; there is no partial output. A snapshot of the constituent and changes tables ships in the package, so this works offline out of the box (and the invariants run against the real tables in CI).

A survivorship-free liquidity universe, with prices (nsepit.bhavcopy, nsepit.universe, nsepit.prices). The exchange's own daily settlement file (the bhavcopy) lists every security that traded that day, including names that later delisted. Membership is trailing liquidity rank: top N by 63-day median traded value, above a floor, with a year of history. A company that delisted in 2013 is in the 2012 universe because it traded in 2012. No committee and no vendor curation involved. The same archive yields the close-price panel, which matters because delisted names are exactly the ones free price vendors erase; closes_from_long builds it, corporate-action adjusted where the evidence is unambiguous, with every applied adjustment logged.

Integrity checks that stop the build (nsepit.integrity). Frame audits (duplicate dates, OHLC violations, non-positive closes, extreme moves), a cross-source check that classifies divergent days instead of averaging over them, and a corporate-action snapper that only undoes a price gap when the close-implied and open-implied factors both point at the same standard ratio, with tighter demands the smaller (more crash-like) the ratio gets. Ambiguous gaps stay real moves, because a large single-day drop is also what a genuine disaster looks like. The scan for correlation's blind spot (both sources agreeing on a real as-traded gap) is review-only by design: it reports, it never auto-excludes.

The bias harness (nsepit.bias). Run any strategy callable twice, same prices, same engine, same costs: once restricted to members as of each decision date, once with current members applied retroactively. The per-strategy delta is the survivorship bias, measured. Ships with toy strategies (equal weight, textbook 12-1 momentum, random picks) and a random-selection baseline. The toys are toys; bring your own strategy as a factory returning select(date) -> list[str].

Why measure it instead of hand-waving at it

Because the number moves. Run the harness over a few strategy families and the bias varies several-fold, and it is worst for exactly the families that concentrate in names heading for trouble. A momentum screen and a low-vol screen sitting on the same survivor-only universe are not lying to you by the same amount. "We acknowledge survivorship bias" tells you nothing about whether your particular backtest is off by half a percent or by half its edge. Running the harness answers the question for the strategy you actually trade.

Why this instead of a CSV and a prompt

Constituent lists you can already download: niftyhistory.in has per-rebalance CSVs, and there are academic datasets on figshare. If a membership list is all you need, use one. This package is for the two things a CSV does not carry.

First, prices for the dead companies. A membership list tells you Ranbaxy was in the index in 2012; it does not give you Ranbaxy's 2012 closes, and free price vendors have a habit of erasing delisted names entirely. The bhavcopy archive is the one free source that cannot have that hole, and this package turns it into a usable panel.

Second, provenance. A static CSV is trust-me data. This package rebuilds membership from public sources and refuses to emit output unless the count invariant, the symbol mapping, and twenty spot checks against known index events all pass. The same checks will happily audit anyone else's CSV.

As for the code: you could prompt an LLM into writing a bias harness in an hour. I extracted this one from my own working pipeline with the original open next to it, and adversarial review still caught eight quiet semantic drifts before release, including a validation gate that had weakened and a corporate-action guard that had become less conservative. Freshly prompted code has nobody catching that. The code is the cheap part. The encoded judgment, and the planted-defect tests that hold it in place, are what you're actually downloading.

Quickstart

pip install -e .

Measure bias on synthetic data first (no downloads, runs in seconds):

python examples/survivorship_bias_demo.py

Real data. Fetching from NSE (or membership --fetch) needs the net extra; nothing else in the package does:

pip install -e ".[net]"

Fetch the bhavcopy archive from NSE (one file per trading day, about 4,300 files since 2009; the first crawl takes a while at a polite request rate):

nsepit crawl --out data/bhavcopy_raw --start 2009-01-01
nsepit parse --raw data/bhavcopy_raw --cache data/bhav_long.pkl
nsepit universe --raw data/bhavcopy_raw --cache data/bhav_long.pkl --out data/universe_mask.csv
nsepit prices --raw data/bhavcopy_raw --cache data/bhav_long.pkl --out data/closes.csv
nsepit membership --out-dir data/membership
nsepit bias --prices data/closes.csv --membership data/membership/membership_monthend.csv --eval-start 2012-01-01

That last command is the point of the package: the toy strategies run twice, point-in-time versus current-members-retroactive, and the per-strategy delta prints as a number. membership uses the bundled snapshot by default (--fetch pulls the live Wikipedia tables; --constituents/--changes read your own saved CSVs).

The same measurement on your own strategy, in Python:

import pandas as pd
from nsepit import measure_bias, momentum_topk, equal_weight, format_report

prices = pd.read_csv("data/closes.csv", index_col=0, parse_dates=True)
membership = pd.read_csv("data/membership/membership_monthend.csv",
                         index_col=0, parse_dates=True)
report = measure_bias(prices, membership,
                      {"EW": equal_weight(), "MOM": momentum_topk(k=10)},
                      eval_start="2012-01-01")
print(format_report(report))

pip install -e .[test] and pytest runs the test suite. Tests run on synthetic fixtures (fabricated symbols, fabricated prices, planted defects) plus the bundled membership tables, whose real invariants and all twenty spot checks run in CI. No network, no market data.

Where the data comes from, and what you may do with it

This package ships no market data, and that is deliberate.

• Bhavcopy files are downloaded by your machine directly from NSE's own archive endpoints. NSE's terms of use and data policy restrict reproduction and redistribution of its data; licensed redistribution goes through NSE Data & Analytics. I could not find a plain-language yes for re-sharing downloaded bhavcopy files, so assume no: keep them on your disk, don't commit them, don't republish them. Read the current terms yourself; they change, and this README does not track them.
• Index constituent and change tables are facts about a public index, derived from Wikipedia's NIFTY 50 page (CC BY-SA, credited here). A snapshot ships in the package so membership reconstruction works offline; the spot checks pin it through 2026-05-31, and the build refuses to reconstruct past the snapshot's end (2026-06-30) rather than silently missing later changes. --fetch pulls the live tables (needs the net extra); when Wikipedia's layout drifts and the fetcher breaks, save the two tables as CSVs and use the offline path.

What it deliberately does not do

• No strategies beyond the toys. Nothing in here is a signal. The point is the universe and the measurement.
• No dividends and no total-return series. Price-only. If your holding periods are long, the gap matters and you should know it is there.
• No fundamentals, no intraday, no derivatives.
• No automatic splicing across ticker renames. The membership mask keys each company by its current symbol, so where the ticker changed mid-history (Tata Motors became TMPV, Zomato became ETERNAL) the old prints sit under the old symbol in your panel. closes_from_long takes a renames map and does the splice when you hand it one; no default map ships, because a rename that rides on a demerger carries a real price break that needs a human decision.
• No automatic exclusion of flagged names. The corporate-action scans produce review lists for a human. A heuristic that cannot tell a 1:2 bonus from a 50% disaster does not get to act alone, and there is a pharma company in the historical record that makes this point vividly.
• No guarantee the snapper catches everything. There is a ground-truth harness: point it at names with known-correct adjusted series and it refuses the build when the snapper fails to recover their real events, or invents events on crash days. A real disaster that gaps exactly like a clean ratio on both legs will still fool it; the harness measures that failure rate, it does not abolish it.

License

MIT.