feat(data): centralize sample-dataset metadata in a manifest (#774)

[SaguaroDev]

Closes #774.

Sample-dataset metadata was duplicated across four places: the load_sample if/elif chain, the tests, docs/library/sample_data.md, and the per-file include list in MANIFEST.in. They had already drifted: sample_data.md listed 23 of 46 datasets (with names like frieldand_uspp_auto_steady_state and friedland_us_industy_auto_case), and MANIFEST.in listed 22, so the sdist shipped incomplete.

This adds chainladder/utils/data/_manifest.py as the single source of truth (a plain Python dict, no new dependency per the issue discussion) and keys everything off it.

load_sample now looks up its Triangle config from the manifest instead of the if/elif chain. This is behavior-preserving: I captured the resolved origin/development/index/columns/cumulative for all 46 bundled samples before and after the refactor and they are byte-identical.

cl.list_samples() is the new utility @henrydingliu asked for. Per the design confirmed on the issue, it returns a DataFrame of name, index, columns, cumulative, and (when include_grain=True, the default) origin/development grain plus period counts. include_grain=False skips loading the data for a fast metadata-only listing. It doubles as the source for the docs table.

MANIFEST.in collapses the 22 hand-listed include lines to one recursive-include chainladder/utils/data *.csv. I built the sdist both ways and confirmed it now contains all 46 CSVs (was 22). Wheels were already complete via pyproject.toml's package-data wildcard; this fixes the sdist.

Tests test_load_sample now iterates the manifest rather than globbing the data directory, so adding a sample is a one-entry change and stray non-CSV files (__init__.py, _manifest.py) can't be mistaken for datasets. Added a both-ways sync assertion (manifest set == CSVs on disk) and a test_list_samples test.

docs/library/sample_data.md is regenerated from cl.list_samples() via scripts/regen_sample_data_docs.py. The table is now complete and accurate; rerun the script after adding a dataset.

Full test suite: 717 passed, 12 xfailed (pre-existing), 0 failures.

---

Note

Low Risk
Refactor of sample-loading and packaging metadata with tests guarding manifest/CSV parity and sdist contents; intended behavior parity for load_sample with no auth or production runtime path changes.

Overview
Introduces chainladder/utils/data/_manifest.py as the single registry for all bundled sample CSV metadata (origin, development, index, columns, cumulative). load_sample now validates keys and builds Triangle instances from that dict instead of a long per-dataset if/elif chain (intended to be behavior-preserving).

Adds cl.list_samples() (exported from chainladder.utils) to return a manifest-driven catalog, optionally loading each sample for grain/period counts when include_grain=True.

MANIFEST.in switches from ~22 explicit CSV includes to recursive-include chainladder/utils/data *.csv, fixing incomplete sdists; tests assert manifest ↔ on-disk CSV sync, cover list_samples, and optionally build an sdist to verify all CSVs ship.

Docs replace the static sample_data.md table with sample_data.ipynb (live cl.list_samples()) and document list_samples in the API reference.

^{Reviewed by Cursor Bugbot for commit 724b455. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.90%. Comparing base (449b5c1) to head (724b455).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #886      +/-   ##
==========================================
- Coverage   87.04%   86.90%   -0.15%     
==========================================
  Files          86       87       +1     
  Lines        4986     4932      -54     
  Branches      646      624      -22     
==========================================
- Hits         4340     4286      -54     
  Misses        456      456              
  Partials      190      190              

Flag	Coverage Δ
unittests	86.90% <100.00%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[henrydingliu]

@SaguaroDev thanks for all this! will the new utility function list_sample get added to the api reference section of the doc automatically? or do we need to manually add it?

[SaguaroDev]

It's manual, not automatic. The API reference (docs/library/api.md) hand-lists each function in an autosummary block, so I added list_samples there plus a matching generated stub, and dropped a duplicate load_sample entry that was already in that block.

On the sdist: added test_sdist_ships_all_samples, which builds a source distribution and asserts every sample CSV is present. It self-skips when the build package or a source checkout isn't available, so it proves completeness in CI without weighing down the fast suite.

Pushed 577fc84.

[henrydingliu]

we've been deleting these stubs in #879. can you confirm if creating list_samples_rst is still necessary?

[SaguaroDev]

Dropped it. The committed stub is redundant anyway since autosummary regenerates it at build, and with #879 removing these on experimental there's no reason to add a new one here. The list_samples entry stays in the api.md autosummary block, which is what actually wires it into the API reference.

[henrydingliu]

actually, now that we have list_samples, would it be cleaner to just change this markdown into a notebook? i.e. just call list_samples in the notebook

[SaguaroDev]

Done in 724b455. sample_data.md is now sample_data.ipynb with a single cl.list_samples() cell, so the table renders live from the manifest and can't drift. I kept the # Sample Dataset H1 so the existing sample_data.html#sample-dataset anchor (linked from the tutorials) still resolves, updated the _toc.yml entry, and removed scripts/regen_sample_data_docs.py since there's nothing static left to regenerate.

[henrydingliu]

reviewed sample_data.ipynb on fork to validate build. confirmed that warning from list_sample is pre-existing.