diff --git a/CHANGELOG.md b/CHANGELOG.md index 48fa76d..19cd7ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.3.0] — 2026-08-29 + +### Added + +- **Canonical reporting layer** — unified schema and grading (#8) +- **Round-trip blotter** — `reports/blotter.py` renders a round-trip blotter with + Massive S3 flat-file backing (#2, #7). **Note:** the module ships but no + `@cli.command()` currently registers it — see `docs/SPEC.md` gap G1 +- **Development specification** — `docs/SPEC.md`: module boundaries and verified import + direction, the `DataProvider` contract, path ownership, CLI registration rules, the + release process, and a register of eight known gaps + +### Changed + +- **On-disk layout now follows the XDG Base Directory spec** (#4, ADR 0001). Paths resolve + via `tradekit.paths`; the two Falcon-suite contract paths tolerate both XDG and legacy + locations so suite components can migrate independently +- **Rebrand** — Personal AI Infrastructure / PAI → LifeOS across docs and metadata +- `__version__` now derives from installed package metadata instead of being a second + hand-maintained copy + +### Fixed + +- Blotter handles open positions and resolves Massive S3 credentials correctly (#7) + +### Security + +- Locked dependencies upgraded to clear 51 known CVEs (#6) + ## [0.2.0] — 2026-07-14 ### Added diff --git a/docs/SPEC.md b/docs/SPEC.md new file mode 100644 index 0000000..0324403 --- /dev/null +++ b/docs/SPEC.md @@ -0,0 +1,286 @@ +# tradekit — Development Specification + +**Status:** Living document +**Applies to:** `src/tradekit/` and the packaging around it + +This is the spec for working *inside* tradekit: how the package is layered, who owns +which path, what a data provider has to implement, when a feature becomes a command, +and how a release is cut. + +## Scope + +| This document governs | Where it lives instead | +|---|---| +| Internal module boundaries and import direction | — | +| The `DataProvider` contract | — | +| Configuration precedence and the no-fallback rule | — | +| CLI registration and what "shipped" means | — | +| Version single-source-of-truth and the release checklist | — | +| Vendoring policy | — | +| **Cross-repo contracts with falcon / falcon-stats** | [`falcon-suite-compatibility.md`](falcon-suite-compatibility.md) — C1, C2, C3 and the version matrix | +| **Why paths are XDG-based** | [`adr/0001-xdg-base-directories.md`](adr/0001-xdg-base-directories.md) | +| **Emitted JSON shapes** | [`output-schema.md`](output-schema.md) | +| **How to set up a dev environment** | [`../CONTRIBUTING.md`](../CONTRIBUTING.md) | + +### Non-goals + +- Restating Python or PEP conventions. Assume competence. +- Documenting individual functions. Docstrings own that. +- Being aspirational. Every rule below is either enforced today or listed in + [Known gaps](#known-gaps) as unenforced. A rule with no enforcer is a wish. + +## Module structure + +Six packages, 6,238 lines. `cli.py` is 2,351 of them — see [Known gaps](#known-gaps). + +| Package | Responsibility | +|---|---| +| `paths` | Every on-disk location tradekit reads or writes. The only module that may touch the home directory. | +| `config` | Settings model and `.env` resolution. Pydantic `BaseSettings`, layered. | +| `data` | Market data acquisition. One module per provider, plus the cache. | +| `analysis` | Pure computation over DataFrames — indicators, levels, patterns, scoring, setups, volume. | +| `screener` | Candidate selection: pre-market scan, filters, ranking. | +| `reports` | Rendering only — terminal, markdown, HTML, alerts, blotter. | +| `agents` | LLM-backed workflows: bull/bear debate, inference. | +| `cli` | Click entry points. The only user-facing surface. | + +### Import direction + +The dependency graph is acyclic and shallow. This is verified, not aspirational: + +``` +paths -> (nothing internal) +config -> paths +data -> config, paths +analysis -> config +screener -> analysis, config, data +reports -> config, data +agents -> paths +cli -> everything +``` + +**Rules:** + +1. **`paths` imports nothing from tradekit.** It is the floor. If `paths` needs + configuration, the design is wrong. +2. **`analysis` never imports `data`.** Analysis takes DataFrames in and returns + values out. This is what makes it testable on synthetic frames with no network, + no keys and no fixtures — the property that keeps `tests/test_analysis.py` + fast and hermetic. Breaking it is the single most expensive change you can + make to this codebase's testability. +3. **`reports` never imports `screener` or `analysis`.** Rendering receives + finished objects; it does not compute what to render. +4. **Nothing imports `cli`.** If a module needs something in `cli.py`, that + something is in the wrong file. +5. **No cycles.** Any new edge must preserve the order above. +6. **Import public names across package boundaries.** A `_leading_underscore` symbol belongs + to its own module. `reports/html.py:323` currently does + `from tradekit.data.finviz import _trade_review_day_dir`; the public + `paths.trade_review_day_dir` is what it wants. This is not a layering violation — `reports` + may import `data` — but it couples a renderer to another package's private helper. + +## The data provider contract + +`data/base.py` defines a `Protocol`, not a base class — providers are structurally +typed and inherit nothing: + +```python +class DataProvider(Protocol): + def get_quote(self, ticker: str) -> dict: ... + def get_history(self, ticker: str, period: str = "3mo", interval: str = "1d") -> pd.DataFrame: ... + def get_premarket(self, ticker: str) -> dict: ... +``` + +Current implementations: `yahoo` (default), `massive`, `backtest` (S3 flat files), +`finviz` / `finviz_elite`, `falcon` (read-only consumer of contract C1). + +**To add a provider:** + +1. New module under `data/`. Implement the three methods. Do not subclass anything. +2. `get_history` returns a DataFrame with a `DatetimeIndex` and the OHLCV columns + the analysis layer expects. Providers normalize; `analysis` must never contain + a provider-shaped special case. +3. Credentials come from `config`, never from a literal and never from a direct + `os.environ` read at module scope. +4. Register it for `Settings.data_source` selection. +5. Cache through `data/cache.py`. Do not invent a second cache. + +**Freshness is a provider property and must be stated.** A provider that returns +delayed data has to make that visible to callers. A quote whose lag is unknown at +the call site is how a 15-minute-old price ends up on a live order ticket. + +## Configuration + +Resolution order, widest to narrowest — later wins: + +1. Package defaults in `config.py` +2. Shared LifeOS `.env`, when present (`$PAI_DIR/.env` or `~/.claude/.env`) +3. Project-local `.env` +4. Process environment +5. Explicit CLI flags + +Three invariants, from `CONTRIBUTING.md` and kept here because they constrain code: + +- **One shared `.env`.** tradekit never requires a second copy of a key the + ecosystem already holds. +- **Standalone must work.** Zero LifeOS context — no `PAI_DIR`, no `~/.claude/.env` — + is a supported configuration, not a degraded one. +- **No personal data in defaults.** Account sizes, watchlists, broker identifiers + and machine-specific paths live in user-owned YAML or `.env`. This repository is + public; treat every default as published. + +### Connection parameters take no fallback default + +**Host, port, and credentials for a user-configured service resolve from +configuration or raise. They do not fall back.** + +```python +port = int(env["DAS_PORT"]) # correct — absent config fails loudly +port = int(env.get("DAS_PORT", "9910")) # wrong +``` + +The failure mode is specific and expensive: a wrong default produces a connection +refusal, and a connection refusal is **indistinguishable from the service being +down**. The caller reports an outage that is not happening, then routes around a +service that was working the whole time. + +This is not hypothetical. On 2026-08-28 a sibling tool in this suite probed a +factory-default port, reported "service down", and a downstream consumer silently +fell back to a 15-minute-delayed data source for a live workflow. + +**Scope of the rule.** It binds parameters a user configures for *their own* +environment. A well-known public service endpoint may carry a default — +`backtest_endpoint = "https://files.massive.com"` is fine, because it is the same +for everyone and a wrong value fails at DNS, not ambiguously at connect. + +## Paths + +`paths.py` is the single source of truth for on-disk layout. **No module outside +`paths.py` constructs a path from the home directory.** Import a function instead. + +Two categories, and the difference decides whether you may move a file: + +| Category | Meaning | Examples | +|---|---|---| +| **Owned** | tradekit may relocate freely | `data_dir()`, `cache_dir()`, `state_dir()`, `debate_dir()`, `group_snapshot()` | +| **Contract** | Shared with another repo; cannot move unilaterally | `falcon_db()` (C1), `trade_review_dir()` (C2) | + +Contract paths resolve both the XDG location and the legacy one, so suite +components can migrate independently. See +[`falcon-suite-compatibility.md`](falcon-suite-compatibility.md) for the contracts +themselves and [ADR 0001](adr/0001-xdg-base-directories.md) for why. + +## CLI + +`cli.py` is the only user-facing surface. Commands are Click functions registered +with `@cli.command()`. + +**The registration rule:** *if any changelog entry, README section, RPM spec, or +release note claims a command exists, `tradekit --help` must list it.* + +Note what this does **not** say. Not every module needs an entry point — tradekit is +importable as a library and internal modules are legitimate. The invariant is about +**advertised** surface. A feature that is documented but unreachable is worse than +one that was never shipped, because the documentation asserts otherwise. + +The RPM spec's `Commands:` list in `%description` is part of that advertised surface +and must match `--help`. + +## Version and release + +**`pyproject.toml` is the single source of truth for the version.** `release.yml` +already enforces this — it refuses to publish when the git tag disagrees with +`project.version`. + +Everything else derives or matches: + +| Location | Rule | +|---|---| +| `pyproject.toml` `project.version` | **Authoritative** | +| `src/tradekit/__init__.py` `__version__` | **Derived** via `importlib.metadata.version("tradekit")`. Never hand-maintained | +| `packaging/tradekit.spec` `Version:` | Must equal the tag `Source0` points at | +| Git tag `vX.Y.Z` | Must equal `project.version` | + +### Release checklist + +1. `CHANGELOG.md` — move `[Unreleased]` into a dated version section. +2. Bump `project.version` in `pyproject.toml`. +3. Bump `Version:` in `packaging/tradekit.spec`, reset `Release:` to `1%{?dist}`, + add a `%changelog` entry. +4. Reconcile the spec's `Requires:` against `pyproject.toml` dependencies, and its + `Commands:` list against `tradekit --help`. +5. Run the CI gates locally (see below). +6. Tag `vX.Y.Z` and push. `release.yml` verifies the tag and publishes to PyPI. +7. COPR build from the tagged tarball. + +Semantic versioning. A new command or provider is a minor bump; a changed output +schema or moved contract path is a major one. + +## Testing and CI + +`tests/` mirrors the package. Currently 528 lines across `test_analysis`, +`test_config`, `test_filters`, `test_paths`. + +**What must have a test:** every analysis function (pure, so there is no excuse), +every path resolver including its legacy fallback, and every screener filter +boundary. Providers may be integration-tested or skipped, but must never require +live credentials to *collect*. + +CI gates on Python 3.14: + +| Gate | Blocking? | +|---|---| +| `ruff check .` | **Yes** | +| `ruff format --check .` | **Yes** — note this formats Python **inside markdown fences too**, so `docs/*.md` is subject to it. `ruff check` is not; it skips markdown. Run `ruff format --check .` from the repo root before pushing docs | +| `pytest` | **Yes** | +| `mypy src/tradekit` | No — `continue-on-error` | +| Build wheel + sdist | **Yes** | + +mypy is advisory because there are currently **156 errors across 57 files**. The +honest near-term target is a ratchet — no *new* errors — not a flip to blocking, +which would stop every PR on day one. + +## Vendoring + +`finvizfinance` (1.3.0, MIT) and `ta` (0.11.0, MIT) are vendored at +`src/tradekit/_vendor/` and made importable by a `sys.path` hook in +`tradekit/__init__.py`. + +- Pinned versions live in `_vendor//VENDORED_VERSION`. +- The RPM spec declares them as `Provides: bundled(...)`. +- Vendored code is excluded from ruff and is never reformatted to this project's + style — it is an upstream copy, and local edits make the next sync unmergeable. +- **Exit condition:** when Fedora ships `python3-finvizfinance` or `python3-ta`, + drop the vendored copy and convert to a `Requires:`. Fedora Packaging Guidelines + forbid bundling except by explicit exception, so this is a hard constraint for + distribution inclusion even though it is a soft one upstream. + +## Security and privacy + +This repository is public. + +- No API keys, tokens, broker account identifiers, or credentials in source, + config defaults, test fixtures, or committed `.env` files. `.env` is gitignored; + keep it that way. +- No personal filesystem paths. Paths come from `paths.py`, which derives from XDG + environment variables. +- No account sizes, position sizes, or risk parameters in defaults. +- Suspected vulnerabilities follow [`SECURITY.md`](../SECURITY.md), not public issues. + +## Known gaps + +Recorded rather than hidden. Each is a rule above that nothing currently enforces. + +| # | Gap | Evidence | Fix | +|---|---|---|---| +| G1 | **`blotter` is documented but unreachable.** PR #2 advertises `tradekit blotter DATE`; `reports/blotter.py` ships; no `@cli.command()` registers it and `--help` does not list it. | `grep blotter src/tradekit/cli.py` returns nothing | Register the command, or remove the claim | +| ~~G2~~ | ~~**Version stated in four places, all disagreeing.**~~ **Closed 2026-08-29.** `__version__` now derives from installed metadata; `pyproject.toml`, `packaging/tradekit.spec` and `CHANGELOG.md` all read 0.3.0. | `pyproject`/`spec`/`CHANGELOG` all 0.3.0; `tradekit.__version__` derived | Remaining: a CI check asserting the three files still agree — otherwise this reopens (see G4) | +| G3 | **Import direction is documented, not enforced.** Nothing fails if `analysis` imports `data` tomorrow. | — | Add an import-direction test | +| G4 | **RPM `Requires:` and `Commands:` drift silently** from `pyproject.toml` and `--help`. | matplotlib was missing; four commands were unlisted | Add a CI check comparing spec metadata to the tree | +| G5 | **`cli.py` is 2,351 lines**, 38% of the codebase, and holds logic that belongs in `screener` and `reports`. | — | Extract incrementally; no new business logic in `cli.py` | +| G6 | **mypy: 156 errors across 57 files.** | `mypy src/tradekit` | Ratchet, do not flip | +| G7 | **`reports/html.py:323` imports a private helper** from `data.finviz` instead of the public `paths.trade_review_day_dir`. | `grep -n _trade_review_day_dir src/tradekit/reports/html.py` | Call the public function | +| G8 | **Account kind classification is broken; `main` is red.** `reporting/ingest.py:34` sets `DEFAULT_ACCOUNT_KINDS = {}` with the note *"Load account mappings from environment or config file instead of hardcoding"* — the hardcoding was removed (correctly, for a public repo) but the replacement loader was never written. `ingest.py:82` therefore falls back to `AccountKind.LIVE` for every account, so a SIM account is reported as LIVE. 3 tests in `tests/test_reporting.py` have been failing on `main` since. | `git checkout origin/main && pytest tests/test_reporting.py` → 3 failed | Implement the config-driven mapping the comment promises. Until then LIVE/SIM separation — the one distinction the risk model depends on — silently does not hold | + +G1 and G4 are the same failure in two places: a claim published without a check +that the claim is true. G3 and G2 are that failure waiting to happen. diff --git a/packaging/tradekit.spec b/packaging/tradekit.spec index 01281a4..45c565d 100644 --- a/packaging/tradekit.spec +++ b/packaging/tradekit.spec @@ -13,9 +13,12 @@ # at src/tradekit/_vendor/ because they are not yet packaged in Fedora repos. # When upstream packages land in Fedora, drop the bundling and convert to # Requires: lines. +# +# NOTE: Source0 resolves to the v%%{version} git tag. Tag the release before +# building — a Version here with no matching tag produces an unbuildable spec. Name: tradekit -Version: 0.1.0 +Version: 0.3.0 Release: 1%{?dist} Summary: Pre-market screening, technical analysis, and trade setup evaluation @@ -27,8 +30,7 @@ BuildArch: noarch BuildRequires: python3-devel BuildRequires: pyproject-rpm-macros -BuildRequires: python3-pip -# For %check +# For %%check BuildRequires: python3-pytest # Runtime deps (all available in Fedora 44 repos as of writing) @@ -36,6 +38,10 @@ Requires: python3 >= 3.14 Requires: python3-yfinance Requires: python3-pandas Requires: python3-numpy +# Required by tradekit.reports.blotter, which ships in the package and is imported +# by %%pyproject_check_import even though no @cli.command() currently exposes it +# (docs/SPEC.md gap G1). Do not drop this until that module is registered or removed. +Requires: python3-matplotlib >= 3.11.1 Requires: python3-pydantic-settings Requires: python3-mcp Requires: python3-pyarrow @@ -57,12 +63,18 @@ Provides: bundled(python3-ta) = 0.11.0 %description tradekit is a personal trading-infrastructure CLI for pre-market screening, technical analysis, sector/industry rotation snapshots, relative-strength -scoring, and bull/bear LLM-driven debate setups. It integrates with -Personal_AI_Infrastructure (PAI), Finviz Elite, the Falcon screener, and -S3-compatible flat-file market data backends. +scoring, and bull/bear LLM-driven debate setups. It integrates with LifeOS +(formerly Personal AI Infrastructure / PAI), Finviz Elite, the Falcon screener, +and S3-compatible flat-file market data backends. + +Commands: analyze, debate, gameplan, groups, init, levels, morning, news, +publish-review, regime, report, rs, sandwich, scan, second-day, watchlist, +weekly-review. -Commands include: scan, analyze, levels, watchlist, morning, report, gameplan, -groups, rs, debate, weekly-review, publish-review, init. +On-disk paths follow the XDG Base Directory specification — data under +$XDG_DATA_HOME, cache under $XDG_CACHE_HOME, state under $XDG_STATE_HOME. Two +paths are shared contracts with the rest of the Falcon suite and tolerate both +XDG and legacy locations; see docs/falcon-suite-compatibility.md. %prep %autosetup -p1 @@ -80,13 +92,16 @@ groups, rs, debate, weekly-review, publish-review, init. %pyproject_install %pyproject_save_files tradekit -# Bash completion (Click generates it from the registered CLI) -install -d -m 0755 %{buildroot}%{_sysconfdir}/bash_completion.d -_TRADEKIT_COMPLETE=bash_source %{buildroot}%{_bindir}/tradekit \ - > %{buildroot}%{_sysconfdir}/bash_completion.d/tradekit \ - 2>/dev/null || true +# Bash completion, generated by Click from the registered CLI. The entry point +# in %%{buildroot} cannot import the package without PYTHONPATH pointing at the +# not-yet-installed sitelib. +install -d -m 0755 %{buildroot}%{_datadir}/bash-completion/completions +PYTHONPATH=%{buildroot}%{python3_sitelib} \ + _TRADEKIT_COMPLETE=bash_source %{buildroot}%{_bindir}/tradekit \ + > %{buildroot}%{_datadir}/bash-completion/completions/tradekit -# Default config skeleton +# Config skeleton. This is an example, not live config — it is documentation, +# so it is not marked %%config and carries no noreplace semantics. install -d -m 0755 %{buildroot}%{_sysconfdir}/tradekit install -m 0644 packaging/tradekit.conf.example \ %{buildroot}%{_sysconfdir}/tradekit/tradekit.conf.example @@ -100,11 +115,15 @@ install -m 0644 packaging/tradekit-groups-daily.timer \ %check -# Smoke test: the CLI must at least produce help without exception -%{buildroot}%{_bindir}/tradekit --help >/dev/null -# Run pytest if any tests are present (best-effort) -%pyproject_check_import || : -[ -d tests ] && %pytest -q || : +# Every import in the package resolves against the installed tree. +%pyproject_check_import +# The CLI must produce help without raising. This catches an import-time error or a +# missing runtime dep, so it must be allowed to fail the build -- no `|| :`. +# It does NOT catch an unregistered command: `tradekit --help` exits 0 whether or not +# a given module is wired to a @cli.command(). See docs/SPEC.md gap G1. +PYTHONPATH=%{buildroot}%{python3_sitelib} \ + %{buildroot}%{_bindir}/tradekit --help >/dev/null +%pytest -q %files -f %{pyproject_files} @@ -113,14 +132,34 @@ install -m 0644 packaging/tradekit-groups-daily.timer \ %license src/tradekit/_vendor/ta/LICENSE %doc README.md CHANGELOG.md %{_bindir}/tradekit -%{_sysconfdir}/bash_completion.d/tradekit +%{_datadir}/bash-completion/completions/tradekit %dir %{_sysconfdir}/tradekit -%config(noreplace) %{_sysconfdir}/tradekit/tradekit.conf.example +%{_sysconfdir}/tradekit/tradekit.conf.example %{_userunitdir}/tradekit-groups-daily.service %{_userunitdir}/tradekit-groups-daily.timer %changelog +* Fri Aug 28 2026 David Duncan - 0.3.0-1 +- Sync spec with the tree after four merged PRs; it had been stale at 0.1.0. +- Add Requires: python3-matplotlib >= 3.11.1 for round-trip blotter rendering. +- Correct the command list: add news, regime, sandwich, second-day. +- Rebrand Personal_AI_Infrastructure -> LifeOS in the description. +- Document the XDG Base Directory layout and the Falcon-suite path contracts. +- Move bash completion from %%{_sysconfdir}/bash_completion.d to + %%{_datadir}/bash-completion/completions, the current Fedora location. +- Fix completion generation and the %%check smoke test: both invoked the + buildroot entry point without PYTHONPATH and could never have imported the + package. +- Stop masking %%check failures with `|| :`. A smoke test that cannot fail is + not a test. +- Drop the %%config(noreplace) marking from tradekit.conf.example; an example + file is documentation, not configuration. +- Drop the redundant BuildRequires: python3-pip (pyproject-rpm-macros pulls it). + +* Tue Jul 14 2026 David Duncan - 0.2.0-1 +- Session-anchored VWAP and the 15-Minute VWAP Sandwich detector. + * Sun May 03 2026 David Duncan - 0.1.0-1 - Initial Fedora packaging. - Vendored finvizfinance 1.3.0 (MIT) and ta 0.11.0 (MIT) at diff --git a/pyproject.toml b/pyproject.toml index 51bcc72..d6ee6b7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "tradekit" -version = "0.2.0" +version = "0.3.0" description = "Personal trading infrastructure for pre-market screening, technical analysis, and trade setup evaluation" readme = "README.md" license = { text = "MIT" } diff --git a/src/tradekit/__init__.py b/src/tradekit/__init__.py index 5ed2e73..6e61652 100644 --- a/src/tradekit/__init__.py +++ b/src/tradekit/__init__.py @@ -1,16 +1,23 @@ """tradekit — Personal trading infrastructure for pre-market screening and technical analysis.""" -# Make vendored deps importable as top-level packages (e.g. `import finvizfinance`, -# `import ta`). Vendored at src/tradekit/_vendor/ — pinned versions are listed in -# src/tradekit/_vendor//VENDORED_VERSION and surfaced as Provides: bundled(...) -# in the Fedora .spec. import os as _os import sys as _sys +from importlib.metadata import PackageNotFoundError as _PkgNotFound +from importlib.metadata import version as _pkg_version +# Make vendored deps importable as top-level packages (e.g. `import finvizfinance`, +# `import ta`). Vendored at src/tradekit/_vendor/ — pinned versions are listed in +# src/tradekit/_vendor//VENDORED_VERSION and surfaced as Provides: bundled(...) +# in the Fedora .spec. This must run before anything imports those names. _VENDOR = _os.path.join(_os.path.dirname(__file__), "_vendor") if _os.path.isdir(_VENDOR) and _VENDOR not in _sys.path: _sys.path.insert(0, _VENDOR) -del _os, _sys, _VENDOR +# Single source of truth for the version is pyproject.toml, read here through the +# installed distribution's metadata. Never hand-maintain a second copy. +try: + __version__ = _pkg_version("tradekit") +except _PkgNotFound: # source checkout with no install + __version__ = "0.0.0.dev0" -__version__ = "0.1.0" +del _os, _sys, _VENDOR, _pkg_version, _PkgNotFound diff --git a/uv.lock b/uv.lock index 79776d2..cd35276 100644 --- a/uv.lock +++ b/uv.lock @@ -1560,7 +1560,7 @@ wheels = [ [[package]] name = "tradekit" -version = "0.2.0" +version = "0.3.0" source = { editable = "." } dependencies = [ { name = "aiohttp" },