diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..46170c611 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,310 @@ +# Agent Guidelines for gramps-project/addons-source + +This document specifies rules and conventions that agents should follow when +making changes to this repository. + +For the normative addon rules — MUST / SHOULD / MAY, with the origin of +each rule (upstream PR, maintainer ruling, Mantis id) cited inline — see +[docs/addon-development/16-guidelines.md](docs/addon-development/16-guidelines.md) +from the Addon Development manual (PR 994). This document covers the +hands-on agent workflow around those rules. + +## Repository Overview + +`addons-source` holds the source code for third-party Gramps addons — +gramplets, views, tools, reports, importers/exporters, filter rules, database +backends, and other plugins that are not part of Gramps core. It is one of +three tightly-linked repositories: + +- **`gramps`** — Gramps core (the application these addons plug into). +- **`addons-source`** (this repo) — addon source code, developed here. +- **`addons`** — built `.addon.tgz` packages and listing JSON consumed by the + in-app Plugin Manager. Populated from this repo via `make.py`; not edited + by hand except by the maintainer doing a release build. + +There is no unified build or CI pipeline for the whole repo (the checked-in +`.travis.yml` is stale/unused) — each addon is largely independent. + +## Branch Model + +Branches are per Gramps major or feature release, e.g. `maintenance/gramps60`, +`maintenance/gramps61` — not one branch per patch release. **Almost all addon +work should target the maintenance branch matching the current released +Gramps version.** A `master` branch exists but is not currently used for +addon work. + +Pick the branch carefully: +- When creating a new addon or fixing an existing one, branch from + `origin/maintenance/gramps6X` (the current release line), not from a local + tracking branch that may carry unrelated WIP commits. +- When opening the PR, target that same branch on + `gramps-project/addons-source`. +- Before pushing, sync/rebase against upstream (`git pull --rebase`) so the + PR applies cleanly. + +All PRs must come from a personal fork, never from a branch pushed directly +to `gramps-project/addons-source` (see `CONTRIBUTING.md`, "Commit Your +Changes": "you want to put your changes into _your_ fork, _not_ the upstream +`gramps-project/addons-source` repository"). + +Weblate translation PRs are the one exception where commits must **not** be +squashed on merge; everything else can be merged/rebased normally. + +## Repository Layout + +Each addon is a single top-level directory, CamelCase-named to match its +Python import name (e.g. `FilterRules/`, `DataEntryGramplet/`). A directory +typically contains: + +- `AddonName.py` — the implementation. +- `AddonName.gpr.py` — the Gramps plugin registration file (required; see + below). +- `po/` — translation `.po` files, managed by `make.py`. +- `tests/` — optional unit tests (see Testing below). +- `MANIFEST` — optional; lists extra files (docs, data) to include in the + built package beyond the default `*.py`, `*.glade`, `*.xml`, `*.txt`, + `locale/*/LC_MESSAGES/*.mo`. +- `locale/` — generated `.mo` files; not something you hand-edit. + +Addon directories generally have **no `__init__.py`** (they rely on Python 3 +namespace packages) so that Gramps can add each one to `sys.path` at load +time and other addons can `import AddonName` directly. + +## Commands + +Addon packaging/translation tasks go through `make.py`, run from the repo +root. Point `GRAMPSPATH` at wherever *your* local Gramps checkout lives (it +defaults to `../../..`, i.e. it assumes `make.py` is being invoked from one +level inside a sibling-layout workspace — don't rely on that default, always +set it explicitly), and make sure that checkout has the matching branch +checked out too (e.g. `maintenance/gramps61` in core when building against +`gramps61` here). Setting `LANGUAGE=en_US.UTF-8` is no longer required on +Gramps v6.0 and later: + +```bash +GRAMPSPATH=/path/to/your/gramps python3 make.py gramps61 build AddonDirectory +``` + +Common subcommands (first positional arg is the branch/version tag, e.g. +`gramps61`): + +- `init AddonDirectory [lang]` — scaffold dirs / `.pot` for a new addon, or + an initial `po/-local.po`. +- `update AddonDirectory ` — refresh a language `.po` from the `.pot`. +- `compile [AddonDirectory|all]` — compile `.po` → `.mo`. +- `build [AddonDirectory|all]` — produce the downloadable `.addon.tgz`. +- `listing [AddonDirectory|all]` — generate/update the Plugin Manager + listing JSON. +- `clean AddonDirectory` — strip generated files (`locale/`, `*.pot`, etc.) + before committing. +- `manifest-check` — validate `MANIFEST` files. +- `as-needed` — build/list/clean only what's out of date, repo-wide. + +Unlike `GRAMPSPATH`, the `build`/`listing`/`check` subcommands write to a +**hardcoded relative path**, `../addons//...` — this isn't a +convention, it's how `make.py` itself resolves the output location, so it +only works if you've cloned the `addons` repo as a sibling of +`addons-source` (see `MAINTAINERS.md` for the recommended three-repo +workspace layout: `gramps` / `addons` / `addons-source` side by side). You +don't need that sibling checkout at all unless you're packaging/publishing +an addon. + +You do not need `make.py` just to write/test an addon's Python code — only +when packaging, translating, or updating the listing. + +## Testing + +Not every addon has tests, but when adding or fixing one, add regression +tests under `AddonName/tests/`, mirroring existing examples (`Form/tests/`, +`DataEntryGramplet/tests/`, `Sqlite/tests/`, `FilterRules/tests/`). + +Conventions here differ from Gramps core: + +- Test files are named `test_*.py` (pytest-style), **not** the `*_test.py` + suffix core Gramps uses — `unittest discover` here is invoked per-addon or + from the repo root without the core `-p "*_test.py"` pattern. +- Still use the `unittest` framework (`unittest.TestCase`), not `pytest` + APIs, even though filenames follow the `test_*.py` convention. +- `AddonName/tests/__init__.py` should be **empty**. Do not add + `gi.require_version(...)` pinning there or in individual test modules — + see GTK/GDK below. +- Because the addon directory has no `__init__.py`, test modules need a + `sys.path` hack to import it as a namespace package: + + ```python + ADDON_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + ``` + + then `from AddonName.addonmodule import SomeClass`. +- Build test data programmatically against a real (temp-dir) database rather + than relying on fixtures that don't exist in this repo: + + ```python + from gramps.gen.db.utils import make_database + from gramps.gen.db import DbTxn + + db = make_database("sqlite") + db.load(tempfile.mkdtemp(prefix="myaddon_")) + with DbTxn("build test db", db) as txn: + ... + ``` +- If a test needs `gramps.gen.filters.CustomFilters`, call + `reload_custom_filters()` **before** importing the `CustomFilters` name + from `gramps.gen.filters` — that function rebinds the module-level global + rather than mutating it in place, so importing the name first leaves you + with a stale `None`. + +### GTK / GDK version pinning + +The repo-root `tests/__init__.py` pins `gi.require_version("Gtk", "3.0")` and +`gi.require_version("Gdk", "3.0")` for the whole test suite (added in PR +#950). `gi.require_version` sets process-global state, so **individual addon +test files should not re-pin GTK/GDK** — that's now redundant. A new test +module that imports something pulling in `gramps.gui.*` (which happens at +import time for most GUI-facing addons) only needs a guard for hosts with no +PyGObject at all: + +```python +try: + import gi +except ImportError as err: + raise unittest.SkipTest("PyGObject not available: %s" % err) +``` + +Do not add `gi.require_version(...)` calls back into new test files — if you +see them in an addon you're touching, they're safe to remove as redundant +(see the `Themes` addon's `tests/__init__.py` cleanup for precedent), but +don't remove them from files you aren't otherwise changing. + +### Running tests + +```bash +export GRAMPS_RESOURCES=/path/to/gramps/build/share # built Gramps checkout +export GDK_BACKEND=- +python3 -m unittest AddonName.tests.test_something -v +``` + +Run from the `addons-source` repo root so the addon's namespace package +resolves. + +## Windows Development + +Per `CONTRIBUTING.md`, addons here are developed and tested primarily on +Linux; Windows and macOS are possible but far less thoroughly reviewed. +"Windows" itself covers two different things agents should not conflate: + +- **MSYS2** — the POSIX-like shell/toolchain environment Gramps's own + Windows installer is built with. A shell running inside MSYS2 behaves like + a Linux shell (`bash`, POSIX paths) and can run this repo's `make.py` and + `unittest` invocations normally. +- **Native Windows** — a bare `cmd.exe` or PowerShell host with no MSYS2 or + WSL layer underneath. The commands and paths used elsewhere in this + document (`make.py`, `python3 -m unittest ...`, the `GRAMPSPATH` / + `GRAMPS_RESOURCES` env vars) assume a POSIX shell and are not verified + against native `cmd`/PowerShell. + +If you are running as an agent on a native Windows host: + +- Reading, writing, and refactoring addon Python code is fine — the code + itself is portable. +- Do not run this repo's tests or `make.py` directly in `cmd`/PowerShell and + report the result as a real pass/fail — the tooling isn't validated there, + so a failure may be environmental noise rather than an actual regression. +- Ask the user to run the same command inside an MSYS2 shell, WSL, or a + Linux container/VM and share the output, rather than guessing at the + result yourself. + +If you're already inside MSYS2 or WSL, treat it as a POSIX shell and follow +the rest of this document (Commands, Testing) as written. + +## Code Style + +- Every new `.py` file needs the same GPL-2.0-or-later header with copyright + used throughout Gramps core. +- Group imports under comment-banner sections (`Standard Python modules`, + `GTK/Gnome modules`, `Gramps modules`), matching the style already used + across this repo (e.g. `FilterRules/isfamilyfiltermatchevent.py`). +- [Black](https://black.readthedocs.io/) formatting is not currently required + for addons — there's no checked-in CI workflow enforcing it here, though + that may change in the future. The existing codebase is largely + Black-formatted, so running `black ` on files you're already + touching is a reasonable default. +- Docstrings: concise; full Sphinx `:param:`/`:returns:` style only when the + parameters aren't already obvious from the signature. + +## Internationalization + +Addons manage their own translations, separately from Gramps core, though +both go through the same Weblate instance: strings already translated in +Gramps core are matched first, so those strings are excluded from the +Weblate _Addons_ component and don't need separate translation there. At the +top of an addon's main module: + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale + +try: + _trans = glocale.get_addon_translator(__file__) +except ValueError: + _trans = glocale.translation +_ = _trans.gettext +``` + +(`get_addon_translator` raises `ValueError` if the addon has no +`locale/` translations yet — fall back to the core translator rather than +letting the import crash.) Use `ngettext(singular, plural, n)` instead of a +single `_()` call for any string that counts a noun, for the same reason as +Gramps core: many languages have plural rules English doesn't, and only +`ngettext` lets gettext apply the target language's actual rule. + +## Plugin Registration (`.gpr.py`) + +Every addon needs an `AddonName.gpr.py` alongside `AddonName.py`: + +```python +register( + RULE, # or TOOL, GRAMPLET, REPORT, VIEW, GENERAL, IMPORT, EXPORT, ... + id="uniqueid", + name=_("Human-Readable Name"), + description=_("What it does."), + version="1.0.0", + gramps_target_version="6.1", + status=STABLE, # or UNSTABLE + fname="AddonName.py", + authors=["Author Name"], + authors_email=["author@example.com"], + maintainers=["Maintainer Name"], + maintainers_email=["maintainer@example.com"], + help_url="Addon:WikiPageName", + ... # PTYPE-specific attributes, e.g. ruleclass/namespace for RULE +) +``` + +`gramps_target_version` and `version` are required. `help_url` should point +at the addon's Gramps wiki page (`Addon:PageName`), not a GitHub URL. +`maintainers`/`maintainers_email` identify who currently maintains the +addon and may differ from `authors`/`authors_email` (the original author) +once upkeep passes to someone else. + +> **Do not manually change `version` in a PR.** The `MAJOR.MINOR.PATCH` +> version in `.gpr.py` is bumped automatically by the `addons` repo's +> packaging build (`make.py`) when a change is released — the patch +> component increments on its own. Hand-editing it in a source PR just +> creates spurious diffs and can conflict with what the release build +> assigns; leave it as-is unless you're intentionally doing a MAJOR/MINOR +> bump for a breaking addon change. + +## Bug Reports and Commit Messages + +Bugs against addons are tracked in two places: the shared +[Gramps Mantis BT](https://gramps-project.org/bugs/view_all_bug_page.php) +and, informally, the [Gramps Discourse forum](https://gramps.discourse.group/) +— many addon issues start as a discourse thread rather than a Mantis report. +Unlike Gramps core, this repo has no automated changelog tooling parsing +commit messages, so there's no required `Fixes #N` keyword syntax — just +describe *why* the change was made, and link whatever report (Mantis bug or +discourse URL) motivated it if one exists. Don't invent a Mantis bug number +if you only have a discourse link or user report — link what you actually +have. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 96a0c7c77..53971752f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,13 +23,21 @@ If you're looking for *existing* addons to install, see If you're looking to contribute to Gramps directly, see [Portal:Developers](https://gramps-project.org/wiki/index.php/Getting_started_with_Gramps_development). +This document is the contributor **workflow** guide: repository setup, the +development loop, the checklist, and the pull-request process. The full +technical reference is the in-repo +[Addon Development manual](docs/addon-development/README.md) — seventeen +pages from a first Gramplet through testing, debugging, packaging, and +cross-version compatibility. Where a section below has a deeper counterpart +in the manual, it links there instead of repeating it. + ## Table of Contents * [What Can Addons Extend?](#what-can-addons-extend) * [Overview of Writing an Addon](#overview-of-writing-an-addon) * [Develop Your Addon](#develop-your-addon) - * [Addons Source Code Repository](#addon-source-code-repository) + * [Addons Source Code Repository](#addons-source-code-repository) * [Addons Download Repository](#addons-download-repository) - * [Set Up a Github Account](#setup-a-github-account) + * [Set Up a Github Account](#set-up-a-github-account) * [Create Project Forks in Github](#create-project-forks-in-github) * [Set Up Addon Development Environment](#set-up-addon-development-environment) * [Gramps Repository](#gramps-repository) @@ -50,7 +58,7 @@ If you're looking to contribute to Gramps directly, see * [Review the Addon Checklist](#review-the-addon-checklist) * [Create a Pull Request](#create-a-pull-request) * [Commit Your Changes](#commit-your-changes) - * [Verify Your Addon Is Current](#verify your addon is current) + * [Verify Your Addon Is Current](#verify-your-addon-is-current) * [Push To Your Fork](#push-to-your-fork) * [Create the PR](#create-the-pr) * [Work Towards a Merge](#work-towards-a-merge) @@ -60,47 +68,21 @@ If you're looking to contribute to Gramps directly, see * [List Your Addon](#list-your-addon) * [Document Your Addon](#document-your-addon) * [Support Your Addon Through Bug Tracker](#support-your-addon-through-bug-tracker) -* [Maintain Your Addon Code as Gramps Evolves](#maintain-your-addon-code-as-gramps-evolves) +* [Maintain Your Addon Code as Gramps Evolves](#maintain-your-addon-as-gramps-evolves) * [Resources](#resources) * [Addon Development Tutorials and Samples](#addon-development-tutorials-and-samples) * [Addons External to Github](#addons-external-to-github) ## What Can Addons Extend? - -Addons for Gramps can extend the program in many different ways. You can -add any of the following [types](https://github.com/gramps-project/gramps/blob/master/gramps/gen/plug/_pluginreg.py) of addons: - -* **Importer** (IMPORT) - adds additional file format import options to Gramps -* **Exporter** (EXPORT) - adds additional file format export options to Gramps -* **[Gramplet](https://gramps-project.org/wiki/index.php/Gramps_Glossary#gramplet)** (GRAMPLET) - adds a new -interactive interface section to a Gramps view mode, which can be -activated by right-clicking on the dashboard View or from the menu -of the Sidebar/Bottombar in other view categories. -* **Gramps** [**View mode**](https://gramps-project.org/wiki/index.php/Gramps_Glossary#viewmode) (VIEW) - adds a -new view mode to the list of views available within a -[View Category](https://gramps-project.org/wiki/index.php/Gramps_Glossary#view) -* **[Map Service](https://gramps-project.org/wiki/index.php/[Map_Services)** -(MAPSERVICE) - adds new mapping options to Gramps -* **Plugin lib** (GENERAL) - libraries that provide extra functionality when -present; can add, replace and/or modify builtin Gramps options. -* **[Quickreport/Quickview](https://gramps-project.org/wiki/indew/Gramps_6.0_Wiki_Manual_-_Reports_-_part_8#Quick_Views)** (QUICKREPORT) - a view -that you can run by right-clicking on an object, or if a person quickview, -then via the Quick View Gramplet -* **[Report](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Reports_-_part_1)** (REPORT) - adds a new output report; this includes -**Website** that outputs a static genealogy website based on your Gramps -Family Tree data. -* **[Rule](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Filters#Add_Rule_dialog)** (RULE) - adds new -[filter](https://gramps-project.org/wiki/index.php/Gramps_Glossary#filter) -rules. New starting with Gramps 5.1. -* **[Tool](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Tools)** (TOOL) - adds a utility that helps process data from your family tree. -* **Doc creator** (DOCGEN) -* **Relationships** (RECALC) -* **Sidebar** (SIDEBAR) -* **[Database](https://gramps-project.org/wiki/index.php/Database_Backends)** -(DATABASE) - add support for another database backend. New starting with -Gramps 5.0. -* **Thumbnailer** (THUMBNAILER) New starting with Gramps 5.2 -* **Citation formatter** (CITE) New starting with Gramps 5.2 +Addons can extend Gramps at almost every plugin point: Importers and +Exporters, Gramplets, View modes, Map Services, plugin libraries (GENERAL), +Quickviews, Reports, filter Rules, Tools, document generators (DOCGEN), +relationship calculators, Sidebars, Database backends, Thumbnailers, and +Citation formatters. + +The full catalogue — with the registration constant, UI location, base +class, and per-kind notes for each — is in the manual: +[Addon Kinds](docs/addon-development/03-addon-kinds.md). ## Overview of Writing an Addon Writing an addon is fairly straightforward if you have a bit of Python @@ -110,11 +92,11 @@ general steps to writing and sharing your own addons are: * [Develop your addon](#develop-your-addon) * [Create a Gramps plugin registration file](#create-a-gramps-plugin-registration-file) - e.g., a file named ```my-addon.gpr.py``` * [Review the Addon Checklist](#review-the-addon-checklist) -* [Create a Pull Request for your addon](#create-pr) -* [Announce it on the Gramps forum](#announce-the-addon) - Let users +* [Create a Pull Request for your addon](#create-a-pull-request) +* [Announce it on the Gramps forum](#announce-your-addon) - Let users know it exists and how to use it. -* [Support it through the issue tracker](#support-it-through-issue-tracker) -* [Maintain the code](#maintain-the-code-as-gramps-continues-to-evolve) as +* [Support it through the issue tracker](#support-your-addon-through-bug-tracker) +* [Maintain the code](#maintain-your-addon-as-gramps-evolves) as the Gramps code continues to evolve We'll now expand on each of these steps. @@ -463,24 +445,24 @@ see [Writing a Plugin](https://gramps-project.org/wiki/index.php/Writing_a_Plugin) specifically. +The manual walks these end-to-end: +[Tutorials](docs/addon-development/02-tutorials.md) builds one addon per +kind, and [Fundamentals](docs/addon-development/04-fundamentals.md) covers +the cross-cutting basics every kind shares. ### Test Your Addon As You Develop -To test your addon as you develop, we recommend you copy your -```NewProjectName``` plugin into your Gramps user plugin directory -from your addon development directory, prior to testing. Or, just -edit in the Gramps user plugin directory until it is ready to publish, -then copy back to your addon development directory. Your installed Gramps -desktop application will search this folder (and subdirectories) for -any ```.gpr.py``` files, and add them to the plugin list. - -You can of course still use the ```git``` branch for your addon to store -intermediate steps and other work in progress. - -> #### Warning -> [Bug #10436](https://gramps-project.org/bugs/view.php?id=10436) -> Symbolic links to folders in the gramps plugin directory are not scanned, so -> you cannot just create a symbolic link pointing to your addon source tree; -> you will have to copy it. +To test your addon as you develop, copy your ```NewProjectName``` folder +into the Gramps user plugin directory and restart Gramps — plugin discovery +happens at startup. On Gramps 6.0, discovery does **not** follow symbolic +links ([Bug #10436](https://gramps-project.org/bugs/view.php?id=10436)), so +a physical copy is required; Gramps 6.1 and later follow symlinks, so you +can link your working tree in once and edit in place. + +The manual covers the full development loop — where addons live, the +restart cycle, and a first working Gramplet — in +[the overview](docs/addon-development/01-overview.md#where-addons-live), +and how to test logic without launching the GUI in +[Testing](docs/addon-development/07-testing.md). If you have code that you want to share between addons, you don't need to do anything special. Gramps adds each directory in which a ```.gpr.py``` @@ -490,123 +472,42 @@ should always make sure you name your addons with a name appropriate for Python imports. ### Addon Configuration -Some addons may want to have persistent data (data settings that remain -between sessions). You can handle this yourself, or you can use Gramps' -builtin configure system. - -At the top of the source file for your addon, you would do this: -``` - from config import config as configman - config = configman.register_manager("grampletname") - # register the values to save: - config.register("section.option-name1", value1) - config.register("section.option-name2", value2) - ... - # load an existing file, if one: - config.load() - # save it, it case it didn't exist: - config.save() -``` - -This will create the file ```grampletname.ini``` and put it in the same -directory as the addon. If the config file already exists, it remains intact. -The natural location for ```.ini``` files is in the directory in which -the addon is installed; using the main ```gramps.ini``` file for addon -preferences could potentially lead to a conflict between addons. Other -locations and file formats are possible. See -[The Gramps architect recommends leaving this decision to the addon developer](https://gramps.discourse.group/t/add-option-for-boolean-options-in-gramplet/6371/19). - -In the addon, you can then: -``` - x = config.get("section.option-name1") - config.set("section.option-name1", 3) -``` - -and when this code is exiting, you might want to save the config. In a -Gramplet that would be: -``` - def on_save(self): - config.save() -``` - -If your code is a system-level file, then you might want to save the -config in the Gramps system folder: -``` - config = configman.register_manager("system", use_config_path=True) -``` - -This is rare; most ```.ini``` files go into the plugins directory. - -In other code that might use this config file, you would do this: -``` - from config import config as configman - config = configman.get_manager("grampletname") - x = config.get("section.option-name1") -``` +Some addons want persistent settings that survive between sessions. Use +Gramps' builtin configuration manager (```configman```) rather than rolling +your own file handling — the addon's ```.ini``` file lands in the addon's +own directory, so it cannot conflict with ```gramps.ini``` or with other +addons ([the Gramps architect recommends leaving the location decision to +the addon developer](https://gramps.discourse.group/t/add-option-for-boolean-options-in-gramplet/6371/19)). + +The full pattern — registering keys, load/save, the rare +```use_config_path``` system-folder case, and reading another addon's +settings with ```get_manager``` — is in the manual: +[Fundamentals → Configuration and persistent settings](docs/addon-development/04-fundamentals.md#configuration-and-persistent-settings). ### Localization - -> #### Note -> These instructions will only work for Python strings. If you have a -> glade file, it will not get translated. - -For general help with translations for Gramps, see -[Coding for translation](https://gramps-project.org/wiki/index.php/Coding_for_translation). However, that will only use translations that come with Gramps, -or allow you to contribute translations to the Gramps core. To have your own -managed translations that will be packaged with your addon, you will need to -add a way to retrieve the translation. Add the following to the top of your -```NewProjectName.py``` file: +Wrap every user-visible string in ```_()```, and bind the addon's own +translation catalog at the top of each implementation module: ``` from gramps.gen.const import GRAMPS_LOCALE as glocale - = glocale.get_addon_translator(__file__).gettext -``` -Then you can use the standard "```_()```" function to translate phrases in -your addon. - -You can use one of a few different types of translation functions: + _ = glocale.get_addon_translator(__file__).gettext ``` - gettext - lgettext - ngettext - lngettext - sgettext -``` - -These are obsolete starting in Gramps 4.x; ```gettext```, ```ngettext```, and -```sgettext``` always return translated strings in Unicode for consistent -portability between Python2 and Python3. - -See the [Python documentation](http://docs.python.org/3/library/gettext.html#the-gnutranslations-class) for using ```gettext``` and ```ngettext```. The -"l" versions return the string encoded according to the -[currently set locale](http://docs.python.org/3/library/locale.html#locale.setlocale); -the "u" versions return Unicode strings in Python2 and are no longer available -in Python3. +Glade files are **not** extracted automatically — mark their strings +translatable and override the labels at runtime in Python. -The method ```sgettext``` should always be used; it is a Gramps extension -that filters out clarifying comments for translators, such as in -```_("Remaining names | rest")``` where "rest" is the English string that -we want to present and "Remaining names" is a hint for translators. +The full workflow — string marking rules, plural forms, context prefixes +(```_("Remaining names|rest")```), ```.pot```/```.po``` generation with +```make.py```, and Weblate — is in the manual: +[Internationalization](docs/addon-development/11-internationalization.md). ### Files Included in Addon Distribution -The process that creates the compressed tar file that the Gramps Download -Manager installs in Gramps to use your addon automatically includes the -following files: -``` - *.py - *.glade - *.xml - *.txt - locale/*/LC_MESSAGES/*.mo -``` -Starting with Gramp 5.0, if you have files other than those listed above, -you should create a ```MANIFEST``` file in the root of your addon folder -that lists the files (or pattern) to be added, one per line, like this -sample ```MANIFEST``` file: -``` - README.md - extra_dir/* - help_files/docs/help.html -``` +The build automatically packages ```*.py```, ```*.glade```, ```*.xml```, +```*.txt```, and ```locale/*/LC_MESSAGES/*.mo```. Anything else (a +```README.md```, help files, extra directories) needs a ```MANIFEST``` +file in the addon root, one file or glob pattern per line (Gramps 5.0+). + +The build flow, ```MANIFEST``` semantics, and what lands in the +```.addon.tgz``` are in the manual: +[Packaging → What build packages](docs/addon-development/12-packaging.md#what-build-packages). > #### TIP > Starting with Gramps 6.0 (and _only_ 6.0) translations can be done on @@ -629,280 +530,65 @@ takes this general form: [PTYPE](https://github.com/gramps-project/gramps/blob/master/gramps/gen/plug/_pluginreg.py#L76) values include: TOOL, GRAMPLET, REPORT, QUICKVIEW (formerly QUICKREPORT), IMPORT, EXPORT, DOCGEN, GENERAL, MAPSERVICE, VIEW, RELCALC, SIDEBAR, DATABASE, RULE, -THUMBNAILER, and CITE. - -ATTR depends on the PTYPE. +THUMBNAILER, and CITE. ATTR depends on the PTYPE. -You must include a ```gramps_target_version``` and addon ```version``` values. -```gramps_target_version``` should be a string of the form "X.Y" matching -a Gramps X (major) and Y (minor) version. ```version``` is a string of -the form "X.Y.Z" representing the version of your addon; X, Y, and Z should -all be integers. +You must include ```gramps_target_version``` (a string "X.Y" matching the +Gramps major and minor version the addon targets) and the addon +```version``` (a string "X.Y.Z"). Include author name(s) and email(s) as +arrays of strings, and — new in Gramps 5.2 — optionally ```maintainers``` +/ ```maintainers_email``` when the maintainer differs from the author; the +maintainer is the primary point of contact. -Be sure to include attributes for author name(s) and email(s) in the form -of an array of comma-separated strings. +In the ```.gpr.py```, the function ```_``` is predefined by the plugin +loader to use your locale translations — mark text with ```_("TEXT")```, +never import ```_``` there. -There is an additional set of attributes, ```maintainers``` and -```maintainers_email``` (new in Gramps 5.2). If you, the author, are also -the maintainer it will be identical to the author attributes, but you may -also designate a maintainer, in which case the maintainer will become the -primary point of contact. - -Here is a sample Tool GPR file: -``` - register(TOOL, - id = 'AttachSource', - name = _("Attach Source"), - description = _("Attaches a shared source to multiple objects."), - version = '1.0.0', - gramps_target_version = '6.0', - status = STABLE, - fname = 'AttachSourceTool.py', - authors = ["Douglas S. Blank"], - authors_email = ["doug.blank@gmail.com"], - maintainers = ["Douglas S. Blank"], - maintainers_email = ["doug.blank@gmail.com"], - category = TOOL_DBPROC, - toolclass = 'AttachSourceWindow', - optionclass = 'AttachSourceOptions', - tool_modes = [TOOL_MODE_GUI], - help_url = "Addon:AttachSourceTool" - ) -``` - -You can see examples of the kinds of addons -[here](https://github.com/gramps-project/gramps/plugins) (such as -[gramps-project/gramps/plugins/drawreport/drawplugins.gpr.py](https://github.com/gramps-project/gramps/plugins/drawreport/drawplugins.gpr.py)) -and see the full documentation in the -[master/gramps/gen/plug/_pluginreg.py][https://github.com/gramps-project/gramps/blob/3f0db9303f29811b43325c30149c8844c7ce24b6/gramps/gen/plug/_pluginreg.py#L23) -comments and docstrings. - -Note that this example ```.gpr.py``` will automatically use translations -if you have them (see [Localization](#localization)). That is, the -function "_" is predefined to use your -locale translations; you only need to mark the text with ```_("TEXT")``` -and include a translation of "TEXT" in your translation file. For example, -in the above example, ```_("Attach Source")``` is marked for translation. -If you have developed and packaged your addon with translation support, -then that phrase will be converted into the user's language. +The manual documents every registration field, the discovery model, and a +complete example per kind: +[Fundamentals → The .gpr.py registration file](docs/addon-development/04-fundamentals.md#the-gprpy-registration-file) +and [Addon Kinds](docs/addon-development/03-addon-kinds.md). ### Report Plugins -The possible report categories are -[gramps/gen/plug/_pluginreg.py](https://github.com/gramps-project/gramps/blob/892fc270592095192947097d22a72834d5c70447/gramps/gen/plug/_pluginreg.py#L141-L149): -``` - #possible report categories - CATEGORY_TEXT = 0 - CATEGORY_DRAW = 1 - CATEGORY_CODE = 2 - CATEGORY_WEB = 3 - CATEGORY_BOOK = 4 - CATEGORY_GRAPHVIZ = 5 - CATEGORY_TREE = 6 - REPORT_CAT = [ CATEGORY_TEXT, CATEGORY_DRAW, CATEGORY_CODE, - CATEGORY_WEB, CATEGORY_BOOK, CATEGORY_GRAPHVIZ, CATEGORY_TREE] -``` - -Each report category has a set of standards and an interface. The categories -```CATEGORY_TEXT``` and ```CATEGORY_DRAW``` use the Document interface of -Gramps. See also -[Report API](https://gramps-project.org/wiki/index.php/Report_API) -for a draft view on this. - -The application programming interface or API for reports is treated in the -[report writing tutorial](https://gramps-project.org/wiki/index.php/Report-writing_tutorial). For general information on Gramps development, see -[Portal:Developers](https://gramps-project.org/wiki/index.php/Portal:Developers) -and [Writing a Plugin](https://gramps-project.org/wiki/index.php/Writing_a_plugin). +A REPORT registration declares one of the report categories +(```CATEGORY_TEXT```, ```CATEGORY_DRAW```, ```CATEGORY_WEB```, and so on — +defined in ```gramps/gen/plug/_pluginreg.py```); the text and draw +categories use Gramps' Document interface. + +The manual covers the Report/ReportOptions pair, the docgen abstraction, +and a complete text-report walkthrough: +[Addon Kinds → REPORT](docs/addon-development/03-addon-kinds.md) and +[Tutorials → A text Report](docs/addon-development/02-tutorials.md#a-text-report). +See also the +[report writing tutorial](https://gramps-project.org/wiki/index.php/Report-writing_tutorial) +on the wiki. ### General Plugins -The plugin framework also allows you to create generic plugins for use. -This includes the ability to create libraries of functions, and plugins -of your own design. - -#### Example: A library of functions -In this example, a file named ```library.py``` will be imported at the -time of registration (i.e., any time Gramps starts): -``` - # file: library.gpr.py - - register(GENERAL, - id = 'My Library', - name = _("My Library"), - description = _("Provides a library for doing something."), - version = '1.0', - gramps_target_version = '6.0', - status = STABLE, - fname = 'library.py', - load_on_reg = True, - ) -``` - -You can access the loaded module in other code by issuing an -```import library``` as Python keeps track of files already -imported. However, the amount of useful code that you can run -when the program is imported is limited. You might like to have -the code do something that requires a ```dbstate``` or ```uistate object```, -but neither of these is available when just importing a file. - -If ```load_on_reg``` was not ```True```, this code would be unavailable -until manually loaded. There is no mechanism in Gramps to load ```GENERAL``` -plugins automatically. - -In addition to importing a file at startup, you can also run a single -function inside a ```GENERAL``` plugin, and it will be passed the -```dbstate```, the ```uistate```, and the plugin data. The function -must be called ```load_on_reg```, and take those three parameters: -``` - # file: library.py - - def load_on_reg(dbstate, uistate, plugin): - """ - Runs when plugin is registered. - """ - print("Hello World!") -``` - -Here, you could connect signals to the ```dbstate```, open windows, etc. - -Another example of what you can do with the plugin interface is to create -a general purpose plugin framework for use by other plugins. Here is the -basis for a plugin system that: - -* allows plugins to list data files -* allows the plugin to process all of the data files - -First, the ```gpr.py``` file: -``` - register(GENERAL, - id = "ID", - category = "CATEGORY", - load_on_reg = True, - process = "FUNCTION_NAME", - ) -``` - -This example uses three new features: - -* ```GENERAL``` plugins can have a category -* ```GENERAL``` plugins can have a load_on_reg function that returns data -* ```GENERAL``` plugins can have a function (called ```process```) which -will process the data - -If you (or someone else) create additional general plugins of this category, -and they follow your ```load_on_reg``` data format API, then they could be -used just like your original data. For example, here is an additional -general plugin in the ```WEBSTUFF``` category: -``` - # anew.gpr.py - - register(GENERAL, - id = 'a new plugin', - category = "WEBSTUFF", - version = '1.0', - gramps_target_version = '6.0', - data = ["a", "b", "c"], - ) -``` - -This doesn't have ```load_on_reg = True```, nor does it have an ```fname``` or -```process```, but it does set the data directly in the ```.gpr.py``` file. -Then, we have the following results: -``` - >>> from gui.pluginmanager import GuiPluginManager - >>> PLUGMAN = GuiPluginManager.get_instance() - >>> PLUGMAN.get_plugin_data('WEBSTUFF') - ["a", "b", "c", "Stylesheet.css", "Another.css"] - >>> PLUGMAN.process_plugin_data('WEBSTUFF') - ["A", "B", "C", "STYLESHEET.CSS", "ANOTHER.CSS"] -``` +```GENERAL``` is the escape hatch for plugin code that doesn't fit any +other kind: shared function libraries (imported at startup with +```load_on_reg = True``` and then available to other addons via a plain +```import```), and pluggable categories such as ```WEBSTUFF``` (narrative +website stylesheets) and ```Filters``` (filter-rule providers). + +The manual documents both uses and the full plugin-data API — the +```load_on_reg(dbstate, uistate, plugin)``` function form, the ```data``` +and ```process``` registration fields, and querying by category through +the plugin manager: +[Addon Kinds → GENERAL](docs/addon-development/03-addon-kinds.md). ### Registered GENERAL Categories -The following are examples of the published secondary plugin categories of -APIs of type ```GENERAL```. - -#### WEBSTUFF -A sample ```gpr.py``` file: -``` - # stylesheet.gpr.py - - register(GENERAL, - id = 'system stylesheets', - category = "WEBSTUFF", - name = _("CSS Stylesheets"), - description = _("Provides a collection of stylesheets for the web"), - version = '1.0', - gramps_target_version = '6.0', - fname = "stylesheet.py", - load_on_reg = True, - process = "process_list", - ) -``` - -Here is the associated program: -``` - # file: stylesheet.py - - def load_on_reg(dbstate, uistate, plugin): - """ - Runs when plugin is registered. - """ - return ["Stylesheet.css", "Another.css"] - - def process_list(files): - return [file.upper() for file in files] -``` - -#### Filters -For example, ```gpr.py```: -``` - register(GENERAL, - category="Filters", - ... - load_on_reg = True - ) -``` -And the actual plugin: -``` - def load_on_reg(dbstate, uistate, plugin): - # returns a function that takes a namespace, 'Person', 'Family', etc. - - def filters(namespace): - print("Ok...", plugin.category, namespace, uistate) - # return a Filter object here - - return filters -``` +The published ```GENERAL``` categories — ```WEBSTUFF``` and ```Filters``` +— with sample registrations and implementations, are covered in +[Addon Kinds → GENERAL](docs/addon-development/03-addon-kinds.md). ### List Your Addon Prerequistes -In your ```.gpr.py``` file, you can have a line like: -``` - ... - depends_on = ["libwebconnect"], - ... -``` - -which is a list of plug-in identifiers from other ```.gpr.py``` files. -This example will ensure that -[libwebconnect](https://gramps-project.org/wiki/index.php/Addon:Web_Connect_Pack#Prerequisites) -is loaded before your addon. If that ID can't be found, or you have a cycle -(a circular import), then your addons won't be loaded. The Gramps architect -summarizes this as: "The ```depends_on``` list is used to specify other plugins -which the plugin depends on. These will be installed automatically." - -Example code in the Addon:Web_Connect_Pack that references ```libwebconnect``` -prerequistes can be seen in -[addons-source/RUWebPack.gpr.py#L17](https://github.com/gramps-project/addons-source/blob/1304b65a7d758bfe17339c26260473ac3e9c4061/RUWebConnectPack/RUWebPack.gpr.py#L17). +```depends_on = ["libwebconnect"]``` in a ```.gpr.py``` lists the ids of +other plugins that must load first (they are installed automatically); +```requires_mod```, ```requires_gi```, and ```requires_exe``` (Gramps 5.2+) +declare Python-module, GObject-introspection, and executable prerequisites. -This allows common prerequisites to be shared between addons. Code can be -maintained in its own ```.gpr.py```/addon file instead of trying to -synchronize the maintenance of multiple copies across various silos. - -Additional requirements properties were implemented starting with the -Gramps 5.2 -[Registration Options](https://gramps-project.org/wiki/index.php/Gramplets_development#Register_Options) that provide for specifying plug-in preqrequisites: - -* For modules: [```requires_mod```](https://github.com/gramps-project/gramps/blob/0f8d4ecd429431b4df64910962f8764af9ff1766/gramps/gen/plug/_pluginreg.py#L689-L719) -* For GObject introspection: [```requires_gi```](https://github.com/gramps-project/gramps/blob/0f8d4ecd429431b4df64910962f8764af9ff1766/gramps/gen/plug/_pluginreg.py#L689-L719) -* For executables: [```requires_exe```](https://github.com/gramps-project/gramps/blob/0f8d4ecd429431b4df64910962f8764af9ff1766/gramps/gen/plug/_pluginreg.py#L689-L719) +Declaration rules — importable module names, verifying entries, version +pins — are in the manual: +[Fundamentals → Declaring dependencies](docs/addon-development/04-fundamentals.md#declaring-dependencies). ## Review the Addon Checklist Before you publish your new addon, review this checklist for completeness: @@ -915,6 +601,10 @@ Before you publish your new addon, review this checklist for completeness: * Has the help_url been changed from the GitHub repository to the wiki page? +The normative MUST / SHOULD / MAY rules a reviewer holds an addon to — +structure, runtime, testing, translation, and the contributor workflow — +are in the manual: [Guidelines](docs/addon-development/16-guidelines.md). + ## Create a Pull Request Once you have created your addon, built the ```.gpr.py``` registration file, and have tested it (you *did* test it, right?) so that you're sure @@ -1141,42 +831,30 @@ Now it is time to announce your addon to those who may not have heard about it yet. ### Gramps Forum -Join the [Gramps Forum](#https://gramps-project.org/wiki/index.php/Contact#Forum) if you have not already. Announce your addon to forum users with general -information on why you created it, what it does for the user, and how to use -it. +Join the [Gramps Forum](https://gramps.discourse.group/) if you have not +already. Announce your addon to forum users with general information on +why you created it, what it does for the user, and how to use it. ### Gramps Wiki Create an account on the -[Gramps Wiki](#https://gramps-project.org/wiki/index.php/Main_page) +[Gramps Wiki](https://gramps-project.org/wiki/index.php/Main_page) if you don't already have one. #### List Your Addon Add a short description of your addon to the Addons list in the wiki by -editing the current release listing: i.e., +editing the current release listing: i.e., [6.0_Addons](https://gramps-project.org/wiki/index.php/6.0_Addons), or if the addon is meant for a future release, [6.1_Addons](https://gramps-project.org/wiki/index.php/6.1_Addons) when available. Examine other addon entries when editing the wiki page, and refer to the -[Addon list legend]]](https://gramps-project.org/wiki/index.php/Addon_list_legend) -list legend]] to understand the meaning of each column. When ready, use the -following template to include your addon in the list: -``` -|- -| -| -| -| -| -| -| -| -|- -``` +[Addon list legend](https://gramps-project.org/wiki/index.php/Addon_list_legend) +to understand the meaning of each column. The row template to copy is in +the manual: [Community → List your addon](docs/addon-development/13-community.md#list-your-addon). #### Document Your Addon -Document your addon in the wiki using the page name format -**Addon:NewProjectName**. Examine some of the other addon documentaion +Document your addon in the wiki using the page name format +**Addon:NewProjectName**. Examine some of the other addon documentation pages for suggestions, and for the general format to use. > ##### TIP @@ -1185,30 +863,9 @@ pages for suggestions, and for the general format to use. > results page you will be provided with a link to create the new page. > Select that link to add your content. -Consider including the following information in your wiki page: - -``` - -{{Third-party plugin}} - - -== Usage == - -=== Configure Options === - -==Features== - -== Prerequisites == - -== Issues == - - -[[Category:Addons]] -[[Category:Plugins]] -[[Category:Developers/General]] -``` +The conventional page skeleton (the ```{{Third-party plugin}}``` banner and +the standard sections) is in the manual: +[Community → Document your addon](docs/addon-development/13-community.md#document-your-addon). ## Support Your Addon Through Bug Tracker Create a user account on the @@ -1245,6 +902,12 @@ often before being officially accepted. core, but are loved by many users (e.g., the Data Entry Gramplet). * A place for experimental components to live. +The technical side of keeping an addon working across Gramps releases — +```gramps_target_version``` semantics, the per-release deltas that bite +ports, and the porting checks — is in the manual: +[Compatibility](docs/addon-development/14-compatibility.md) and +[What's New](docs/addon-development/15-whats-new.md). + ### Examples of Common Enhancements And here are just some of the kinds of enhancements that might make sense: @@ -1289,6 +952,8 @@ Also you may want to [[Addons_development#Package_your_addon |Package your addon --> ## Resources +* [Addon Development manual](docs/addon-development/README.md) — the +in-repo technical reference this document links throughout. * [Brief introduction to Git](https://gramps-project.org/wiki/index.php/Brief_introduction_to_git) * [Getting started with Gramps development](https://gramps-project.org/wiki/index.php/Getting_started_with_Gramps_development) * [Portal:Developers](https://gramps-project.org/wiki/index.php/Portal:Developers) @@ -1301,6 +966,8 @@ Also you may want to [[Addons_development#Package_your_addon |Package your addon * For 4.1.x and earlier, see [Addons development old](https://gramps-project.org/wiki/index.php/Addons_development_old). ## Addon Development Tutorials and Samples +* [Tutorials](docs/addon-development/02-tutorials.md) — in-repo end-to-end +walkthroughs, one per addon kind (Gramplet, Tool, Report, Quick View, Rule). * [Develop an Addon Gramplet](https://gramps-project.org/wiki/index.php/Gramplets_development) (or add a [custom filtering option](https://gramps.discourse.group/t/looking-for-an-example-of-a-gramplet-with-a-custom-filter-configuration-option/5967)) * [Develop_an_Addon_Rule](https://gramps-project.org/wiki/index.php/Develop_an_Addon_Rule) for custom filters * [Develop_an_Addon_Tool](https://gramps-project.org/wiki/index.php/Develop_an_Addon_Tool) diff --git a/GrampsWebApiDb/MANIFEST b/GrampsWebApiDb/MANIFEST new file mode 100644 index 000000000..1cb06e6ea --- /dev/null +++ b/GrampsWebApiDb/MANIFEST @@ -0,0 +1 @@ +GrampsWebApiDb/README.md diff --git a/GrampsWebApiDb/README.md b/GrampsWebApiDb/README.md new file mode 100644 index 000000000..17d4dad87 --- /dev/null +++ b/GrampsWebApiDb/README.md @@ -0,0 +1,103 @@ +GrampsWebApiDb is a Gramps database backend that uses a Gramps Web API +server (e.g. gramps-connect or Gramps Web) as a live database, mirrored +locally in SQLite for speed. Reads are served from the local mirror, which +is kept current via the server's transaction-history feed -- both at load +time and on an ongoing poll while the tree stays open, so a change made +from another client (the web app, another desktop instance) shows up here +without closing and reopening the tree; local edits are pushed back to the +server as they're committed. Every already-open Gramps view (People, +Families, ...) refreshes itself automatically as synced changes land, the +same as it would for a local edit -- see `grampswebapidb.py`'s module +docstring for how. The initial sync when opening a tree reports real +progress through Gramps' own load-progress bar, not just a spinning +cursor. + +## Credentials + +The addon takes a single credential, via the `GRAMPS_WEB_API_KEY` +environment variable, shaped `*`. There is +deliberately no login dialog wired into WebApiDB itself, and no per-tree +settings.ini. Generate one once via username/password. + +The easiest way is the **Generate Gramps Web API key** tool (this addon +also installs `mintapikeytool.py`/`mintapikeytool.gpr.py`): open it from +Tools → Utilities → Generate Gramps Web API key, enter the server URL, +username, and password, and click **Generate API Key**. Gramps only shows +the Tools menu once *some* Family Tree is open -- it doesn't have to be a +WebApiDB one, even an empty local tree works, so open (or create) one +first if you don't already have one open. On success the tool sets +`GRAMPS_WEB_API_KEY` in the +running Gramps process's environment, so a WebApiDB-backed Family Tree +can be opened right away without restarting Gramps -- but that only lasts +for this process; it is not written to a shell profile, settings.ini, or +any open Family Tree. Copy the displayed key into your shell's startup +file too if you want it set automatically next time. + +Click **Create Synced Family Tree for this key** to also create a new, +empty Family Tree for that key's account, using the `grampswebapidb` +database backend and already named correctly (see "Family Tree naming" +below) -- equivalent to creating one by hand via Family Trees → Manage +Family Trees, just with the name and backend filled in for you. It +creates the tree but does not open it; open it from Family Trees → +Manage Family Trees afterward to start syncing. + +Alternatively, generate one from the command line with the standalone +`gramps-api-client` package (not yet published; pip-installable from +its own repo, e.g. `pip install -e path/to/gramps-api-client`): + +```bash +export GRAMPS_WEB_API_KEY=$(gramps-api-client generate-key --url https://your-server/api --username youruser) +``` + +or from Python, using either that package's `Client.mint_api_key(url, +username, password)` or this addon's own vendored copy, +`WebApiHandler.mint_api_key(url, username, password)` (see +`webapi_client.py`) — same method, same result, no addon-specific +dependency either way. + +**Security tradeoff:** the token embedded in `GRAMPS_WEB_API_KEY` is a +standard JWT *refresh* token obtained from the server's normal `/token/` +login endpoint — the same endpoint and flow the official web client uses, +not an undocumented or exploited access path. gramps-web-api leaves refresh +tokens non-expiring by default, so this key is a long-lived, general-purpose +credential carrying the full permissions of the account that minted it. It +is *not* the same as a real scoped, independently revocable personal access +token (gramps-web-api has that machinery, but it isn't generally wired into +request auth yet). Practically, that means: + +* A leaked `GRAMPS_WEB_API_KEY` is as damaging as a leaked password — it + grants full account access until the underlying password is changed. + There is no "revoke this key" action independent of that. +* Treat it accordingly: don't commit it, don't log it, and store it the + same way you'd store a password. + +This is a documented engineering tradeoff, made because the properly-scoped +alternative isn't available server-side today — not a vulnerability in +gramps-web-api or a loophole being exploited. + +## Family Tree naming + +Because credentials come from an environment variable rather than a +per-tree setting, nothing else ties a Family Tree's local mirror to one +particular server account. Gramps must therefore name each Family Tree +using this backend `@` for the account `GRAMPS_WEB_API_KEY` +authenticates as — e.g. `dblank@hadaly.duckdns.org`. Opening a Family Tree +whose name doesn't match the currently-set `GRAMPS_WEB_API_KEY` fails to +load rather than silently mixing that account's data into a mirror synced +from a different one. To connect to a different server or account, create +a new Family Tree named accordingly rather than reusing an existing one. + +Gramps' own Family Tree Manager silently replaces characters like `.` with +`_` in any name you type (it needs the name safe to use as a filename), so +a hostname's dots never survive intact — name the tree +`dblank@hadaly_duckdns_org`, not `dblank@hadaly.duckdns.org`. The error +dialog shown for a mismatch always spells out the exact typeable name to +use. + +## See also + +* `grampswebapidb.py` for the sync/write-through design (module docstring). +* `webapi_client.py` for the token fetch/refresh implementation. This is a + hand-synced vendored copy (see its own docstring) -- the canonical, + standalone source is the `gramps-api-client` package, which also + has the `generate-key` CLI referenced above. diff --git a/GrampsWebApiDb/grampswebapidb.gpr.py b/GrampsWebApiDb/grampswebapidb.gpr.py new file mode 100644 index 000000000..22dc04480 --- /dev/null +++ b/GrampsWebApiDb/grampswebapidb.gpr.py @@ -0,0 +1,37 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +register( + DATABASE, + id="grampswebapidb", + status=BETA, + name=_("GrampsWebApiDb"), + name_accell=_("Gramps _Web API Database"), + description=_( + "Use a Gramps Web API server (e.g. gramps-connect or Gramps Web) " + "as a live database, mirrored locally for speed." + ), + version = '0.1.2', + gramps_target_version="6.0", + fname="grampswebapidb.py", + databaseclass="WebApiDB", + authors=["Doug Blank"], + authors_email=["doug.blank@gmail.com"], + help_url="Addon:GrampsWebApiDb", +) diff --git a/GrampsWebApiDb/grampswebapidb.py b/GrampsWebApiDb/grampswebapidb.py new file mode 100644 index 000000000..b99b67eba --- /dev/null +++ b/GrampsWebApiDb/grampswebapidb.py @@ -0,0 +1,734 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Database backend that mirrors a Gramps Web API server locally. + +Design +------ +This subclasses the stock SQLite DBAPI backend rather than DbReadBase / +DbWriteBase directly. DbGeneric (gramps.gen.db.generic) already implements +every get_*_from_handle / iter_* / get_number_of_* method generically on +top of a small Connection-like object (execute/fetchone/fetchall/commit/ +table_exists/...) -- see SQLite in gramps/plugins/db/dbapi/sqlite.py. So +reads only need a local, fast, complete SQLite mirror; nothing above the +Connection layer needs reimplementing. + +The mirror is kept current via GET /api/transactions/history/?after=, +the same per-object transaction log gramps-web-api's own undo system uses +(gramps_webapi/undodb.py's DbUndoSQLWeb.get_transactions()). Confirmed +against a live server: each entry is a *transaction* dict with a nested +"changes" list, each change carrying obj_class ("Person", "Family", ...), +trans_type (TXNADD=0/TXNUPD=1/TXNDEL=2), obj_handle, and -- when the +"new" query param is set -- new_data, a "_class"-tagged dict in the same +shape gramps.gen.lib.json_utils.data_to_object() reconstructs objects +from (it's literally what the server's own object_to_data(obj) produced +when the change was committed). So syncing is: remember the timestamp of +the last transaction applied, ask for everything after it, and for each +change either data_to_object(new_data) + commit_() (add and update +both being upserts, no need to distinguish) or remove_() for a +delete. + +This "_class"-tagged new_data shape is only produced by gramps-web-api +servers running against Gramps >= 6.0; a server still on Gramps 5.2 (e.g. +gramps-web-api itself untouched) serializes objects differently (no +"_class"/"value"/"string" triplet on GrampsType-derived fields), and +data_to_object() raises KeyError on it. Confirmed against a live gramps52 +server: read-only endpoints (auth, /trees/, /people/ counts, etc.) work +fine, but _sync_from_server() cannot deserialize its transaction history. + +Credentials come from a single environment variable, GRAMPS_WEB_API_KEY +(see webapi_client.py for its "*" shape and +the tradeoffs of using a refresh token here rather than a real scoped +personal-access-token). There is deliberately no per-tree settings.ini and +no login dialog: the same env var also works as a bare SDK credential +(WebApiHandler.from_env()) for scripts that talk to the server directly, +without going through Gramps at all -- one credential, two consumers. + +Because of that, nothing but the Family Tree's own name ties its local +mirror to one particular server account. _check_identity() requires that +name to be "@" (modulo Gramps' own filename-safe-character +substitution on tree names, e.g. dots -> underscores -- see +_FAMILY_TREE_NAME_UNSAFE_CHARS) for whoever GRAMPS_WEB_API_KEY currently +authenticates as, checked on every load() -- so pointing the env var at a +different account while reopening the same Family Tree fails loudly +instead of quietly mixing that account's data into the old mirror. + +Write-through (local edits pushed back to the server) hooks +transaction_commit() rather than the individual commit_person/ +commit_family/... methods: DbTxn.__exit__ calls self.db.transaction_commit +(gramps/gen/db/txn.py) exactly once per completed local transaction, and +DbTxn already accumulates every add/update/delete in that transaction via +its own get_recnos()/get_record() -- transaction_to_json() below turns +that into the flat {type, handle, _class, old, new} list POST +/transactions/ expects (confirmed against base.py's own POST /people/ +handler, which builds its response the same way). This must run *before* +super().transaction_commit(), since DBAPI.transaction_commit() clears the +transaction's records as its last step. + +The other place a DbTxn gets used is _sync_from_server() itself, applying +server-pulled changes -- that uses batch=True, and DBAPI._commit_base() +skips trans.add() entirely for batch transactions (see dbapi.py), so +transaction_to_json() naturally sees nothing there and no push happens. +No separate "am I currently syncing" flag is needed to stop synced +changes from being echoed straight back to the server. + +_sync_from_server() can only replay what the history feed actually +logged, and a batch=True commit -- any bulk import, merge, or tool run +through gramps-web-api, not just a one-off -- logs nothing per-object: +DBAPI's own commit_*/remove_* methods guard their trans.add() undo-log +call with `if not trans.batch`, so a batch transaction leaves behind an +empty-changes marker (a real Transaction row, but with no Change rows) +instead of the usual per-object entries. Confirmed live: bulk-importing +example.gramps produced exactly one such marker, and the 2157 people it +added were otherwise invisible to this addon's sync no matter how often +it resynced, because the transaction history itself never recorded +them. _sync_from_server() treats an empty-changes transaction as a +signal that its history-replay approach cannot describe what happened, +and falls back to _full_resync() -- downloading the server's current +full Gramps XML export and reimporting it into a wiped local mirror, +the only way to recover completeness when the incremental feed has a +blind spot by construction. + +Pushes go out without force=1, so the server compares each item's "old" +snapshot against its own current data and rejects the whole batch with +WebApiPushConflict (see webapi_client.push_transaction()) if anything +changed server-side since the local mirror last synced -- a real, if +coarse, optimistic-concurrency check: the whole push either applies or +none of it does, with no indication of which item conflicted. On a +conflict, _push_payload() below resyncs from the server (so the local +mirror picks up whatever changed) and then, for a plain commit (not an +undo/redo -- see _retry_after_conflict()), replays each object's intended +*new* state as a fresh local edit via commit_()/remove_() on +top of that just-resynced data. That fresh edit goes through the normal +transaction_commit() -> _push_payload() path again with is_retry=True, +so it carries an up-to-date "old" snapshot and will only be rejected a +second time if something changes server-side in the brief window between +the resync and the retry -- in which case it is logged and dropped rather +than retried again, to avoid retrying forever against a genuinely hot +object. + +For an add/update whose handle still exists after the resync (i.e. the +conflicting server-side edit changed the same object rather than deleting +it), _merge_or_overwrite() below combines the two edits with the object's +own merge() -- the same list-unioning logic behind Gramps' Merge People/ +Family/... tools (ported from GrampsWebSync's diffhandler.py, credit +David Straub, same license) -- rather than letting the retry blindly +clobber whatever the other side changed. merge() only unions *list*-valued +fields (notes, citations, media, urls, event/family refs, ...); it never +touches scalar fields (a name, a date, a gender), so two edits to the +exact same scalar field still resolve as local-overwrites-remote -- real +field-level conflict *resolution* for that narrower case (diff, prompt the +user) is still out of scope. If the push fails for a non-conflict reason +(network error, auth failure), the local commit has already happened and +is not rolled back -- the local mirror just drifts from the server until +the next successful push or read sync. + +The mirror stays current while the tree is open, not just at load() time: +load() also schedules a GLib.timeout_add_seconds() tick (POLL_INTERVAL_SECONDS) +that re-runs _sync_from_server() for as long as the database stays open -- +the same timestamp-cursor poll gramps-connect's browser client uses against +this same endpoint (see gramps-connect's store/historyPoll.ts), so a change +made from any other client shows up here without closing and reopening the +tree. It runs synchronously on the GTK main thread (like the initial +load()-time sync already did, and like viewmanager.py's own autobackup +timer) rather than on a background thread -- correct but simple, at the +cost of a brief UI pause during each poll's network round trip; moving it +off-thread (GrampsWebSync's GLibTaskRunner is the precedent, not imported +here for the same no-cross-addon-dependency reason as transaction_to_json() +below) is a reasonable future improvement, not attempted here. close() +cancels the pending timeout so a closed database doesn't keep polling. + +_sync_from_server()'s replay runs inside a batch=True DbTxn deliberately +(see the write-through section below for why), but that has a side effect +beyond suppressing trans.add(): DBAPI.transaction_commit() only emits its +person-add/family-update/event-delete/... signals `if not transaction.batch` +(see dbapi.py), so a batch replay is otherwise invisible to every +already-open GTK view -- the local mirror would update on disk with nothing +on screen changing. _emit_change_signals() reproduces just that signal half +by hand, once per synced page, using the exact same +KEY_TO_NAME_MAP[key] + {"add"/"update"/"delete"} signal names DBAPI itself +emits for a normal (non-batch) local edit -- so every view refreshes exactly +the way it already knows how to for a local change, with no new view-side +code needed. Collapsed to one signal per (obj_class, handle) -- the net +effect across everything applied in that page, so e.g. an update +immediately followed by a delete of the same object only fires the delete +signal, not both. + +A _full_resync() (see below) is the one path that doesn't go through +_emit_change_signals(): a full wipe-and-reimport is exactly the "too much +changed to describe incrementally" case DbGeneric's own request_rebuild() +exists for (it emits a single -rebuild signal per object type, +telling every view to reload wholesale rather than replay a specific +add/update/delete) -- so _full_resync() calls that once after a successful +reimport instead. + +Undo/redo integration hooks undo()/redo() the same way transaction_commit() +hooks commits: Gramps core's own DbGenericUndo._undo()/_redo() +(gramps/gen/db/generic.py) revert the local mirror directly via low-level +_txn_begin()/undo_data()/_txn_commit() calls that never go through +transaction_commit(), so without this override a local Undo/Redo would +silently desync the server -- worse than a push conflict, since nothing +would even be logged. The fix reuses transaction_to_json() on the DbTxn +DbGenericUndo already stores in its undo/redo queues (the same object +transaction_commit() turned into a payload the first time), then pushes +it again: undo() sends it to POST /transactions/?undo=1, where the server +reverses it itself (swaps old/new, add<->delete -- see +gramps_webapi/api/resources/util.py's reverse_transaction()); redo() just +pushes the original forward payload again, no different from a fresh +commit. Both go through the same conflict-detection/resync path as a +normal commit. Gramps' own undo history is in-memory/per-session, not +persisted, so this only ever matters within a single running session. +""" + +import logging +import os +import re +from copy import deepcopy +from tempfile import NamedTemporaryFile +from urllib.error import HTTPError, URLError + +from gi.repository import GLib + +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.db import DbTxn +from gramps.gen.db.dbconst import ( + CLASS_TO_KEY_MAP, + KEY_TO_CLASS_MAP, + KEY_TO_NAME_MAP, + TXNADD, + TXNDEL, + TXNUPD, +) +from gramps.gen.db.exceptions import DbConnectionError +from gramps.gen.lib.baseobj import BaseObject +from gramps.gen.lib.json_utils import data_to_object, remove_object +from gramps.gen.user import User +from gramps.plugins.db.dbapi.sqlite import SQLite +from gramps.plugins.importer.importxml import importData + +from webapi_client import WebApiHandler, WebApiPushConflict + +try: + _trans = glocale.get_addon_translator(__file__) +except ValueError: + _trans = glocale.translation +_ = _trans.gettext +LOG = logging.getLogger("grampswebapidb") + +#: How many transactions to request per page while syncing. +SYNC_PAGE_SIZE = 100 + +#: How often (seconds) load() re-polls the server for as long as the +#: database stays open -- see the module docstring's note on why this runs +#: synchronously on the GTK main thread rather than a background timer. +POLL_INTERVAL_SECONDS = 10 + +#: Failure modes from WebApiHandler.from_env()/push_transaction(): a +#: malformed/missing key (ValueError), a bad server response shape +#: (KeyError/JSONDecodeError, the latter a ValueError subclass), or the +#: server being unreachable (HTTPError/URLError/OSError -- socket.timeout +#: is an OSError subclass). +_CONNECTION_ERRORS = (ValueError, KeyError, HTTPError, URLError, OSError) + + +def _describe_connection_error(err): + """ + Turn a _CONNECTION_ERRORS exception into DbConnectionError's message + body. A 403 means the account GRAMPS_WEB_API_KEY authenticates as was + correctly identified but isn't allowed to do this -- worth calling out + specifically, since the raw HTTPError text ("HTTP Error 403: + Forbidden") reads like an auth failure rather than a permissions one. + """ + if isinstance(err, HTTPError) and err.code == 403: + return _( + "The account authenticating via GRAMPS_WEB_API_KEY does not " + "have permission on the server for this operation (HTTP 403 " + "Forbidden). Generate a key for an account with sufficient " + "permissions, or ask the server administrator to grant this " + "one access." + ) + return str(err) + +#: Same substitution gramps.gui.dbman's Family Tree Manager applies to +#: whatever a user types renaming a tree (dbman.py's __change_name(): "kill +#: special characters so can use as file name in backup"). A hostname- +#: bearing name can't survive that GUI round-trip with its dots intact, so +#: _check_identity() normalizes through this same substitution on both +#: sides before comparing -- see that method. +_FAMILY_TREE_NAME_UNSAFE_CHARS = re.compile(r"[':<>|,;=\"\[\]\.\+\*\/\?\\]") + +_TRANS_TYPE_NAME = {TXNADD: "add", TXNUPD: "update", TXNDEL: "delete"} + +#: Same signal-name suffixes DBAPI.transaction_commit() uses (dbapi.py's +#: own `action` dict) -- see _emit_change_signals(). +_TRANS_TYPE_ACTION = {TXNADD: "-add", TXNUPD: "-update", TXNDEL: "-delete"} + + +def transaction_to_json(transaction): + """ + Build the flat change-list payload POST /transactions/ expects, from + a just-committed local DbTxn. Ported from GrampsWebSync's + webapihandler.transaction_to_json (same repo, same license, credit + David Straub) instead of imported, for the same no-cross-addon- + dependency reason as webapi_client.py. + """ + out = [] + for recno in transaction.get_recnos(reverse=False): + key, action, handle, old_data, new_data = transaction.get_record(recno) + obj_cls_name = KEY_TO_CLASS_MAP.get(key) + if obj_cls_name is None: + continue # reference-type record, not a primary object + out.append( + { + "type": _TRANS_TYPE_NAME[action], + "handle": handle, + "_class": obj_cls_name, + "old": None if old_data is None else remove_object(old_data), + "new": None if new_data is None else remove_object(new_data), + } + ) + return out + + +def _merge_or_overwrite(current, local_obj): + """Combine local_obj's content into current via the object's own + merge() -- the same list-unioning logic behind Gramps' Merge People/ + Family/... tools (ported from GrampsWebSync's diffhandler.py, credit + David Straub, same license) -- when the type actually implements it. + + Falls back to local_obj outright for a type (e.g. Tag) that only + inherits BaseObject's no-op merge(): "merging" into a no-op would + silently keep current's content and discard the local edit entirely, + which is worse than the plain overwrite this replaces. + + local_obj's gramps_id is cleared before merging so merge() doesn't + misread it as a real second object being absorbed (which is what + merge() is for) and tag on a spurious "Merged Gramps ID" attribute -- + this is the same object, edited twice, not two objects becoming one. + """ + if type(current).merge is BaseObject.merge: + return local_obj + merged = deepcopy(current) + local_copy = deepcopy(local_obj) + local_copy.gramps_id = None + merged.merge(local_copy) + return merged + + +class WebApiDB(SQLite): + """ + DBAPI backend whose local SQLite connection is a mirror of a + Gramps Web API server, kept in sync via the server's transaction + history endpoint. + """ + + #: Set around _retry_after_conflict()'s own DbTxn so the + #: transaction_commit() it triggers can tell _push_payload() this push + #: is itself a conflict retry -- see _push_payload(). + _retrying = False + + def requires_login(self): + # Credentials come from GRAMPS_WEB_API_KEY, not a login dialog. + return False + + def _initialize(self, directory, username, password): + try: + self.web_client = WebApiHandler.from_env() + except _CONNECTION_ERRORS as err: + raise DbConnectionError(_describe_connection_error(err), directory) from err + + # Local mirror: reuse SQLite's own _initialize for the on-disk + # cache file, then sync from the server on load(). + super()._initialize(directory, username, password) + + def load(self, *args, **kwargs): + # callback is Gramps' own load-progress hook -- position 2 in + # DbGeneric.load()'s signature, or the "callback" kwarg -- the same + # plain percentage function cli/grampscli.py's _pulse_progress and + # gui/dbloader.py's real progress-bar wiring already provide. + # Forwarded to _sync_from_server() so a slow initial catch-up (a + # new mirror, or one that's been offline a while) shows real + # progress instead of Gramps just looking hung; _poll_tick()'s own + # background-poll call deliberately leaves this at its None + # default, since a 10-second background tick shouldn't pop a + # progress bar. + callback = kwargs.get("callback") + if callback is None and len(args) >= 2: + callback = args[1] + super().load(*args, **kwargs) + self._check_identity() + try: + self._sync_from_server(progress_callback=callback) + except _CONNECTION_ERRORS as err: + raise DbConnectionError( + _describe_connection_error(err), self._directory + ) from err + self._poll_source_id = GLib.timeout_add_seconds( + POLL_INTERVAL_SECONDS, self._poll_tick + ) + + def _check_identity(self): + """Require this Family Tree's own name to be "@" + for whoever GRAMPS_WEB_API_KEY currently authenticates as. + + Nothing else ties a local mirror to one particular server account: + there is no per-tree settings.ini (see the module docstring), and + _sync_from_server() only ever asks for changes *after* its stored + sync_last_time -- it has no way to notice the mirror belongs to a + different account entirely and would just quietly go on mixing old + and new data. Requiring (and reading back) the account identity in + the tree's own display name catches that at load time instead, and + costs nothing extra: get_dbname() just rereads the same name.txt + Gramps already writes for the Family Tree Manager. + + Both sides are compared after _FAMILY_TREE_NAME_UNSAFE_CHARS's + substitution, not the raw "@" string: the Family + Tree Manager's own rename callback silently applies that same + substitution to anything typed in (dbman.py's __change_name()), so + a hostname's dots can never actually reach name.txt intact -- an + exact-string comparison would reject every tree name Gramps itself + would let you type. + """ + try: + expected = self.web_client.get_identity() + except _CONNECTION_ERRORS as err: + raise DbConnectionError( + _describe_connection_error(err), self._directory + ) from err + expected_typeable = _FAMILY_TREE_NAME_UNSAFE_CHARS.sub("_", expected) + actual = self.get_dbname() + actual_normalized = _FAMILY_TREE_NAME_UNSAFE_CHARS.sub("_", actual) + if actual_normalized != expected_typeable: + raise DbConnectionError( + _( + 'This Family Tree is named "%(actual)s", but ' + "GRAMPS_WEB_API_KEY currently authenticates as " + '"%(expected)s". Rename this Family Tree to ' + '"%(expected_typeable)s" (Family Trees -> Manage ' + "Family Trees) if it's meant to mirror that account, " + "or open/create the Family Tree already named that -- " + "reusing this one would mix its existing local data " + "with the other account's." + ) + % { + "actual": actual, + "expected": expected, + "expected_typeable": expected_typeable, + }, + self._directory, + ) + + def close(self, *args, **kwargs): + # Stop polling a database that's no longer open -- otherwise the + # next tick would run _sync_from_server() (and touch self.dbapi) + # against a connection that's about to be (or already) closed. + poll_source_id = getattr(self, "_poll_source_id", None) + if poll_source_id is not None: + GLib.source_remove(poll_source_id) + self._poll_source_id = None + super().close(*args, **kwargs) + + def _poll_tick(self): + """GLib.timeout_add_seconds callback -- see the module docstring's + polling section. Must return True (GLib.SOURCE_CONTINUE) to keep + firing; returning a falsy value cancels the timeout, so a network + error is caught and logged here rather than left to propagate.""" + try: + self._sync_from_server() + except _CONNECTION_ERRORS: + LOG.exception("Periodic sync from server failed; will retry.") + return GLib.SOURCE_CONTINUE + + def transaction_commit(self, transaction): + # Must run before super(): it clears the transaction's records. + payload = transaction_to_json(transaction) + super().transaction_commit(transaction) + # self._retrying is set by _retry_after_conflict() while it holds + # its own DbTxn open, so the push this commit triggers knows it is + # itself a conflict retry and won't retry again on a second + # conflict -- see _push_payload(). + self._push_payload(payload, is_retry=self._retrying) + + def undo(self, update_history=True): + # Peek before super(): DbGenericUndo._undo() pops this DbTxn off + # undoq. The DbTxn's own backing data isn't touched by that (it + # just moves queues), so building its JSON payload could happen + # either side of super() -- only grabbing the reference itself + # can't wait. + transaction = self.undodb.undoq[-1] if self.undodb.undo_count else None + result = super().undo(update_history) + if result and transaction is not None: + self._push_payload(transaction_to_json(transaction), undo=True) + return result + + def redo(self, update_history=True): + transaction = self.undodb.redoq[-1] if self.undodb.redo_count else None + result = super().redo(update_history) + if result and transaction is not None: + # Redo is just re-applying the original transaction forward -- + # not a variant of undo=True. See push_transaction()'s docstring. + self._push_payload(transaction_to_json(transaction)) + return result + + def _push_payload(self, payload, undo=False, is_retry=False): + """Push a change-list payload to the server, handling a rejected + push (conflict or otherwise) the same way regardless of whether it + came from a plain commit, an undo, or a redo. + + is_retry marks a push that is itself the replay _retry_after_conflict() + made from an earlier conflict -- a second conflict on that replay is + logged and dropped rather than retried again, so a genuinely hot + object can't send this into an unbounded retry loop. + """ + if not payload: + return + try: + self.web_client.push_transaction(payload, undo=undo) + except WebApiPushConflict: + LOG.warning( + "Server rejected %d local change(s): the object(s) changed " + "server-side since the local mirror last synced. Resyncing " + "the mirror from the server now.", + len(payload), + ) + try: + self._sync_from_server() + except _CONNECTION_ERRORS: + LOG.exception("Resync after a push conflict also failed.") + return + if undo or is_retry: + LOG.warning( + "Giving up on %d local change(s) after a repeated or " + "undo/redo conflict; the local mirror was not resent to " + "the server.", + len(payload), + ) + return + self._retry_after_conflict(payload) + except _CONNECTION_ERRORS: + LOG.exception( + "Failed to push %d local change(s) to the server; " + "local mirror has drifted from the server until the " + "next successful push or read sync.", + len(payload), + ) + + def _retry_after_conflict(self, payload): + """Reapply each locally-intended change on top of the mirror + _push_payload() just resynced, as a fresh local edit -- see the + module docstring's write-through section. An add/update whose + object still exists after the resync is combined with the current + (server-fresh) object via _merge_or_overwrite() rather than + blindly replacing it. + + Runs as one ordinary (non-batch) DbTxn, so it goes through the + normal transaction_commit() -> _push_payload() path again -- this + time with an "old" snapshot that matches what the resync just + pulled down, so it will only be rejected again if something else + changed server-side in the brief window since that resync. + """ + self._retrying = True + try: + with DbTxn(_("Retry local change after server conflict"), self) as trans: + for entry in payload: + key = CLASS_TO_KEY_MAP.get(entry["_class"]) + if key is None: + continue + name = KEY_TO_NAME_MAP[key] + handle = entry["handle"] + has_handle = getattr(self, f"has_{name}_handle") + if entry["type"] == "delete": + if has_handle(handle): + getattr(self, f"remove_{name}")(handle, trans) + else: + obj = data_to_object(entry["new"]) + if has_handle(handle): + current = getattr(self, f"get_{name}_from_handle")(handle) + obj = _merge_or_overwrite(current, obj) + getattr(self, f"commit_{name}")(obj, trans) + finally: + self._retrying = False + + def _sync_from_server(self, progress_callback=None): + """ + Pull every transaction after the last-seen timestamp and replay + its changes into the local mirror. Returns the number of changes + applied. + + An empty "changes" list on a transaction is not a no-op: it is + what a batch=True commit leaves behind (see the module + docstring's note on trans.batch guards around trans.add()) -- + something happened server-side that this feed cannot describe. + Flagged rather than silently skipped; _full_resync() is the + fallback once the whole page range has been walked (so + sync_last_time still advances past it and any *describable* + changes around it are applied normally either way). + + progress_callback, if given, is called with an int 0-100 after + each page -- see load()'s callback param. "total" comes from the + server's X-Total-Count for this "after" filter (get_transaction_ + history()'s docstring), so it stays a stable denominator across + pages barring concurrent server-side writes during the sync. + """ + after = self._get_metadata("sync_last_time", default=0) + applied = 0 + needs_full_resync = False + page = 1 + seen = 0 + while True: + transactions, total = self.web_client.get_transaction_history( + after=after, page=page, pagesize=SYNC_PAGE_SIZE + ) + if not transactions: + break + # (obj_class, handle) -> trans_type, collapsed to the net + # effect within this page -- see _emit_change_signals(). + net_changes = {} + with DbTxn("Sync from server", self, batch=True) as trans: + for server_trans in transactions: + if not server_trans["changes"]: + needs_full_resync = True + for change in server_trans["changes"]: + if self._apply_change(change, trans): + applied += 1 + net_changes[(change["obj_class"], change["obj_handle"])] = ( + change["trans_type"] + ) + after = max(after, server_trans["timestamp"]) + self._emit_change_signals(net_changes) + seen += len(transactions) + if progress_callback is not None and total: + progress_callback(min(100, int(seen * 100 / total))) + if len(transactions) < SYNC_PAGE_SIZE: + break + page += 1 + self._set_metadata("sync_last_time", after) + if needs_full_resync: + self._full_resync(progress_callback=progress_callback) + return applied + + def _full_resync(self, progress_callback=None): + """ + Rebuild the local mirror from scratch: download the server's own + current Gramps XML export and reimport it, after clearing every + local primary object first. Called by _sync_from_server() when + the transaction-history feed contains an empty-changes marker -- + by definition there is nothing in that history to replay for + whatever produced it, so the only way to recover is to fetch the + server's current state wholesale, the same way populating a + brand new local mirror already works. + + Deliberately reuses the stock ImportXml importer against a raw + XML export rather than reconstructing objects from the REST + /people/, /families/, ... endpoints: those return a marshalled + display schema (plain ints for GrampsType fields, no "_class" + tag), not the json_utils shape data_to_object() needs. Only the + transaction-history feed's new_data and a raw XML export share + that shape, and the whole point of this method is that the + former can't be trusted here. + + The clear-then-import pair each run inside their own batch=True + DbTxn (ImportXml's own, internally, for the import half -- see + importxml.py), so neither triggers transaction_commit()'s + push-to-server path (transaction_to_json() sees nothing to + push for a batch transaction) -- this is a purely local rebuild, + same as _sync_from_server()'s own transactions. + + progress_callback, if given, only gets 0/100 markers bookending + the download+reimport -- unlike _sync_from_server()'s page-by-page + reporting, ImportXml has no internal step reporting to forward + finer-grained progress from. + """ + if progress_callback is not None: + progress_callback(0) + data = self.web_client.download_export() + with NamedTemporaryFile(suffix=".gramps", delete=False) as tmp_file: + tmp_file.write(data) + tmp_path = tmp_file.name + try: + with DbTxn( + _("Clear local mirror before full resync"), self, batch=True + ) as trans: + for key in set(CLASS_TO_KEY_MAP.values()): + name = KEY_TO_NAME_MAP[key] + handles = list(getattr(self, f"get_{name}_handles")()) + remove = getattr(self, f"remove_{name}") + for handle in handles: + remove(handle, trans) + importData(self, tmp_path, User()) + # importData() runs its own batch=True DbTxn internally, so + # (like _sync_from_server()'s replay) it emits nothing to + # already-open views on its own -- request_rebuild() is the + # "too much changed to describe incrementally" signal DbGeneric + # itself defines for exactly this case (one -rebuild per + # object type, telling every view to reload wholesale). + self.request_rebuild() + finally: + os.remove(tmp_path) + if progress_callback is not None: + progress_callback(100) + + def _apply_change(self, change, trans): + """Replay one server change into the local mirror. Returns True + if it was a recognized primary-object change (as opposed to a + reference-type change, which carries no obj_class we can map).""" + obj_class = change["obj_class"] + key = CLASS_TO_KEY_MAP.get(obj_class) + if key is None: + return False + name = KEY_TO_NAME_MAP[key] + handle = change["obj_handle"] + if change["trans_type"] == TXNDEL: + getattr(self, f"remove_{name}")(handle, trans) + else: + # add and update are both upserts at the DBAPI level, so + # there's no need to treat them differently here. + obj = data_to_object(change["new_data"]) + getattr(self, f"commit_{name}")(obj, trans) + return True + + def _emit_change_signals(self, net_changes): + """Emit the person-add/family-update/event-delete/... signals a + normal (non-batch) local commit would have emitted for these same + changes -- see the module docstring's note on why + _sync_from_server()'s batch=True replay needs this done by hand. + + net_changes: {(obj_class, obj_handle): trans_type}, already + collapsed to the net effect per handle (see _sync_from_server()). + Unrecognized obj_class values (reference-type changes never reach + here in the first place -- see _apply_change()) are skipped the + same way _apply_change() skips them. + + Grouped and emitted in the same order DBAPI.transaction_commit() + uses for a normal commit -- deletes and adds before updates -- so + a view that (for instance) cares about total counts sees them + change before it sees an in-place update to one of the survivors. + """ + by_type = {TXNDEL: {}, TXNADD: {}, TXNUPD: {}} + for (obj_class, handle), trans_type in net_changes.items(): + key = CLASS_TO_KEY_MAP.get(obj_class) + if key is None: + continue + name = KEY_TO_NAME_MAP[key] + by_type[trans_type].setdefault(name, []).append(handle) + for trans_type in (TXNDEL, TXNADD, TXNUPD): + for name, handles in by_type[trans_type].items(): + self.emit(name + _TRANS_TYPE_ACTION[trans_type], (handles,)) diff --git a/GrampsWebApiDb/mintapikeytool.gpr.py b/GrampsWebApiDb/mintapikeytool.gpr.py new file mode 100644 index 000000000..1e4e21d4d --- /dev/null +++ b/GrampsWebApiDb/mintapikeytool.gpr.py @@ -0,0 +1,41 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +register( + TOOL, + id="MintApiKeyTool", + name=_("Generate Gramps Web API key"), + description=_( + "Turn a Gramps Web API server URL, username, and password into a " + "GRAMPS_WEB_API_KEY value for GrampsWebApiDb, and optionally " + "create a matching, correctly-named GrampsWebApiDb Family Tree " + "for it." + ), + status=BETA, + version = '0.1.1', + gramps_target_version="6.0", + fname="mintapikeytool.py", + authors=["Doug Blank"], + authors_email=["doug.blank@gmail.com"], + category=TOOL_UTILS, + toolclass="MintApiKeyTool", + optionclass="MintApiKeyToolOptions", + tool_modes=[TOOL_MODE_GUI], + help_url="Addon:GrampsWebApiDb", +) diff --git a/GrampsWebApiDb/mintapikeytool.py b/GrampsWebApiDb/mintapikeytool.py new file mode 100644 index 000000000..f26d71cad --- /dev/null +++ b/GrampsWebApiDb/mintapikeytool.py @@ -0,0 +1,380 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +"""Tools/Utilities/Generate Gramps Web API key + +Dialog front end for WebApiHandler.mint_api_key() (webapi_client.py): +username/password in, a GRAMPS_WEB_API_KEY value out. Exists because +generating a key otherwise requires either a shell with the standalone +gramps-api-client package installed, or hand-writing the three lines of +Python from GrampsWebApiDb's own README -- this tool is that same call +behind a form, for anyone who just wants the key string. + +A TOOL rather than a gramplet: generating a key is a one-off setup action, +not something worth keeping permanently docked in a gramplet bar. It +doesn't read or write anything in dbstate's Family Tree -- see +build_dialog() below -- but Gramps only populates the Tools menu once +some Family Tree is open, so in practice the user still needs one open +(any backend, even an empty local tree) to reach this tool at all. + +On success, it also sets GRAMPS_WEB_API_KEY in this running Gramps +process's environment, so a WebApiDB-backed Family Tree can be opened in +the same session without restarting Gramps -- but that lasts only for +this process; it is not written to a shell profile, settings.ini, or any +open Family Tree, matching the README's "Credentials" section on why +persistence otherwise stays a manual step. + +"Create Synced Family Tree for this key" goes one step further: it +creates (but does not open) a new, empty Family Tree using the +"grampswebapidb" DATABASE plugin, named "@" for whoever +the key authenticates as -- the exact name grampswebapidb.py's +_check_identity() requires, via the same CLIDbManager.create_new_db_cli() +Gramps' own Family Tree Manager uses for its "New" button, just with an +explicit dbid instead of the configured default backend. See README.md's +"Family Tree naming" section for why that name is required. +""" + +# ------------------------------------------------------------------------ +# +# Standard Python modules +# +# ------------------------------------------------------------------------ +import os +import re +import threading +from urllib.error import HTTPError, URLError + +# ------------------------------------------------------------------------ +# +# GTK/Gnome modules +# +# ------------------------------------------------------------------------ +from gi.repository import GLib, Gtk + +# ------------------------------------------------------------------------ +# +# Gramps modules +# +# ------------------------------------------------------------------------ +from gramps.cli.clidbman import CLIDbManager +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gui.managedwindow import ManagedWindow +from gramps.gui.plug import tool +from gramps.gui.utils import text_to_clipboard + +from webapi_client import API_KEY_ENV_VAR, WebApiHandler + +try: + _trans = glocale.get_addon_translator(__file__) +except ValueError: + _trans = glocale.translation +_ = _trans.gettext + +#: Same failure modes WebApiHandler.from_env()/push_transaction() can +#: raise elsewhere in this addon -- see grampswebapidb.py's +#: _CONNECTION_ERRORS -- but here they mean "bad URL/credentials or an +#: unreachable server" rather than "lost sync", so they're just reported +#: in the status label rather than acted on. +_MINT_ERRORS = (ValueError, HTTPError, URLError, OSError) + +#: Same substitution grampswebapidb.py's _check_identity() applies to a +#: Family Tree's own name before comparing it against the server identity +#: -- keep in sync if that changes. Applied here too so a tree created by +#: this button already has the name _check_identity() will accept. +_FAMILY_TREE_NAME_UNSAFE_CHARS = re.compile(r"[':<>|,;=\"\[\]\.\+\*\/\?\\]") + +#: The DATABASE plugin id grampswebapidb.gpr.py registers WebApiDB under. +_WEBAPIDB_ID = "grampswebapidb" + + +class MintApiKeyTool(tool.Tool, ManagedWindow): + """ + Dialog that turns a server URL + username + password into a + GRAMPS_WEB_API_KEY value, via WebApiHandler.mint_api_key(). + """ + + def __init__(self, dbstate, user, options_class, name, callback=None): + self.dbstate = dbstate + self.uistate = user.uistate + self.mint_thread = None + self.tree_thread = None + ManagedWindow.__init__(self, self.uistate, [], self.__class__) + self.set_window(Gtk.Window(), Gtk.Label(), "") + tool.Tool.__init__(self, dbstate, options_class, name) + + dialog = self.build_dialog() + dialog.run() + dialog.destroy() + self.close() + + def build_dialog(self): + dialog = Gtk.Dialog( + _("Generate Gramps Web API key"), + self.uistate.window if self.uistate else None, + Gtk.DialogFlags.MODAL | Gtk.DialogFlags.DESTROY_WITH_PARENT, + (Gtk.STOCK_CLOSE, Gtk.ResponseType.CLOSE), + ) + dialog.set_default_size(420, -1) + + vbox = dialog.get_content_area() + vbox.set_border_width(6) + vbox.set_spacing(6) + + self.url_entry = self.__add_entry( + vbox, _("Server URL"), _("e.g. https://your-server/api") + ) + self.username_entry = self.__add_entry(vbox, _("Username")) + self.password_entry = self.__add_entry(vbox, _("Password")) + self.password_entry.set_visibility(False) + self.password_entry.connect("activate", self.mint_clicked) + + button_box = Gtk.ButtonBox() + button_box.set_layout(Gtk.ButtonBoxStyle.START) + button_box.set_spacing(6) + button_box.set_border_width(6) + + self.mint_button = Gtk.Button(label=_("Generate API Key")) + self.mint_button.connect("clicked", self.mint_clicked) + button_box.add(self.mint_button) + vbox.pack_start(button_box, False, False, 0) + + self.status_label = Gtk.Label(halign=Gtk.Align.START) + self.status_label.set_line_wrap(True) + self.status_label.set_text( + _( + "Enter your server URL, username, and password, then " + "click Generate API Key." + ) + ) + vbox.pack_start(self.status_label, False, False, 0) + + self.key_entry = self.__add_entry(vbox, _("GRAMPS_WEB_API_KEY")) + self.key_entry.set_editable(False) + + copy_box = Gtk.ButtonBox() + copy_box.set_layout(Gtk.ButtonBoxStyle.START) + copy_box.set_spacing(6) + copy_box.set_border_width(6) + + copy_button = Gtk.Button(label=_("Copy")) + copy_button.connect("clicked", self.copy_clicked) + copy_box.add(copy_button) + vbox.pack_start(copy_box, False, False, 0) + + self.create_tree_button = Gtk.Button( + label=_("Create Synced Family Tree for this key") + ) + self.create_tree_button.set_sensitive(False) + self.create_tree_button.connect("clicked", self.create_tree_clicked) + vbox.pack_start(self.create_tree_button, False, False, 0) + + self.tree_name_entry = self.__add_entry(vbox, _("Family Tree")) + self.tree_name_entry.set_editable(False) + + existing_key = os.environ.get(API_KEY_ENV_VAR) + if existing_key: + self.key_entry.set_text(existing_key) + self.create_tree_button.set_sensitive(True) + self.status_label.set_text( + _( + "Found an existing %s for this session. Click Create " + "Synced Family Tree to use it directly, or fill in the " + "form above to generate a different key." + ) + % API_KEY_ENV_VAR + ) + + dialog.show_all() + return dialog + + def __add_entry(self, vbox, name, tooltip=""): + label = Gtk.Label(halign=Gtk.Align.START) + label.set_markup("%s" % name) + vbox.pack_start(label, False, False, 0) + entry = Gtk.Entry() + entry.set_tooltip_text(tooltip) + vbox.pack_start(entry, False, False, 0) + return entry + + def mint_clicked(self, obj): + url = self.url_entry.get_text().strip() + username = self.username_entry.get_text().strip() + password = self.password_entry.get_text() + if not url or not username or not password: + self.status_label.set_text( + _("Please fill in the server URL, username, and password.") + ) + return + + self.key_entry.set_text("") + self.tree_name_entry.set_text("") + self.password_entry.set_text("") + self.status_label.set_text(_("Generating API key…")) + self.mint_button.set_sensitive(False) + self.create_tree_button.set_sensitive(False) + + if self.mint_thread and self.mint_thread.is_alive(): + return + self.mint_thread = threading.Thread( + target=self._mint_api_key, args=(url, username, password) + ) + self.mint_thread.daemon = True + self.mint_thread.start() + + def _mint_api_key(self, url, username, password): + """Run off the GTK main thread; hands the result back via idle_add.""" + try: + key = WebApiHandler.mint_api_key(url, username, password) + except _MINT_ERRORS as exc: + GLib.idle_add(self._mint_failed, self._describe_mint_error(exc)) + else: + GLib.idle_add(self._mint_succeeded, key) + + @staticmethod + def _describe_mint_error(exc): + """ + Turn a mint_api_key() exception into a message that says which of + URL/username/password is the likely problem, instead of a raw + urllib exception the user has to decode themselves. + """ + if isinstance(exc, HTTPError): + if exc.code in (401, 403): + return _( + "Login failed (HTTP %d): check your username and password." + ) % exc.code + return _( + "Server returned an error (HTTP %d %s): check the Server URL." + ) % (exc.code, exc.reason) + if isinstance(exc, OSError): + # Covers URLError (DNS failure, connection refused, ...) and + # socket.timeout, both OSError subclasses -- the server at + # that URL could not be reached at all. + reason = getattr(exc, "reason", exc) + return ( + _("Could not reach the server: check the Server URL. (%s)") + % reason + ) + return _("Unexpected response from the server: %s") % exc + + def _mint_failed(self, message): + self.status_label.set_text(_("Error: %s") % message) + self.mint_button.set_sensitive(True) + return False + + def _mint_succeeded(self, key): + os.environ[API_KEY_ENV_VAR] = key + self.key_entry.set_text(key) + self.status_label.set_text( + _( + "Success. %s is now set for this Gramps session -- no " + "restart needed. Copy the key below to also set it in your " + "shell environment for next time." + ) + % API_KEY_ENV_VAR + ) + self.mint_button.set_sensitive(True) + self.create_tree_button.set_sensitive(True) + self.key_entry.grab_focus() + self.key_entry.select_region(0, -1) + return False + + def copy_clicked(self, obj): + text_to_clipboard(self.key_entry.get_text()) + + def create_tree_clicked(self, obj): + key = self.key_entry.get_text().strip() + if not key: + return + + self.tree_name_entry.set_text("") + self.status_label.set_text(_("Looking up account identity…")) + self.create_tree_button.set_sensitive(False) + + if self.tree_thread and self.tree_thread.is_alive(): + return + self.tree_thread = threading.Thread(target=self._lookup_identity, args=(key,)) + self.tree_thread.daemon = True + self.tree_thread.start() + + def _lookup_identity(self, key): + """Run off the GTK main thread; hands the result back via idle_add.""" + try: + identity = WebApiHandler.from_api_key(key).get_identity() + except _MINT_ERRORS as exc: + GLib.idle_add(self._create_tree_failed, self._describe_mint_error(exc)) + else: + GLib.idle_add(self._create_tree, identity) + + def _create_tree(self, identity): + """ + Create the local Family Tree entry (mkdir + name.txt + + database.txt) for `identity`, via the same CLIDbManager Gramps' + own Family Tree Manager uses -- see clidbman.py's + create_new_db_cli(). Runs on the GTK main thread (via idle_add): + it's local filesystem work, not network, and touches self.dbstate. + """ + tree_name = _FAMILY_TREE_NAME_UNSAFE_CHARS.sub("_", identity) + dbman = CLIDbManager(self.dbstate) + if tree_name in [existing[0] for existing in dbman.current_names]: + self.tree_name_entry.set_text(tree_name) + self.status_label.set_text( + _( + 'A Family Tree named "%s" already exists. Open it from ' + "Family Trees -> Manage Family Trees instead of " + "creating another one." + ) + % tree_name + ) + self.create_tree_button.set_sensitive(True) + return False + try: + new_path, title = dbman.create_new_db_cli( + title=tree_name, dbid=_WEBAPIDB_ID + ) + except Exception as exc: + self.status_label.set_text(_("Error creating Family Tree: %s") % exc) + self.create_tree_button.set_sensitive(True) + return False + self.tree_name_entry.set_text(title) + self.status_label.set_text( + _( + 'Created Family Tree "%s". Open it from Family Trees -> ' + "Manage Family Trees to start syncing." + ) + % title + ) + self.create_tree_button.set_sensitive(True) + self.tree_name_entry.grab_focus() + self.tree_name_entry.select_region(0, -1) + return False + + def _create_tree_failed(self, message): + self.status_label.set_text(_("Error: %s") % message) + self.create_tree_button.set_sensitive(True) + return False + + +class MintApiKeyToolOptions(tool.ToolOptions): + """ + Defines options and provides handling interface. + """ + + def __init__(self, name, person_id=None): + tool.ToolOptions.__init__(self, name, person_id) diff --git a/GrampsWebApiDb/po/template.pot b/GrampsWebApiDb/po/template.pot new file mode 100644 index 000000000..0c55ace15 --- /dev/null +++ b/GrampsWebApiDb/po/template.pot @@ -0,0 +1,181 @@ +# SOME DESCRIPTIVE TITLE. +# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER +# This file is distributed under the same license as the PACKAGE package. +# FIRST AUTHOR , YEAR. +# +#, fuzzy +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: \n" +"POT-Creation-Date: 2026-08-09 10:03-0700\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" +"Last-Translator: FULL NAME \n" +"Language-Team: LANGUAGE \n" +"Language: \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" + +#: GrampsWebApiDb/mintapikeytool.py:129 GrampsWebApiDb/mintapikeytool.gpr.py:23 +msgid "Generate Gramps Web API key" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:141 +msgid "Server URL" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:141 +msgid "e.g. https://your-server/api" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:143 +msgid "Username" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:144 +msgid "Password" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:153 +msgid "Generate API Key" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:162 +msgid "" +"Enter your server URL, username, and password, then click Generate API Key." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:168 +msgid "GRAMPS_WEB_API_KEY" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:176 +msgid "Copy" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:182 +msgid "Create Synced Family Tree for this key" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:188 +msgid "Family Tree" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:197 +#, python-format +msgid "" +"Found an existing %s for this session. Click Create Synced Family Tree to " +"use it directly, or fill in the form above to generate a different key." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:222 +msgid "Please fill in the server URL, username, and password." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:229 +msgid "Generating API key…" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:260 +#, python-format +msgid "Login failed (HTTP %d): check your username and password." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:263 +#, python-format +msgid "Server returned an error (HTTP %d %s): check the Server URL." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:271 +#, python-format +msgid "Could not reach the server: check the Server URL. (%s)" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:274 +#, python-format +msgid "Unexpected response from the server: %s" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:277 GrampsWebApiDb/mintapikeytool.py:369 +#, python-format +msgid "Error: %s" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:286 +#, python-format +msgid "" +"Success. %s is now set for this Gramps session -- no restart needed. Copy " +"the key below to also set it in your shell environment for next time." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:307 +msgid "Looking up account identity…" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:339 +#, python-format +msgid "" +"A Family Tree named \"%s\" already exists. Open it from Family Trees -> " +"Manage Family Trees instead of creating another one." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:352 +#, python-format +msgid "Error creating Family Tree: %s" +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.py:358 +#, python-format +msgid "" +"Created Family Tree \"%s\". Open it from Family Trees -> Manage Family Trees " +"to start syncing." +msgstr "" + +#: GrampsWebApiDb/mintapikeytool.gpr.py:25 +msgid "" +"Turn a Gramps Web API server URL, username, and password into a " +"GRAMPS_WEB_API_KEY value for GrampsWebApiDb, and optionally create a " +"matching, correctly-named GrampsWebApiDb Family Tree for it." +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.py:262 +msgid "" +"The account authenticating via GRAMPS_WEB_API_KEY does not have permission " +"on the server for this operation (HTTP 403 Forbidden). Generate a key for an " +"account with sufficient permissions, or ask the server administrator to " +"grant this one access." +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.py:422 +#, python-format +msgid "" +"This Family Tree is named \"%(actual)s\", but GRAMPS_WEB_API_KEY currently " +"authenticates as \"%(expected)s\". Rename this Family Tree to " +"\"%(expected_typeable)s\" (Family Trees -> Manage Family Trees) if it's " +"meant to mirror that account, or open/create the Family Tree already named " +"that -- reusing this one would mix its existing local data with the other " +"account's." +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.py:550 +msgid "Retry local change after server conflict" +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.py:668 +msgid "Clear local mirror before full resync" +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.gpr.py:24 +msgid "GrampsWebApiDb" +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.gpr.py:25 +msgid "Gramps _Web API Database" +msgstr "" + +#: GrampsWebApiDb/grampswebapidb.gpr.py:27 +msgid "" +"Use a Gramps Web API server (e.g. gramps-connect or Gramps Web) as a live " +"database, mirrored locally for speed." +msgstr "" diff --git a/GrampsWebApiDb/tests/__init__.py b/GrampsWebApiDb/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/GrampsWebApiDb/tests/test_grampswebapidb.py b/GrampsWebApiDb/tests/test_grampswebapidb.py new file mode 100644 index 000000000..1aaff0a5f --- /dev/null +++ b/GrampsWebApiDb/tests/test_grampswebapidb.py @@ -0,0 +1,1289 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Unit tests for grampswebapidb.WebApiDB: the sync/write-through logic. + +WebApiDB subclasses the stock SQLite DBAPI backend, but these tests never +open a real database file -- SQLite's own commit_*/remove_* methods and +transaction machinery are stubbed out (WebApiDB.__new__() plus per-test +attribute overrides), the same pattern SharedPostgreSQL/tests/ +test_initialize.py uses to test _create_settings() without a real Postgres +connection. This isolates exactly the logic this addon adds: + + - transaction_to_json(): local DbTxn -> flat change-list payload + - _apply_change(): one server change -> a commit_*/remove_* call + - _sync_from_server(): pagination + sync_last_time bookkeeping + - transaction_commit(): push-after-commit, ordering, and error swallowing + +Run with:: + + python3 -m unittest GrampsWebApiDb.tests.test_grampswebapidb -v +""" + +# ------------------------------------------------------------------------- +# +# Standard python modules +# +# ------------------------------------------------------------------------- +import os +import sys +import unittest +from urllib.error import HTTPError +from unittest import mock + +# ------------------------------------------------------------------------- +# +# Make the addon importable the way Gramps loads it: its own directory on +# sys.path (grampswebapidb.py does a bare ``from webapi_client import +# WebApiHandler`` -- see CLAUDE.md Testing conventions). +# +# ------------------------------------------------------------------------- +ADDON_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + +try: + import gramps +except ImportError as _err: + raise unittest.SkipTest("gramps package not available: %s" % _err) + +from gramps.gen.db.dbconst import REFERENCE_KEY, TXNADD, TXNDEL, TXNUPD +from gramps.gen.db.exceptions import DbConnectionError +from gramps.gen.lib import Person, Tag +from gramps.gen.lib.json_utils import object_to_data, remove_object + +from GrampsWebApiDb import grampswebapidb +from GrampsWebApiDb.grampswebapidb import ( + WebApiDB, + WebApiPushConflict, + transaction_to_json, +) + +# grampswebapidb.py imports webapi_client with a bare `from webapi_client +# import ...` (see CLAUDE.md Testing conventions -- this addon has no +# __init__.py, so Gramps and tests alike add its own directory to +# sys.path). That makes "webapi_client" and "GrampsWebApiDb.webapi_client" +# two distinct sys.modules entries for the same file, so an exception +# class must come from whichever import path the code under test actually +# uses -- grampswebapidb.WebApiPushConflict here, not a fresh +# `from GrampsWebApiDb.webapi_client import WebApiPushConflict`, or +# `except WebApiPushConflict` in transaction_commit() won't match it. + + +# ------------------------------------------------------------------------- +# +# Test helpers +# +# ------------------------------------------------------------------------- +def person_data(handle="H1", gramps_id="I0001"): + """A real Person's data-dict, the shape new_data/old_data actually take.""" + person = Person() + person.set_handle(handle) + person.set_gramps_id(gramps_id) + return object_to_data(person) + + +class FakeTransaction: + """Duck-types the bit of DbTxn that transaction_to_json() reads: + get_recnos()/get_record(). Avoids needing a real commitdb/pickle round + trip just to test the flattening logic.""" + + def __init__(self, records): + # records: list of (key, action, handle, old_data, new_data) + self._records = records + + def get_recnos(self, reverse=False): + idx = range(len(self._records)) + return reversed(idx) if reverse else idx + + def get_record(self, recno): + return self._records[recno] + + +def new_instance(): + """A WebApiDB that never touched a real SQLite file or server.""" + return WebApiDB.__new__(WebApiDB) + + +# ------------------------------------------------------------------------- +# +# TestTransactionToJson +# +# ------------------------------------------------------------------------- +class TestTransactionToJson(unittest.TestCase): + def test_add_record_shape(self): + new_data = person_data() + trans = FakeTransaction([(0, TXNADD, "H1", None, new_data)]) # PERSON_KEY + out = transaction_to_json(trans) + self.assertEqual(len(out), 1) + entry = out[0] + self.assertEqual(entry["type"], "add") + self.assertEqual(entry["handle"], "H1") + self.assertEqual(entry["_class"], "Person") + self.assertIsNone(entry["old"]) + self.assertNotIn("_object", entry["new"]) + + def test_update_record_carries_old_and_new(self): + old_data = person_data(gramps_id="I0001") + new_data = person_data(gramps_id="I0002") + trans = FakeTransaction([(0, TXNUPD, "H1", old_data, new_data)]) + out = transaction_to_json(trans) + self.assertEqual(out[0]["type"], "update") + self.assertNotIn("_object", out[0]["old"]) + self.assertNotIn("_object", out[0]["new"]) + + def test_delete_record_has_no_new_data(self): + old_data = person_data() + trans = FakeTransaction([(0, TXNDEL, "H1", old_data, None)]) + out = transaction_to_json(trans) + self.assertEqual(out[0]["type"], "delete") + self.assertIsNone(out[0]["new"]) + self.assertIsNotNone(out[0]["old"]) + + def test_reference_type_record_is_skipped(self): + # REFERENCE_KEY has no entry in KEY_TO_CLASS_MAP -- see dbconst.py. + trans = FakeTransaction([(REFERENCE_KEY, TXNADD, "H1", None, {})]) + self.assertEqual(transaction_to_json(trans), []) + + def test_multiple_records_preserve_order(self): + trans = FakeTransaction( + [ + (0, TXNADD, "H1", None, person_data("H1")), + (0, TXNUPD, "H2", person_data("H2"), person_data("H2", "I0002")), + ] + ) + out = transaction_to_json(trans) + self.assertEqual([e["handle"] for e in out], ["H1", "H2"]) + + +# ------------------------------------------------------------------------- +# +# TestApplyChange +# +# ------------------------------------------------------------------------- +class TestApplyChange(unittest.TestCase): + def setUp(self): + self.db = new_instance() + self.db.commit_person = mock.MagicMock() + self.db.remove_person = mock.MagicMock() + self.trans = object() # opaque; just forwarded + + def test_unrecognized_obj_class_is_ignored(self): + change = {"obj_class": "NotAThing", "trans_type": TXNADD, "obj_handle": "H1"} + applied = self.db._apply_change(change, self.trans) + self.assertFalse(applied) + self.db.commit_person.assert_not_called() + self.db.remove_person.assert_not_called() + + def test_delete_calls_remove(self): + change = {"obj_class": "Person", "trans_type": TXNDEL, "obj_handle": "H1"} + applied = self.db._apply_change(change, self.trans) + self.assertTrue(applied) + self.db.remove_person.assert_called_once_with("H1", self.trans) + self.db.commit_person.assert_not_called() + + def test_add_calls_commit_with_reconstructed_object(self): + new_data = remove_object(person_data("H1", "I0001")) + change = { + "obj_class": "Person", + "trans_type": TXNADD, + "obj_handle": "H1", + "new_data": new_data, + } + applied = self.db._apply_change(change, self.trans) + self.assertTrue(applied) + self.db.commit_person.assert_called_once() + obj, trans = self.db.commit_person.call_args[0] + self.assertIsInstance(obj, Person) + self.assertEqual(obj.get_handle(), "H1") + self.assertIs(trans, self.trans) + + def test_update_is_also_an_upsert(self): + new_data = remove_object(person_data("H1", "I0002")) + change = { + "obj_class": "Person", + "trans_type": TXNUPD, + "obj_handle": "H1", + "new_data": new_data, + } + applied = self.db._apply_change(change, self.trans) + self.assertTrue(applied) + self.db.commit_person.assert_called_once() + self.db.remove_person.assert_not_called() + + +# ------------------------------------------------------------------------- +# +# TestEmitChangeSignals +# +# ------------------------------------------------------------------------- +class TestEmitChangeSignals(unittest.TestCase): + """_emit_change_signals() reproduces the person-add/family-update/... + signals a normal (non-batch) local commit would have emitted -- see + _sync_from_server()'s batch=True DbTxn and the module docstring's note + on why that otherwise leaves already-open views unaware anything + changed.""" + + def setUp(self): + self.db = new_instance() + self.db.emit = mock.MagicMock() + + def emitted(self): + return {call.args[0]: call.args[1][0] for call in self.db.emit.call_args_list} + + def test_add_emits_person_add_with_handle(self): + self.db._emit_change_signals({("Person", "H1"): TXNADD}) + self.assertEqual(self.emitted(), {"person-add": ["H1"]}) + + def test_update_emits_dash_update(self): + self.db._emit_change_signals({("Family", "F1"): TXNUPD}) + self.assertEqual(self.emitted(), {"family-update": ["F1"]}) + + def test_delete_emits_dash_delete(self): + self.db._emit_change_signals({("Event", "E1"): TXNDEL}) + self.assertEqual(self.emitted(), {"event-delete": ["E1"]}) + + def test_unrecognized_obj_class_is_skipped(self): + self.db._emit_change_signals({("NotAThing", "H1"): TXNADD}) + self.db.emit.assert_not_called() + + def test_same_class_and_trans_type_batched_into_one_call(self): + self.db._emit_change_signals( + {("Person", "H1"): TXNUPD, ("Person", "H2"): TXNUPD} + ) + self.db.emit.assert_called_once() + name, (handles,) = self.db.emit.call_args[0] + self.assertEqual(name, "person-update") + self.assertEqual(set(handles), {"H1", "H2"}) + + def test_deletes_and_adds_emitted_before_updates(self): + # Same ordering as DBAPI.transaction_commit()'s own signal loop. + self.db._emit_change_signals( + {("Person", "H1"): TXNUPD, ("Family", "F1"): TXNDEL} + ) + names = [call.args[0] for call in self.db.emit.call_args_list] + self.assertEqual(names, ["family-delete", "person-update"]) + + +# ------------------------------------------------------------------------- +# +# TestSyncFromServer +# +# ------------------------------------------------------------------------- +class FakeDbTxn: + """Stand-in for gramps.gen.db.DbTxn: a plain context manager, so + _sync_from_server's pagination/bookkeeping can be tested without a + real transaction_begin/transaction_commit or get_undodb().""" + + def __init__(self, msg, grampsdb, batch=False): + pass + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + return False + + +class TestSyncFromServer(unittest.TestCase): + def setUp(self): + self.db = new_instance() + self.db.web_client = mock.MagicMock() + # _sync_from_server() now emits change signals per page (see + # TestEmitChangeSignals for that logic in isolation) -- emit() + # itself needs Callback.__init__'s instance state, which + # new_instance()'s bare __new__() never runs, so it's stubbed here + # the same way commit_person/remove_person are stubbed elsewhere. + self.db.emit = mock.MagicMock() + self.metadata = {} + self.db._get_metadata = lambda key, default=0: self.metadata.get(key, default) + self.db._set_metadata = ( + lambda key, value, use_txn=True: self.metadata.__setitem__(key, value) + ) + self.patcher = mock.patch.object(grampswebapidb, "DbTxn", FakeDbTxn) + self.patcher.start() + self.addCleanup(self.patcher.stop) + # Every existing test here predates the empty-changes-marker -> + # full-resync fallback (see TestFullResyncTrigger below) and uses + # "changes": [] purely as pagination/timestamp filler, not to + # exercise that fallback -- stub it out so those tests keep + # testing what they always tested. + self.db._full_resync = mock.MagicMock() + + def test_stops_after_short_page(self): + change = {"obj_class": "Person", "trans_type": TXNADD, "obj_handle": "H1"} + self.db.web_client.get_transaction_history.return_value = ( + [{"timestamp": 5.0, "changes": [change]}], + 1, + ) + with mock.patch.object(self.db, "_apply_change", return_value=True) as apply: + applied = self.db._sync_from_server() + self.assertEqual(applied, 1) + apply.assert_called_once_with(change, mock.ANY) + self.db.web_client.get_transaction_history.assert_called_once() + + def test_pagination_continues_on_full_page(self): + full_page = [ + {"timestamp": float(i), "changes": []} + for i in range(grampswebapidb.SYNC_PAGE_SIZE) + ] + short_page = [{"timestamp": 999.0, "changes": []}] + self.db.web_client.get_transaction_history.side_effect = [ + (full_page, len(full_page) + 1), + (short_page, 1), + ] + applied = self.db._sync_from_server() + self.assertEqual(applied, 0) + self.assertEqual(self.db.web_client.get_transaction_history.call_count, 2) + calls = self.db.web_client.get_transaction_history.call_args_list + self.assertEqual(calls[0].kwargs["page"], 1) + self.assertEqual(calls[1].kwargs["page"], 2) + + def test_no_transactions_leaves_sync_time_unchanged(self): + self.metadata["sync_last_time"] = 42.0 + self.db.web_client.get_transaction_history.return_value = ([], 0) + applied = self.db._sync_from_server() + self.assertEqual(applied, 0) + self.assertEqual(self.metadata["sync_last_time"], 42.0) + + def test_sync_last_time_advances_to_max_timestamp_seen(self): + page = [ + {"timestamp": 10.0, "changes": []}, + {"timestamp": 30.0, "changes": []}, + {"timestamp": 20.0, "changes": []}, + ] + self.db.web_client.get_transaction_history.return_value = (page, 3) + self.db._sync_from_server() + self.assertEqual(self.metadata["sync_last_time"], 30.0) + + def test_unrecognized_changes_are_not_counted(self): + page = [ + { + "timestamp": 1.0, + "changes": [ + {"obj_class": "Bogus", "trans_type": TXNADD, "obj_handle": "H1"} + ], + } + ] + self.db.web_client.get_transaction_history.return_value = (page, 1) + applied = self.db._sync_from_server() + self.assertEqual(applied, 0) + + def test_emits_a_signal_per_applied_change(self): + change = {"obj_class": "Person", "trans_type": TXNADD, "obj_handle": "H1"} + self.db.web_client.get_transaction_history.return_value = ( + [{"timestamp": 5.0, "changes": [change]}], + 1, + ) + with mock.patch.object(self.db, "_apply_change", return_value=True): + self.db._sync_from_server() + self.db.emit.assert_called_once_with("person-add", (["H1"],)) + + def test_repeated_changes_to_one_handle_collapse_to_the_last(self): + # Same handle, updated then deleted within one page/poll -- only + # the net (delete) signal should fire, not both. + page = [ + { + "timestamp": 1.0, + "changes": [ + {"obj_class": "Person", "trans_type": TXNUPD, "obj_handle": "H1"} + ], + }, + { + "timestamp": 2.0, + "changes": [ + {"obj_class": "Person", "trans_type": TXNDEL, "obj_handle": "H1"} + ], + }, + ] + self.db.web_client.get_transaction_history.return_value = (page, 2) + with mock.patch.object(self.db, "_apply_change", return_value=True): + self.db._sync_from_server() + self.db.emit.assert_called_once_with("person-delete", (["H1"],)) + + def test_unrecognized_changes_emit_no_signal(self): + page = [ + { + "timestamp": 1.0, + "changes": [ + {"obj_class": "Bogus", "trans_type": TXNADD, "obj_handle": "H1"} + ], + } + ] + self.db.web_client.get_transaction_history.return_value = (page, 1) + self.db._sync_from_server() + self.db.emit.assert_not_called() + + def test_no_progress_callback_by_default(self): + # _poll_tick()'s background call relies on this: no callback means + # no attempt to report progress, so a periodic poll can't raise + # trying to call None. + page = [{"timestamp": 1.0, "changes": []}] + self.db.web_client.get_transaction_history.return_value = (page, 1) + self.db._sync_from_server() # must not raise + + def test_progress_reported_as_percent_of_total(self): + page = [{"timestamp": 1.0, "changes": []}] * 25 + self.db.web_client.get_transaction_history.return_value = (page, 100) + progress = mock.MagicMock() + self.db._sync_from_server(progress_callback=progress) + progress.assert_called_once_with(25) + + def test_progress_accumulates_and_caps_at_100_across_pages(self): + full_page = [ + {"timestamp": float(i), "changes": []} + for i in range(grampswebapidb.SYNC_PAGE_SIZE) + ] + short_page = [{"timestamp": 999.0, "changes": []}] + total = grampswebapidb.SYNC_PAGE_SIZE # short page pushes seen > total + self.db.web_client.get_transaction_history.side_effect = [ + (full_page, total), + (short_page, total), + ] + progress = mock.MagicMock() + self.db._sync_from_server(progress_callback=progress) + self.assertEqual( + [call.args[0] for call in progress.call_args_list], [100, 100] + ) + + def test_no_progress_call_when_total_is_zero(self): + # An empty-history sync (a brand new server-side tree, or nothing + # new since last sync) has no meaningful denominator to report + # against -- guards a ZeroDivisionError, not just noise. + page = [{"timestamp": 1.0, "changes": []}] + self.db.web_client.get_transaction_history.return_value = (page, 0) + progress = mock.MagicMock() + self.db._sync_from_server(progress_callback=progress) + progress.assert_not_called() + + def test_progress_callback_passed_through_to_full_resync(self): + page = [{"timestamp": 1.0, "changes": []}] + self.db.web_client.get_transaction_history.return_value = (page, 1) + progress = mock.MagicMock() + self.db._sync_from_server(progress_callback=progress) + self.db._full_resync.assert_called_once_with(progress_callback=progress) + + +# ------------------------------------------------------------------------- +# +# TestFullResyncTrigger +# +# A batch=True commit (any bulk import/merge/tool run through +# gramps-web-api) leaves an empty "changes" list on its transaction +# row -- see the module docstring's note on trans.batch guards around +# trans.add(). _sync_from_server() can't replay what was never logged, +# so it falls back to _full_resync() whenever it sees one. +# +# ------------------------------------------------------------------------- +class TestFullResyncTrigger(unittest.TestCase): + def setUp(self): + self.db = new_instance() + self.db.web_client = mock.MagicMock() + self.db.emit = mock.MagicMock() # see TestSyncFromServer.setUp's note + self.metadata = {} + self.db._get_metadata = lambda key, default=0: self.metadata.get(key, default) + self.db._set_metadata = ( + lambda key, value, use_txn=True: self.metadata.__setitem__(key, value) + ) + self.db._full_resync = mock.MagicMock() + self.patcher = mock.patch.object(grampswebapidb, "DbTxn", FakeDbTxn) + self.patcher.start() + self.addCleanup(self.patcher.stop) + + def test_empty_changes_transaction_triggers_full_resync(self): + page = [{"timestamp": 1.0, "changes": []}] + self.db.web_client.get_transaction_history.return_value = (page, 1) + self.db._sync_from_server() + self.db._full_resync.assert_called_once_with(progress_callback=None) + + def test_normal_transactions_do_not_trigger_full_resync(self): + change = {"obj_class": "Person", "trans_type": TXNADD, "obj_handle": "H1"} + page = [{"timestamp": 1.0, "changes": [change]}] + self.db.web_client.get_transaction_history.return_value = (page, 1) + with mock.patch.object(self.db, "_apply_change", return_value=True): + self.db._sync_from_server() + self.db._full_resync.assert_not_called() + + def test_marker_alongside_real_changes_still_applies_the_real_ones(self): + # A marker transaction doesn't block replaying whatever *is* + # describable elsewhere in the same page -- only the parts the + # history feed genuinely has no record of need the fallback. + change = {"obj_class": "Person", "trans_type": TXNADD, "obj_handle": "H1"} + page = [ + {"timestamp": 1.0, "changes": []}, + {"timestamp": 2.0, "changes": [change]}, + ] + self.db.web_client.get_transaction_history.return_value = (page, 2) + with mock.patch.object(self.db, "_apply_change", return_value=True): + applied = self.db._sync_from_server() + self.assertEqual(applied, 1) + self.db._full_resync.assert_called_once_with(progress_callback=None) + + def test_marker_still_advances_sync_last_time(self): + page = [{"timestamp": 42.0, "changes": []}] + self.db.web_client.get_transaction_history.return_value = (page, 1) + self.db._sync_from_server() + self.assertEqual(self.metadata["sync_last_time"], 42.0) + + +# ------------------------------------------------------------------------- +# +# TestFullResync +# +# ------------------------------------------------------------------------- +class TestFullResync(unittest.TestCase): + def setUp(self): + self.db = new_instance() + self.db.web_client = mock.MagicMock() + self.db.web_client.download_export.return_value = b"fake gramps xml bytes" + self.db.emit = mock.MagicMock() # see TestSyncFromServer.setUp's note + self.patcher = mock.patch.object(grampswebapidb, "DbTxn", FakeDbTxn) + self.patcher.start() + self.addCleanup(self.patcher.stop) + + def test_downloads_export_wipes_and_reimports(self): + # get__handles/remove_ for every primary type, plus + # importData itself, are all faked out -- this test is only + # confirming the wiring (download -> wipe every type -> import + # the downloaded file -> clean up the temp file), not any real + # Gramps object storage. + for key, name in grampswebapidb.KEY_TO_NAME_MAP.items(): + if key not in grampswebapidb.CLASS_TO_KEY_MAP.values(): + continue + setattr(self.db, f"get_{name}_handles", mock.MagicMock(return_value=["H1"])) + setattr(self.db, f"remove_{name}", mock.MagicMock()) + + captured_path = {} + + def fake_import_data(database, filename, user): + captured_path["path"] = filename + self.assertTrue(os.path.exists(filename)) + with open(filename, "rb") as f: + self.assertEqual(f.read(), b"fake gramps xml bytes") + + with mock.patch.object(grampswebapidb, "importData", fake_import_data): + self.db._full_resync() + + self.db.web_client.download_export.assert_called_once_with() + for key, name in grampswebapidb.KEY_TO_NAME_MAP.items(): + if key not in grampswebapidb.CLASS_TO_KEY_MAP.values(): + continue + getattr(self.db, f"remove_{name}").assert_called_once_with("H1", mock.ANY) + # The temp file is cleaned up after import, not left behind. + self.assertFalse(os.path.exists(captured_path["path"])) + # A successful reimport can't be described as specific add/update/ + # delete signals, so every view is told to reload wholesale instead + # -- see request_rebuild() in gramps.gen.db.generic. + emitted = [call.args[0] for call in self.db.emit.call_args_list] + self.assertIn("person-rebuild", emitted) + self.assertIn("family-rebuild", emitted) + + def test_failed_import_does_not_trigger_rebuild(self): + # request_rebuild() sits after importData() in _full_resync(), not + # in a finally -- a reimport that raised partway through left the + # mirror in an unknown state, which is not something to tell every + # view "reload, this is now correct" about. + for key, name in grampswebapidb.KEY_TO_NAME_MAP.items(): + if key not in grampswebapidb.CLASS_TO_KEY_MAP.values(): + continue + setattr(self.db, f"get_{name}_handles", mock.MagicMock(return_value=[])) + setattr(self.db, f"remove_{name}", mock.MagicMock()) + + def failing_import_data(database, filename, user): + raise RuntimeError("boom") + + with mock.patch.object(grampswebapidb, "importData", failing_import_data): + with self.assertRaises(RuntimeError): + self.db._full_resync() + + self.db.emit.assert_not_called() + + def test_no_progress_callback_by_default(self): + for key, name in grampswebapidb.KEY_TO_NAME_MAP.items(): + if key not in grampswebapidb.CLASS_TO_KEY_MAP.values(): + continue + setattr(self.db, f"get_{name}_handles", mock.MagicMock(return_value=[])) + setattr(self.db, f"remove_{name}", mock.MagicMock()) + with mock.patch.object(grampswebapidb, "importData"): + self.db._full_resync() # must not raise + + def test_progress_bookends_the_download_and_reimport(self): + for key, name in grampswebapidb.KEY_TO_NAME_MAP.items(): + if key not in grampswebapidb.CLASS_TO_KEY_MAP.values(): + continue + setattr(self.db, f"get_{name}_handles", mock.MagicMock(return_value=[])) + setattr(self.db, f"remove_{name}", mock.MagicMock()) + progress = mock.MagicMock() + with mock.patch.object(grampswebapidb, "importData"): + self.db._full_resync(progress_callback=progress) + self.assertEqual([call.args[0] for call in progress.call_args_list], [0, 100]) + + def test_progress_not_completed_if_import_fails(self): + # The 100% marker means "the rebuild finished" -- a failed reimport + # must not claim that, same reasoning as request_rebuild() above. + for key, name in grampswebapidb.KEY_TO_NAME_MAP.items(): + if key not in grampswebapidb.CLASS_TO_KEY_MAP.values(): + continue + setattr(self.db, f"get_{name}_handles", mock.MagicMock(return_value=[])) + setattr(self.db, f"remove_{name}", mock.MagicMock()) + + def failing_import_data(database, filename, user): + raise RuntimeError("boom") + + progress = mock.MagicMock() + with mock.patch.object(grampswebapidb, "importData", failing_import_data): + with self.assertRaises(RuntimeError): + self.db._full_resync(progress_callback=progress) + progress.assert_called_once_with(0) + + +# ------------------------------------------------------------------------- +# +# TestTransactionCommit +# +# ------------------------------------------------------------------------- +class TestTransactionCommit(unittest.TestCase): + def setUp(self): + self.db = new_instance() + self.db.web_client = mock.MagicMock() + + def test_no_local_changes_does_not_push(self): + trans = FakeTransaction([]) + with mock.patch.object(grampswebapidb.SQLite, "transaction_commit"): + self.db.transaction_commit(trans) + self.db.web_client.push_transaction.assert_not_called() + + def test_local_changes_are_pushed(self): + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + with mock.patch.object(grampswebapidb.SQLite, "transaction_commit"): + self.db.transaction_commit(trans) + self.db.web_client.push_transaction.assert_called_once() + payload = self.db.web_client.push_transaction.call_args[0][0] + self.assertEqual(payload[0]["handle"], "H1") + + def test_payload_built_before_super_clears_records(self): + # The base class's transaction_commit() clears the transaction's + # records as its last step -- see the module docstring's "must run + # before super()" note. Simulate that by having the (mocked) super + # call wipe the fake transaction, and confirm the push still saw + # the pre-clear data. + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + + def clear_records(transaction): + transaction._records = [] + + with mock.patch.object( + grampswebapidb.SQLite, "transaction_commit", side_effect=clear_records + ): + self.db.transaction_commit(trans) + payload = self.db.web_client.push_transaction.call_args[0][0] + self.assertEqual(len(payload), 1) + + def test_push_failure_is_logged_not_raised(self): + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.web_client.push_transaction.side_effect = HTTPError( + "https://example.com/api/transactions/", 500, "boom", None, None + ) + with mock.patch.object(grampswebapidb.SQLite, "transaction_commit"): + with self.assertLogs(grampswebapidb.LOG, level="ERROR"): + self.db.transaction_commit(trans) # must not raise + + def test_conflict_triggers_resync_then_retry(self): + # A WebApiPushConflict means the server rejected the whole batch + # because something changed server-side since the local mirror's + # snapshot -- the response is to resync from the server and then + # retry the local edit on top of that fresh data (see + # _retry_after_conflict()), not to propagate the exception (the + # local commit already happened). + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.web_client.push_transaction.side_effect = WebApiPushConflict( + "Object has changed" + ) + with mock.patch.object( + grampswebapidb.SQLite, "transaction_commit" + ), mock.patch.object(self.db, "_sync_from_server") as resync, mock.patch.object( + self.db, "_retry_after_conflict" + ) as retry: + with self.assertLogs(grampswebapidb.LOG, level="WARNING"): + self.db.transaction_commit(trans) # must not raise + resync.assert_called_once_with() + retry.assert_called_once() + payload = retry.call_args[0][0] + self.assertEqual(payload[0]["handle"], "H1") + + def test_conflict_resync_failure_is_also_swallowed(self): + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.web_client.push_transaction.side_effect = WebApiPushConflict( + "Object has changed" + ) + with mock.patch.object( + grampswebapidb.SQLite, "transaction_commit" + ), mock.patch.object( + self.db, + "_sync_from_server", + side_effect=HTTPError( + "https://example.com/api/transactions/history/", 500, "boom", None, None + ), + ), mock.patch.object( + self.db, "_retry_after_conflict" + ) as retry: + with self.assertLogs(grampswebapidb.LOG, level="WARNING"): + self.db.transaction_commit(trans) # must not raise + # A failed resync means the mirror still doesn't reflect the + # server, so retrying the edit on top of it would be pointless. + retry.assert_not_called() + + def test_repeated_conflict_on_a_retry_is_not_retried_again(self): + # is_retry=True marks a push that is itself _retry_after_conflict()'s + # replay -- a second conflict on that replay must not recurse into + # another retry, or a genuinely hot object could retry forever. + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.web_client.push_transaction.side_effect = WebApiPushConflict( + "Object has changed" + ) + with mock.patch.object( + grampswebapidb.SQLite, "transaction_commit" + ), mock.patch.object(self.db, "_sync_from_server") as resync, mock.patch.object( + self.db, "_retry_after_conflict" + ) as retry: + with self.assertLogs(grampswebapidb.LOG, level="WARNING"): + self.db._push_payload( + transaction_to_json(trans), is_retry=True + ) # must not raise + resync.assert_called_once_with() + retry.assert_not_called() + + def test_undo_conflict_is_not_retried(self): + # Retrying an undo/redo against data that changed underneath it is + # a murkier case (are we replaying the reversal, or the original + # edit?) than retrying a plain commit -- see the module docstring. + # Left as resync-and-drop, same as before this feature existed. + trans = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.web_client.push_transaction.side_effect = WebApiPushConflict( + "Object has changed" + ) + with mock.patch.object( + grampswebapidb.SQLite, "transaction_commit" + ), mock.patch.object(self.db, "_sync_from_server") as resync, mock.patch.object( + self.db, "_retry_after_conflict" + ) as retry: + with self.assertLogs(grampswebapidb.LOG, level="WARNING"): + self.db._push_payload(transaction_to_json(trans), undo=True) + resync.assert_called_once_with() + retry.assert_not_called() + + +# ------------------------------------------------------------------------- +# +# TestRetryAfterConflict +# +# ------------------------------------------------------------------------- +class TestRetryAfterConflict(unittest.TestCase): + """_retry_after_conflict() replays each payload entry as a fresh + commit_()/remove_() call on top of the mirror _push_payload() + just resynced -- see the module docstring's write-through section. + DbTxn itself is stubbed out (a bare context-manager stand-in) so this + isolates just the replay logic, the same way TestApplyChange isolates + _apply_change() from a real transaction. _merge_or_overwrite() itself + is covered separately by TestMergeOrOverwrite, so here it's mocked out + to just confirm it's consulted (and with what) when the handle still + exists after the resync.""" + + def setUp(self): + self.db = new_instance() + self.db.commit_person = mock.MagicMock() + self.db.remove_person = mock.MagicMock() + self.db.has_person_handle = mock.MagicMock() + self.db.get_person_from_handle = mock.MagicMock() + dbtxn_patch = mock.patch.object(grampswebapidb, "DbTxn") + mock_dbtxn_class = dbtxn_patch.start() + mock_dbtxn_class.return_value.__enter__.return_value = "TRANS" + self.addCleanup(dbtxn_patch.stop) + + def test_add_to_a_handle_that_does_not_exist_is_committed_as_is(self): + # A true add (or an update whose object was deleted server-side in + # the same window) has nothing to merge into. + self.db.has_person_handle.return_value = False + new_data = remove_object(person_data("H1", "I0001")) + payload = [ + { + "type": "add", + "handle": "H1", + "_class": "Person", + "old": None, + "new": new_data, + } + ] + self.db._retry_after_conflict(payload) + self.db.get_person_from_handle.assert_not_called() + self.db.commit_person.assert_called_once() + obj, trans = self.db.commit_person.call_args[0] + self.assertIsInstance(obj, Person) + self.assertEqual(obj.get_handle(), "H1") + self.assertEqual(trans, "TRANS") + + def test_update_of_a_still_present_handle_is_merged_with_the_current_object(self): + self.db.has_person_handle.return_value = True + current = Person() + current.set_handle("H1") + self.db.get_person_from_handle.return_value = current + new_data = remove_object(person_data("H1", "I0002")) + payload = [ + { + "type": "update", + "handle": "H1", + "_class": "Person", + "old": None, + "new": new_data, + } + ] + with mock.patch.object(grampswebapidb, "_merge_or_overwrite") as merge_fn: + merge_fn.return_value = "MERGED" + self.db._retry_after_conflict(payload) + merge_current, merge_local = merge_fn.call_args[0] + self.assertIs(merge_current, current) + self.assertIsInstance(merge_local, Person) + self.assertEqual(merge_local.get_gramps_id(), "I0002") + self.db.commit_person.assert_called_once_with("MERGED", "TRANS") + + def test_delete_removes_if_handle_still_present(self): + self.db.has_person_handle.return_value = True + payload = [ + { + "type": "delete", + "handle": "H1", + "_class": "Person", + "old": remove_object(person_data("H1")), + "new": None, + } + ] + self.db._retry_after_conflict(payload) + self.db.remove_person.assert_called_once_with("H1", "TRANS") + + def test_delete_is_skipped_if_handle_already_gone(self): + # The conflicting server-side change may have been a delete of the + # same object -- nothing left to remove a second time. + self.db.has_person_handle.return_value = False + payload = [ + { + "type": "delete", + "handle": "H1", + "_class": "Person", + "old": remove_object(person_data("H1")), + "new": None, + } + ] + self.db._retry_after_conflict(payload) + self.db.remove_person.assert_not_called() + + def test_unrecognized_class_is_skipped(self): + payload = [ + { + "type": "update", + "handle": "H1", + "_class": "NotAThing", + "old": None, + "new": {}, + } + ] + self.db._retry_after_conflict(payload) # must not raise + self.db.commit_person.assert_not_called() + self.db.has_person_handle.assert_not_called() + + def test_retrying_flag_set_during_replay_and_cleared_after(self): + self.db.has_person_handle.return_value = False + seen = {} + + def check_flag(obj, trans): + seen["during"] = self.db._retrying + + self.db.commit_person.side_effect = check_flag + new_data = remove_object(person_data("H1")) + payload = [ + { + "type": "update", + "handle": "H1", + "_class": "Person", + "old": None, + "new": new_data, + } + ] + self.db._retry_after_conflict(payload) + self.assertTrue(seen["during"]) + self.assertFalse(self.db._retrying) + + def test_retrying_flag_cleared_even_if_commit_raises(self): + self.db.has_person_handle.return_value = False + self.db.commit_person.side_effect = RuntimeError("boom") + new_data = remove_object(person_data("H1")) + payload = [ + { + "type": "update", + "handle": "H1", + "_class": "Person", + "old": None, + "new": new_data, + } + ] + with self.assertRaises(RuntimeError): + self.db._retry_after_conflict(payload) + self.assertFalse(self.db._retrying) + + +# ------------------------------------------------------------------------- +# +# TestMergeOrOverwrite +# +# ------------------------------------------------------------------------- +class TestMergeOrOverwrite(unittest.TestCase): + """_merge_or_overwrite() ports GrampsWebSync's diffhandler.py A_MRG_REM + handling: combine two edits of the same object via the object's own + merge() -- the same list-unioning logic behind Gramps' Merge People/ + Family/... tools -- rather than letting one edit silently clobber the + other. Uses real Person/Tag objects rather than mocks, since the whole + point is exercising Gramps' own merge() implementation.""" + + def test_list_valued_fields_from_both_sides_are_unioned(self): + current = Person() + current.set_handle("H1") + current.set_gramps_id("I0001") + current.add_note("N-remote") + + local = Person() + local.set_handle("H1") + local.set_gramps_id("I0002") + local.add_note("N-local") + + merged = grampswebapidb._merge_or_overwrite(current, local) + self.assertEqual(set(merged.get_note_list()), {"N-remote", "N-local"}) + + def test_current_object_is_not_mutated(self): + current = Person() + current.set_handle("H1") + current.add_note("N-remote") + local = Person() + local.set_handle("H1") + local.add_note("N-local") + + grampswebapidb._merge_or_overwrite(current, local) + self.assertEqual(current.get_note_list(), ["N-remote"]) + + def test_local_obj_gramps_id_is_cleared_before_merging(self): + # merge() tags on a "Merged Gramps ID" attribute if the acquisition + # has a gramps_id -- appropriate for absorbing a second, separate + # object (Gramps' Merge People tool), but this is one object edited + # twice, not two objects becoming one, so that attribute must not + # appear, and local's own gramps_id object must be untouched. + current = Person() + current.set_handle("H1") + local = Person() + local.set_handle("H1") + local.set_gramps_id("I0002") + + merged = grampswebapidb._merge_or_overwrite(current, local) + self.assertEqual(merged.get_attribute_list(), []) + self.assertEqual(local.get_gramps_id(), "I0002") + + def test_type_without_a_real_merge_falls_back_to_local_obj(self): + # Tag only inherits BaseObject's no-op merge() -- "merging" into it + # would silently keep current's content and drop the local edit. + current = Tag() + current.set_handle("H1") + current.set_name("Remote name") + local = Tag() + local.set_handle("H1") + local.set_name("Local name") + + result = grampswebapidb._merge_or_overwrite(current, local) + self.assertIs(result, local) + + +# ------------------------------------------------------------------------- +# +# TestUndoRedo +# +# ------------------------------------------------------------------------- +class TestUndoRedo(unittest.TestCase): + """undo()/redo() peek the relevant DbTxn off DbGenericUndo's queue, + turn it back into a change-list payload, and push it -- undo via + push_transaction(..., undo=True) (server reverses it), redo via a + plain push (same as an ordinary commit). Both delegate to + _push_payload(), whose conflict/error handling is already covered by + TestTransactionCommit, so these just confirm the wiring: the right + payload, the right undo flag, and the peek-before-super ordering.""" + + def setUp(self): + self.db = new_instance() + self.db.undodb = mock.MagicMock() + + def test_undo_pushes_with_undo_flag(self): + txn = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.undodb.undo_count = 1 + self.db.undodb.undoq = [txn] + self.db.undodb.undo.return_value = True + with mock.patch.object(self.db, "_push_payload") as push: + result = self.db.undo() + self.assertTrue(result) + push.assert_called_once() + payload = push.call_args[0][0] + self.assertEqual(payload[0]["handle"], "H1") + self.assertEqual(push.call_args.kwargs, {"undo": True}) + + def test_redo_pushes_without_undo_flag(self): + txn = FakeTransaction([(0, TXNDEL, "H1", person_data("H1"), None)]) + self.db.undodb.redo_count = 1 + self.db.undodb.redoq = [txn] + self.db.undodb.redo.return_value = True + with mock.patch.object(self.db, "_push_payload") as push: + result = self.db.redo() + self.assertTrue(result) + payload = push.call_args[0][0] + self.assertEqual(payload[0]["handle"], "H1") + self.assertEqual(push.call_args.kwargs, {}) + + def test_no_push_when_nothing_to_undo(self): + self.db.undodb.undo_count = 0 + self.db.undodb.undo.return_value = False + with mock.patch.object(self.db, "_push_payload") as push: + result = self.db.undo() + self.assertFalse(result) + push.assert_not_called() + + def test_no_push_when_nothing_to_redo(self): + self.db.undodb.redo_count = 0 + self.db.undodb.redo.return_value = False + with mock.patch.object(self.db, "_push_payload") as push: + result = self.db.redo() + self.assertFalse(result) + push.assert_not_called() + + def test_undo_not_pushed_if_super_reports_nothing_undone(self): + # undo_count > 0 doesn't guarantee _undo() actually ran (e.g. a + # readonly db -- see DbUndo.undo()); only a truthy result pushes. + txn = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.undodb.undo_count = 1 + self.db.undodb.undoq = [txn] + self.db.undodb.undo.return_value = False + with mock.patch.object(self.db, "_push_payload") as push: + result = self.db.undo() + self.assertFalse(result) + push.assert_not_called() + + def test_transaction_grabbed_before_super_pops_the_queue(self): + # WebApiDB.undo() must read undoq[-1] before delegating to + # super().undo() (-> DbGenericUndo._undo(), which pops it) -- grab + # it too late and the payload would be built from the wrong (or a + # missing) transaction. + txn = FakeTransaction([(0, TXNADD, "H1", None, person_data("H1"))]) + self.db.undodb.undo_count = 1 + self.db.undodb.undoq = [txn] + + def pop_on_undo(update_history): + self.db.undodb.undoq.pop() + return True + + self.db.undodb.undo.side_effect = pop_on_undo + with mock.patch.object(self.db, "_push_payload") as push: + self.db.undo() + payload = push.call_args[0][0] + self.assertEqual(payload[0]["handle"], "H1") + + +# ------------------------------------------------------------------------- +# +# TestMisc +# +# ------------------------------------------------------------------------- +class TestMisc(unittest.TestCase): + def test_requires_login_is_false(self): + self.assertFalse(new_instance().requires_login()) + + def test_initialize_wraps_connection_errors(self): + db = new_instance() + with mock.patch.object( + grampswebapidb.WebApiHandler, + "from_env", + side_effect=ValueError("GRAMPS_WEB_API_KEY is not set"), + ): + with self.assertRaises(DbConnectionError): + db._initialize("/tmp/some-tree", None, None) + + def test_initialize_stores_web_client_and_calls_super(self): + db = new_instance() + sentinel_client = mock.MagicMock() + with mock.patch.object( + grampswebapidb.WebApiHandler, "from_env", return_value=sentinel_client + ), mock.patch.object(grampswebapidb.SQLite, "_initialize") as super_init: + db._initialize("/tmp/some-tree", "user", "pw") + self.assertIs(db.web_client, sentinel_client) + super_init.assert_called_once_with("/tmp/some-tree", "user", "pw") + + +# ------------------------------------------------------------------------- +# +# TestCheckIdentity +# +# Nothing but a Family Tree's own name ties its local mirror to one +# particular GRAMPS_WEB_API_KEY account (see the module docstring) -- +# _check_identity() requires that name to be "@" for +# whoever the current key authenticates as, so pointing the key at a +# different account while reopening the same tree fails loudly at load() +# instead of quietly mixing that account's data into the old mirror. +# +# ------------------------------------------------------------------------- +class TestCheckIdentity(unittest.TestCase): + def setUp(self): + self.db = new_instance() + self.db._directory = "/tmp/some-tree" + self.db.web_client = mock.MagicMock() + self.db.web_client.get_identity.return_value = "dblank@hadaly.duckdns.org" + + def test_matching_name_passes(self): + self.db.get_dbname = mock.MagicMock(return_value="dblank@hadaly.duckdns.org") + self.db._check_identity() # must not raise + + def test_name_sanitized_the_same_way_dbman_does_still_passes(self): + # gramps.gui.dbman's Family Tree Manager replaces "." (among other + # characters) with "_" in any name typed through its rename UI, so + # a hostname's dots can never actually reach name.txt -- the check + # must accept the sanitized form as a match, not just the literal + # "@" string. + self.db.get_dbname = mock.MagicMock(return_value="dblank@hadaly_duckdns_org") + self.db._check_identity() # must not raise + + def test_mismatched_name_raises(self): + self.db.get_dbname = mock.MagicMock(return_value="Gramps Web API DB") + with self.assertRaises(DbConnectionError): + self.db._check_identity() + + def test_mismatch_error_names_the_typeable_form(self): + self.db.get_dbname = mock.MagicMock(return_value="Gramps Web API DB") + with self.assertRaises(DbConnectionError) as ctx: + self.db._check_identity() + self.assertIn("dblank@hadaly_duckdns_org", str(ctx.exception)) + + def test_connection_error_resolving_identity_is_wrapped(self): + self.db.get_dbname = mock.MagicMock(return_value="dblank@hadaly.duckdns.org") + self.db.web_client.get_identity.side_effect = HTTPError( + "https://example.com/api/users/-/", 500, "boom", None, None + ) + with self.assertRaises(DbConnectionError): + self.db._check_identity() + + +# ------------------------------------------------------------------------- +# +# TestPolling +# +# load() schedules a GLib.timeout_add_seconds() tick that re-syncs for as +# long as the database stays open (see the module docstring's polling +# section); close() must cancel it so a closed database doesn't keep +# polling on a connection that's going away. +# +# ------------------------------------------------------------------------- +class TestPolling(unittest.TestCase): + def setUp(self): + self.db = new_instance() + + def test_load_syncs_and_schedules_polling(self): + with mock.patch.object( + grampswebapidb.SQLite, "load" + ) as super_load, mock.patch.object( + self.db, "_check_identity" + ) as check_identity, mock.patch.object( + self.db, "_sync_from_server" + ) as sync, mock.patch.object( + grampswebapidb.GLib, "timeout_add_seconds", return_value=42 + ) as timeout_add: + self.db.load("some/path") + super_load.assert_called_once_with("some/path") + check_identity.assert_called_once_with() + sync.assert_called_once_with(progress_callback=None) + timeout_add.assert_called_once_with( + grampswebapidb.POLL_INTERVAL_SECONDS, self.db._poll_tick + ) + self.assertEqual(self.db._poll_source_id, 42) + + def test_load_forwards_positional_callback_to_sync(self): + # DbGeneric.load()'s own signature is (directory, callback=None, + # mode=..., ...) -- cli/grampscli.py calls it positionally + # (db.load(filename, self._pulse_progress, mode, ...)), so load() + # must recognize the callback there too, not just as a kwarg. + my_callback = mock.MagicMock() + with mock.patch.object(grampswebapidb.SQLite, "load"), mock.patch.object( + self.db, "_check_identity" + ), mock.patch.object(self.db, "_sync_from_server") as sync, mock.patch.object( + grampswebapidb.GLib, "timeout_add_seconds" + ): + self.db.load("some/path", my_callback, "w") + sync.assert_called_once_with(progress_callback=my_callback) + + def test_load_forwards_keyword_callback_to_sync(self): + my_callback = mock.MagicMock() + with mock.patch.object(grampswebapidb.SQLite, "load"), mock.patch.object( + self.db, "_check_identity" + ), mock.patch.object(self.db, "_sync_from_server") as sync, mock.patch.object( + grampswebapidb.GLib, "timeout_add_seconds" + ): + self.db.load("some/path", callback=my_callback) + sync.assert_called_once_with(progress_callback=my_callback) + + def test_close_cancels_pending_poll(self): + self.db._poll_source_id = 42 + with mock.patch.object( + grampswebapidb.SQLite, "close" + ) as super_close, mock.patch.object( + grampswebapidb.GLib, "source_remove" + ) as source_remove: + self.db.close() + source_remove.assert_called_once_with(42) + self.assertIsNone(self.db._poll_source_id) + super_close.assert_called_once_with() + + def test_close_without_a_poll_scheduled_is_a_no_op(self): + # e.g. close() called after a failed load(), before the timeout + # was ever scheduled. + with mock.patch.object( + grampswebapidb.SQLite, "close" + ) as super_close, mock.patch.object( + grampswebapidb.GLib, "source_remove" + ) as source_remove: + self.db.close() + source_remove.assert_not_called() + super_close.assert_called_once_with() + + def test_poll_tick_syncs_and_keeps_repeating(self): + with mock.patch.object(self.db, "_sync_from_server") as sync: + result = self.db._poll_tick() + sync.assert_called_once_with() + self.assertEqual(result, grampswebapidb.GLib.SOURCE_CONTINUE) + + def test_poll_tick_swallows_connection_errors_and_keeps_repeating(self): + with mock.patch.object( + self.db, "_sync_from_server", side_effect=OSError("network down") + ): + with self.assertLogs(grampswebapidb.LOG, level="ERROR"): + result = self.db._poll_tick() + self.assertEqual(result, grampswebapidb.GLib.SOURCE_CONTINUE) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebApiDb/tests/test_webapi_client.py b/GrampsWebApiDb/tests/test_webapi_client.py new file mode 100644 index 000000000..7b18cfdd2 --- /dev/null +++ b/GrampsWebApiDb/tests/test_webapi_client.py @@ -0,0 +1,764 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Unit tests for webapi_client.WebApiHandler and its helper functions. + +No real Gramps Web API server is contacted: urlopen() is patched throughout, +via a small FakeResponse context manager and a queue of canned +responses/exceptions. Covers: + + - the GRAMPS_WEB_API_KEY codec (make_api_key/parse_api_key round trip) + - JWT payload decoding + - username/password vs. refresh-token authentication + - the 429 rate-limit backoff-and-retry-once behavior + - the "no /api prefix yet" fallback retry + - 401 re-authentication on expired access tokens + - transaction_history/push_transaction request shape + +Run with:: + + python3 -m unittest GrampsWebApiDb.tests.test_webapi_client -v +""" + +# ------------------------------------------------------------------------- +# +# Standard python modules +# +# ------------------------------------------------------------------------- +import base64 +import io +import json +import os +import sys +import unittest +from urllib.error import HTTPError, URLError +from unittest import mock + +# ------------------------------------------------------------------------- +# +# Make the addon importable the way Gramps loads it: its own directory on +# sys.path (grampswebapidb.py/webapi_client.py use bare, not package- +# relative, imports of each other -- see CLAUDE.md Testing conventions). +# +# ------------------------------------------------------------------------- +ADDON_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + +try: + import gramps # noqa: F401 (only to trigger the SkipTest below if absent) +except ImportError as _err: + raise unittest.SkipTest("gramps package not available: %s" % _err) + +from GrampsWebApiDb import webapi_client +from GrampsWebApiDb.webapi_client import ( + WebApiHandler, + WebApiPushConflict, + decode_jwt_payload, + make_api_key, + parse_api_key, +) + + +# ------------------------------------------------------------------------- +# +# Test helpers +# +# ------------------------------------------------------------------------- +def b64url_json(payload: dict) -> str: + """Base64url-encode a dict, stripped of padding, like a real JWT segment.""" + raw = json.dumps(payload).encode() + return base64.urlsafe_b64encode(raw).decode().rstrip("=") + + +def fake_jwt(payload: dict) -> str: + """A JWT-shaped string whose payload segment decodes to ``payload``. + + decode_jwt_payload() only ever looks at segment [1], so the header and + signature segments don't need to be real. + """ + return f"header.{b64url_json(payload)}.signature" + + +def token(tag: str) -> str: + """A distinct, real-JWT-shaped access token for ``tag``. + + Every access token this module hands back (even a canned "AT1"-style + placeholder) is real enough to satisfy decode_jwt_payload(), because + access_token's getter unconditionally checks the token's remaining + lifetime -- see get_access_token_remaining_time(). + """ + return fake_jwt({"tag": tag}) + + +class FakeResponse: + """Stand-in for the object returned by ``urlopen(...).__enter__()``.""" + + def __init__(self, body=None, headers=None): + self._body = json.dumps(body if body is not None else {}).encode() + self.headers = headers or {} + + def read(self): + return self._body + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + return False + + +class FakeBinaryResponse: + """Like FakeResponse, but for endpoints that return a raw file body + rather than JSON (see download_export()/_get_binary()).""" + + def __init__(self, body: bytes, headers=None): + self._body = body + self.headers = headers or {} + + def read(self): + return self._body + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + return False + + +def http_error(code, url="https://example.com/api"): + return HTTPError(url, code, f"HTTP {code}", None, None) + + +def http_error_with_body(code, body, url="https://example.com/api"): + """An HTTPError whose .read() yields a JSON-encoded body, the way a + real gramps-web-api error response (abort_with_message()) looks.""" + fp = io.BytesIO(json.dumps(body).encode()) + return HTTPError(url, code, f"HTTP {code}", None, fp) + + +class QueuedUrlopen: + """``urlopen`` replacement that returns/raises each queued item in turn, + recording every ``Request`` it was called with.""" + + def __init__(self, items): + self._items = list(items) + self.requests = [] + + def __call__(self, req, context=None, timeout=None): + self.requests.append(req) + item = self._items.pop(0) + if isinstance(item, Exception): + raise item + return item + + +# ------------------------------------------------------------------------- +# +# TestApiKeyCodec +# +# ------------------------------------------------------------------------- +class TestApiKeyCodec(unittest.TestCase): + """make_api_key()/parse_api_key() are inverses, and reject malformed input.""" + + def test_roundtrip(self): + key = make_api_key("refresh-tok-123", "https://example.com/api") + self.assertEqual( + parse_api_key(key), ("refresh-tok-123", "https://example.com/api") + ) + + def test_roundtrip_with_padding_needed(self): + # A URL whose base64url encoding needs '=' padding restored. + url = "https://example.com/api/x" + key = make_api_key("tok", url) + self.assertEqual(parse_api_key(key), ("tok", url)) + + def test_missing_delimiter_is_malformed(self): + with self.assertRaises(ValueError): + parse_api_key("no-delimiter-here") + + def test_bad_url_encoding_is_malformed(self): + with self.assertRaises(ValueError): + parse_api_key("tok*not-valid-base64---") + + def test_empty_token_is_rejected(self): + encoded_url = base64.urlsafe_b64encode(b"https://example.com").decode() + with self.assertRaises(ValueError): + parse_api_key("*" + encoded_url) + + def test_empty_url_is_rejected(self): + encoded_empty = base64.urlsafe_b64encode(b"").decode() + with self.assertRaises(ValueError): + parse_api_key("tok*" + encoded_empty) + + +# ------------------------------------------------------------------------- +# +# TestDecodeJwtPayload +# +# ------------------------------------------------------------------------- +class TestDecodeJwtPayload(unittest.TestCase): + def test_decodes_payload_claims(self): + jwt_str = fake_jwt({"sub": "user1", "exp": 1234}) + self.assertEqual(decode_jwt_payload(jwt_str), {"sub": "user1", "exp": 1234}) + + def test_handles_payload_needing_padding(self): + # Pick a payload whose base64url segment length isn't a multiple of 4, + # to exercise the padding-restoration branch. + jwt_str = fake_jwt({"a": "bit-of-text-to-shift-the-length"}) + payload = decode_jwt_payload(jwt_str) + self.assertEqual(payload["a"], "bit-of-text-to-shift-the-length") + + +# ------------------------------------------------------------------------- +# +# TestAuthentication +# +# ------------------------------------------------------------------------- +class TestAuthentication(unittest.TestCase): + """Constructing a handler authenticates once, via whichever credential + was supplied.""" + + def test_username_password_login(self): + fake = QueuedUrlopen( + [FakeResponse({"access_token": token("AT1"), "refresh_token": "RT1"})] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler( + "https://example.com/api", username="alice", password="secret" + ) + self.assertEqual(handler._access_token, token("AT1")) + self.assertEqual(handler._refresh_token, "RT1") + req = fake.requests[0] + self.assertEqual(req.full_url, "https://example.com/api/token/") + self.assertEqual( + json.loads(req.data), {"username": "alice", "password": "secret"} + ) + + def test_refresh_token_login(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT2")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT0") + self.assertEqual(handler._access_token, token("AT2")) + # Refresh token is unchanged; the endpoint used is /token/refresh/. + self.assertEqual(handler._refresh_token, "RT0") + req = fake.requests[0] + self.assertEqual(req.full_url, "https://example.com/api/token/refresh/") + self.assertEqual(req.get_header("Authorization"), "Bearer RT0") + + def test_mint_api_key_returns_encoded_key(self): + fake = QueuedUrlopen( + [FakeResponse({"access_token": token("AT1"), "refresh_token": "RT1"})] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + key = WebApiHandler.mint_api_key( + "https://example.com/api", "alice", "secret" + ) + self.assertEqual(parse_api_key(key), ("RT1", "https://example.com/api")) + + def test_mint_api_key_requires_refresh_token_in_response(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT1")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(ValueError): + WebApiHandler.mint_api_key("https://example.com/api", "alice", "pw") + + def test_from_env_missing_var_raises(self): + with mock.patch.dict(os.environ, {}, clear=True): + with self.assertRaises(ValueError): + WebApiHandler.from_env() + + def test_from_env_builds_handler_from_refresh_key(self): + key = make_api_key("RT9", "https://example.com/api") + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT9")})]) + with mock.patch.dict(os.environ, {webapi_client.API_KEY_ENV_VAR: key}): + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler.from_env() + self.assertEqual(handler.url, "https://example.com/api") + self.assertEqual(handler._access_token, token("AT9")) + + +# ------------------------------------------------------------------------- +# +# TestAccessTokenProperty +# +# ------------------------------------------------------------------------- +class TestAccessTokenProperty(unittest.TestCase): + def _handler_with_token(self, exp_offset): + """A handler whose access token expires ``exp_offset`` seconds from now.""" + fake = QueuedUrlopen( + [FakeResponse({"access_token": fake_jwt({"exp": time_now() + exp_offset})})] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + return handler, fake + + def test_remaining_time_none_without_exp_claim(self): + fake = QueuedUrlopen([FakeResponse({"access_token": fake_jwt({})})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + self.assertIsNone(handler.get_access_token_remaining_time()) + + def test_access_token_refreshes_when_near_expiry(self): + handler, fake = self._handler_with_token(exp_offset=30) # < 60s left + fake._items.append(FakeResponse({"access_token": token("FRESH")})) + with mock.patch.object(webapi_client, "urlopen", fake): + refreshed = handler.access_token + self.assertEqual(refreshed, token("FRESH")) + self.assertEqual(len(fake.requests), 2) # initial auth + re-auth + + def test_access_token_reused_when_far_from_expiry(self): + handler, fake = self._handler_with_token(exp_offset=3600) + access_token = handler.access_token + self.assertEqual(len(fake.requests), 1) # no re-auth triggered + self.assertTrue(access_token) + + def test_get_permissions_reads_token_claim(self): + fake = QueuedUrlopen( + [ + FakeResponse( + {"access_token": fake_jwt({"permissions": ["edit", "view"]})} + ) + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + self.assertEqual(handler.get_permissions(), ["edit", "view"]) + + +# ------------------------------------------------------------------------- +# +# TestIdentity +# +# hostname/get_current_username()/get_identity() resolve who and where a +# credential authenticates as -- grampswebapidb.py's _check_identity() uses +# get_identity() to bind a local mirror to one particular server account +# (see that module's docstring). +# +# ------------------------------------------------------------------------- +class TestIdentity(unittest.TestCase): + def _handler(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT1")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + return handler, fake + + def test_hostname_strips_scheme_and_path(self): + handler, _fake = self._handler() + self.assertEqual(handler.hostname, "example.com") + + def test_get_current_username_resolves_via_users_dash_and_caches(self): + # A refresh-token credential (the normal from_env() path) carries + # no plaintext username -- unlike get_permissions()'s claim, this + # isn't in the JWT at all, so it takes a real GET /users/-/. + handler, fake = self._handler() + fake._items.append(FakeResponse({"name": "dblank"})) + with mock.patch.object(webapi_client, "urlopen", fake): + self.assertEqual(handler.get_current_username(), "dblank") + self.assertEqual(handler.username, "dblank") + self.assertEqual(fake.requests[-1].full_url, "https://example.com/api/users/-/") + + # Cached: a second call makes no further request. + fake._items.append(FakeResponse({"name": "someone-else"})) + with mock.patch.object(webapi_client, "urlopen", fake): + self.assertEqual(handler.get_current_username(), "dblank") + + def test_get_current_username_skips_lookup_for_password_login(self): + # A username+password login already knows its own username -- + # no need to ask the server to confirm it. + fake = QueuedUrlopen( + [FakeResponse({"access_token": token("AT1"), "refresh_token": "RT"})] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler( + "https://example.com/api", username="dblank", password="pw" + ) + self.assertEqual(handler.get_current_username(), "dblank") + self.assertEqual(len(fake.requests), 1) + + def test_get_identity_combines_username_and_hostname(self): + handler, fake = self._handler() + fake._items.append(FakeResponse({"name": "dblank"})) + with mock.patch.object(webapi_client, "urlopen", fake): + self.assertEqual(handler.get_identity(), "dblank@example.com") + + +def time_now(): + import time + + return time.time() + + +# ------------------------------------------------------------------------- +# +# TestRateLimitAndFallbackRetries +# +# ------------------------------------------------------------------------- +class TestRateLimitAndFallbackRetries(unittest.TestCase): + """429 responses back off and retry once; a URL missing '/api' is + retried with it appended.""" + + def test_fetch_token_retries_once_after_429(self): + fake = QueuedUrlopen( + [ + http_error(429), + FakeResponse({"access_token": token("AT"), "refresh_token": "RT"}), + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ) as mock_sleep: + handler = WebApiHandler( + "https://example.com/api", username="alice", password="pw" + ) + self.assertEqual(handler._access_token, token("AT")) + mock_sleep.assert_called_once_with(webapi_client.RATE_LIMIT_BACKOFF) + + def test_refresh_retries_once_after_429(self): + fake = QueuedUrlopen([http_error(429), FakeResponse({"access_token": token("AT")})]) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + self.assertEqual(handler._access_token, token("AT")) + + def test_fetch_token_appends_api_prefix_on_non_rate_limit_error(self): + fake = QueuedUrlopen( + [ + http_error(404, url="https://example.com/token/"), + FakeResponse({"access_token": token("AT"), "refresh_token": "RT"}), + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler( + "https://example.com", username="alice", password="pw" + ) + self.assertEqual(handler.url, "https://example.com/api") + self.assertEqual(fake.requests[1].full_url, "https://example.com/api/token/") + + def test_fetch_token_does_not_re_append_api_prefix(self): + # If the URL already ends in /api, a second failure must propagate + # rather than looping. + fake = QueuedUrlopen([http_error(404), http_error(404)]) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(HTTPError): + WebApiHandler("https://example.com/api", username="a", password="p") + + +# ------------------------------------------------------------------------- +# +# TestGetJsonRetries +# +# ------------------------------------------------------------------------- +class TestGetJsonRetries(unittest.TestCase): + """_get_json() re-authenticates on 401, backs off on 429, and retries + once on a transient network error.""" + + def _authed_handler(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT0")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + return handler + + def test_401_triggers_reauth_and_one_retry(self): + handler = self._authed_handler() + fake = QueuedUrlopen( + [ + http_error(401), + FakeResponse({"access_token": token("AT1")}), # the re-auth call + FakeResponse({"ok": True}), + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + body, _headers = handler._get_json("https://example.com/api/thing/") + self.assertEqual(body, {"ok": True}) + self.assertEqual(handler._access_token, token("AT1")) + + def test_429_retries_once(self): + handler = self._authed_handler() + fake = QueuedUrlopen([http_error(429), FakeResponse({"ok": True})]) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + body, _headers = handler._get_json("https://example.com/api/thing/") + self.assertEqual(body, {"ok": True}) + + def test_network_error_retries_once(self): + handler = self._authed_handler() + fake = QueuedUrlopen( + [URLError("connection refused"), FakeResponse({"ok": True})] + ) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + body, _headers = handler._get_json("https://example.com/api/thing/") + self.assertEqual(body, {"ok": True}) + + def test_second_failure_propagates(self): + handler = self._authed_handler() + fake = QueuedUrlopen([http_error(500), http_error(500)]) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(HTTPError): + handler._get_json("https://example.com/api/thing/") + + +# ------------------------------------------------------------------------- +# +# TestTransactionHistory +# +# ------------------------------------------------------------------------- +class TestTransactionHistory(unittest.TestCase): + def _authed_handler(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT0")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + return handler + + def test_request_shape_and_total_count_header(self): + handler = self._authed_handler() + body = [{"id": 1, "timestamp": 10.0, "changes": []}] + fake = QueuedUrlopen([FakeResponse(body, headers={"X-Total-Count": "5"})]) + with mock.patch.object(webapi_client, "urlopen", fake): + transactions, total = handler.get_transaction_history( + after=100, page=2, pagesize=50 + ) + self.assertEqual(transactions, body) + self.assertEqual(total, 5) + url = fake.requests[0].full_url + self.assertIn("after=100", url) + self.assertIn("new=1", url) + self.assertIn("sort=id", url) + self.assertIn("page=2", url) + self.assertIn("pagesize=50", url) + + def test_total_count_falls_back_to_body_length(self): + handler = self._authed_handler() + body = [{"id": 1, "timestamp": 1.0, "changes": []}] * 3 + fake = QueuedUrlopen([FakeResponse(body)]) # no X-Total-Count header + with mock.patch.object(webapi_client, "urlopen", fake): + _transactions, total = handler.get_transaction_history() + self.assertEqual(total, 3) + + +# ------------------------------------------------------------------------- +# +# TestDownloadExport +# +# ------------------------------------------------------------------------- +class TestDownloadExport(unittest.TestCase): + """download_export() (grampswebapidb.py's _full_resync() fallback) + hits GET /exporters//file and returns the raw body, + sharing _get_binary()'s 401/429/network retry behavior with + _get_json().""" + + def _authed_handler(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT0")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + return handler + + def test_request_url_and_returns_raw_bytes(self): + handler = self._authed_handler() + fake = QueuedUrlopen([FakeBinaryResponse(b"gzip-bytes-here")]) + with mock.patch.object(webapi_client, "urlopen", fake): + data = handler.download_export() + self.assertEqual(data, b"gzip-bytes-here") + self.assertEqual( + fake.requests[0].full_url, "https://example.com/api/exporters/gramps/file" + ) + + def test_extension_is_configurable(self): + handler = self._authed_handler() + fake = QueuedUrlopen([FakeBinaryResponse(b"gedcom-bytes")]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler.download_export(extension="ged") + self.assertEqual( + fake.requests[0].full_url, "https://example.com/api/exporters/ged/file" + ) + + def test_401_triggers_reauth_and_retry(self): + handler = self._authed_handler() + fake = QueuedUrlopen( + [ + http_error(401), + FakeResponse({"access_token": token("AT1")}), # the re-auth call + FakeBinaryResponse(b"data-after-reauth"), + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + data = handler.download_export() + self.assertEqual(data, b"data-after-reauth") + + def test_429_retries_once(self): + handler = self._authed_handler() + fake = QueuedUrlopen([http_error(429), FakeBinaryResponse(b"data")]) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + data = handler.download_export() + self.assertEqual(data, b"data") + + def test_second_failure_propagates(self): + handler = self._authed_handler() + fake = QueuedUrlopen([http_error(500), http_error(500)]) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(HTTPError): + handler.download_export() + + +# ------------------------------------------------------------------------- +# +# TestPushTransaction +# +# ------------------------------------------------------------------------- +class TestPushTransaction(unittest.TestCase): + def _authed_handler(self): + fake = QueuedUrlopen([FakeResponse({"access_token": token("AT0")})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler = WebApiHandler("https://example.com/api", refresh_token="RT") + return handler + + def test_empty_payload_sends_no_request(self): + handler = self._authed_handler() + fake = QueuedUrlopen([]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler.push_transaction([]) + self.assertEqual(fake.requests, []) + + def test_non_empty_payload_posts_without_force(self): + # No force=1: the server's old-data-mismatch check must run, or + # WebApiPushConflict can never fire -- see push_transaction()'s + # docstring. + handler = self._authed_handler() + payload = [{"type": "add", "handle": "H1", "_class": "Person"}] + fake = QueuedUrlopen([FakeResponse({})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler.push_transaction(payload) + req = fake.requests[0] + self.assertEqual(req.full_url, "https://example.com/api/transactions/") + self.assertEqual(req.get_method(), "POST") + self.assertEqual(json.loads(req.data), payload) + self.assertEqual(req.get_header("Authorization"), f"Bearer {token('AT0')}") + + def test_undo_appends_query_param(self): + handler = self._authed_handler() + payload = [{"type": "add", "handle": "H1", "_class": "Person"}] + fake = QueuedUrlopen([FakeResponse({})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler.push_transaction(payload, undo=True) + req = fake.requests[0] + self.assertEqual( + req.full_url, "https://example.com/api/transactions/?undo=1" + ) + # The payload itself is the original (forward) one -- the server + # reverses it, not the caller. See push_transaction()'s docstring. + self.assertEqual(json.loads(req.data), payload) + + def test_undo_defaults_to_false(self): + handler = self._authed_handler() + fake = QueuedUrlopen([FakeResponse({})]) + with mock.patch.object(webapi_client, "urlopen", fake): + handler.push_transaction([{"type": "add"}]) + self.assertEqual(fake.requests[0].full_url, "https://example.com/api/transactions/") + + def test_undo_flag_survives_401_retry(self): + handler = self._authed_handler() + fake = QueuedUrlopen( + [ + http_error(401), + FakeResponse({"access_token": token("AT1")}), + FakeResponse({}), + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + handler.push_transaction([{"type": "add"}], undo=True) + # requests[0] = failed push, [1] = re-auth, [2] = retried push + self.assertEqual( + fake.requests[2].full_url, "https://example.com/api/transactions/?undo=1" + ) + + def test_401_triggers_reauth_and_retry(self): + handler = self._authed_handler() + fake = QueuedUrlopen( + [http_error(401), FakeResponse({"access_token": token("AT1")}), FakeResponse({})] + ) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + handler.push_transaction([{"type": "add"}]) + self.assertEqual(handler._access_token, token("AT1")) + + def test_429_retries_once(self): + handler = self._authed_handler() + fake = QueuedUrlopen([http_error(429), FakeResponse({})]) + with mock.patch.object(webapi_client, "urlopen", fake), mock.patch.object( + webapi_client, "sleep" + ): + handler.push_transaction([{"type": "add"}]) + self.assertEqual(len(fake.requests), 2) + + def test_object_changed_400_raises_push_conflict(self): + handler = self._authed_handler() + fake = QueuedUrlopen( + [ + http_error_with_body( + 400, {"error": {"code": 400, "message": "Object has changed"}} + ) + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(WebApiPushConflict): + handler.push_transaction([{"type": "add"}]) + # Not retried -- a conflict isn't transient, so exactly one request. + self.assertEqual(len(fake.requests), 1) + + def test_other_400_reasons_are_not_conflicts(self): + # e.g. a payload item missing a required Gramps ID -- our own bug, + # not a concurrent edit -- must propagate as a plain HTTPError. + handler = self._authed_handler() + fake = QueuedUrlopen( + [ + http_error_with_body( + 400, {"error": {"code": 400, "message": "Gramps ID missing"}} + ) + ] + ) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(HTTPError) as ctx: + handler.push_transaction([{"type": "add"}]) + self.assertNotIsInstance(ctx.exception, WebApiPushConflict) + + def test_400_with_unparseable_body_propagates_as_http_error(self): + handler = self._authed_handler() + fake = QueuedUrlopen([http_error_with_body(400, "not-a-dict-body")]) + with mock.patch.object(webapi_client, "urlopen", fake): + with self.assertRaises(HTTPError) as ctx: + handler.push_transaction([{"type": "add"}]) + self.assertNotIsInstance(ctx.exception, WebApiPushConflict) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebApiDb/webapi_client.py b/GrampsWebApiDb/webapi_client.py new file mode 100644 index 000000000..c90145db6 --- /dev/null +++ b/GrampsWebApiDb/webapi_client.py @@ -0,0 +1,525 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2021-2024 David Straub +# Copyright (C) 2026 Douglas S. Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Minimal Gramps Web API client: authentication and read access. + +Trimmed from the WebApiHandler class in the GrampsWebSync addon (same +repo, same license) -- credit to David Straub for the original token +fetch/refresh and SSL-context handling. Dropped everything specific to +GrampsWebSync's push-a-local-transaction / XML-export / media-file-sync +job, since WebApiDB only needs auth plus reading the transaction-history +feed for now. Re-add pieces here (rather than importing GrampsWebSync +directly) so this addon has no runtime dependency on another addon being +installed. + +This file is a vendored copy: the canonical, standalone source is now the +gramps-api-client package (not yet published; local checkout at +~/gramps/gramps-api-client as of this writing), module +gramps_api_client/client.py, class Client -- the same class as +WebApiHandler below, just renamed. It was split out so the client could +be discoverable/pip-installable on its own, independent of the Gramps +addon ecosystem. Gramps addons are self-contained tarballs with no +mechanism to declare a pip dependency, so this copy has to stay vendored +here rather than importing that package directly; sync changes by hand in +both directions. + +Credentials +----------- +Two ways in: username+password (POST /token/, matches GrampsWebSync), or +a GRAMPS_WEB_API_KEY-shaped string: "*". + +The REFRESH_TOKEN half is a JWT *refresh* token obtained once via +POST /token/ with include_refresh (gramps-web-api's JWT_REFRESH_TOKEN_EXPIRES +is False by default, so it doesn't expire on its own). From then on, +POST /token/refresh/ trades it for fresh short-lived access tokens -- +no username/password re-entry, no server-side change needed. This is +*not* the same as a real scoped/revocable personal access token +(gramps-web-api has that machinery too, but today it's hardcoded to a +single "anniversaries_ics" scope and isn't wired into general request +auth) -- it's a shortcut that works today at the cost of not being +independently revocable. '*' is a safe delimiter here: neither a JWT +(base64url segments joined by '.') nor base64url output ever contains it. +""" + +from __future__ import annotations + +import base64 +import json +import logging +import os +import platform +import socket +import time +from tempfile import NamedTemporaryFile +from time import sleep +from typing import Any +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode, urlparse +from urllib.request import Request, urlopen + +LOG = logging.getLogger("grampswebapidb") + +#: Environment variable read by WebApiHandler.from_env(). +API_KEY_ENV_VAR = "GRAMPS_WEB_API_KEY" + +#: Seconds before a request that has produced nothing is abandoned. Without +#: this, ``urlopen`` waits forever and an unreachable-but-listening server +#: hangs Gramps with no way out. +TIMEOUT = 60 + +#: gramps-web-api rate-limits /token/ and /token/refresh/ to 1/second (no +#: Retry-After header is sent on 429); this is how long to back off before +#: the one retry attempt. Found by live testing: minting a key and then +#: immediately constructing another WebApiHandler in the same second +#: reliably 429s otherwise. +RATE_LIMIT_BACKOFF = 1.1 + +#: The exact message gramps_webapi/api/tasks.py's old_unchanged() check +#: raises as ValueError("Object has changed"), which POST /transactions/ +#: (without force=1) surfaces as HTTP 400 {"error": {"message": ...}}. +#: push_transaction() matches on this to tell a real conflict apart from +#: the endpoint's other 400s (malformed payload, missing Gramps ID, ...), +#: which are our own bugs, not conflicts, and should propagate as-is. +_CONFLICT_MESSAGE = "Object has changed" + + +class WebApiPushConflict(Exception): + """A push was rejected because the server-side object changed since + the local mirror's snapshot of it (see push_transaction()).""" + + +def _raise_for_push_conflict(exc: HTTPError) -> None: + """Given a 400 from POST /transactions/, raise WebApiPushConflict if + it's the server's old-data-mismatch check; otherwise re-raise ``exc`` + unchanged (a genuinely different 400, e.g. a malformed payload).""" + try: + body = json.loads(exc.read()) + message = body["error"]["message"] + except (ValueError, KeyError, TypeError): + raise exc + if message == _CONFLICT_MESSAGE: + raise WebApiPushConflict(message) from exc + raise exc + + +def create_macos_ssl_context(): + """Create an SSL context using macOS system certificates.""" + import ssl + import subprocess + + ctx = ssl.create_default_context() + macos_ca_certs = subprocess.run( + [ + "security", + "find-certificate", + "-a", + "-p", + "/System/Library/Keychains/SystemRootCertificates.keychain", + ], + stdout=subprocess.PIPE, + ).stdout + + with NamedTemporaryFile("w+b") as tmp_file: + tmp_file.write(macos_ca_certs) + ctx.load_verify_locations(tmp_file.name) + + return ctx + + +def decode_jwt_payload(jwt: str) -> dict[str, Any]: + """Decode and return the payload from a JWT.""" + payload_part = jwt.split(".")[1] + padding = len(payload_part) % 4 + if padding > 0: + payload_part += "=" * (4 - padding) + decoded_bytes = base64.urlsafe_b64decode(payload_part) + decoded_str = decoded_bytes.decode("utf-8") + return json.loads(decoded_str) + + +def parse_api_key(api_key: str) -> tuple[str, str]: + """Split a GRAMPS_WEB_API_KEY value into ``(refresh_token, url)``.""" + try: + token, encoded_url = api_key.split("*", 1) + except ValueError as exc: + raise ValueError( + "Malformed GRAMPS_WEB_API_KEY: expected '*'" + ) from exc + padding = "=" * (-len(encoded_url) % 4) + try: + url = base64.urlsafe_b64decode(encoded_url + padding).decode("utf-8") + except (ValueError, UnicodeDecodeError) as exc: + raise ValueError("Malformed GRAMPS_WEB_API_KEY: bad URL encoding") from exc + if not token or not url: + raise ValueError("Malformed GRAMPS_WEB_API_KEY: empty token or URL") + return token, url + + +def make_api_key(refresh_token: str, url: str) -> str: + """Build a GRAMPS_WEB_API_KEY value from a refresh token and URL.""" + encoded_url = base64.urlsafe_b64encode(url.encode("utf-8")).decode("ascii") + return f"{refresh_token}*{encoded_url.rstrip('=')}" + + +class WebApiHandler: + """Web API connection handler: token auth plus authenticated GET.""" + + def __init__( + self, + url: str, + username: str | None = None, + password: str | None = None, + refresh_token: str | None = None, + ) -> None: + """ + Initialize given a server URL, plus either a username+password or + a non-expiring refresh token (exactly one of the two is expected). + """ + self.url = url.rstrip("/") + self.username = username + self.password = password + self._refresh_token = refresh_token + self._access_token: str | None = None + self._ctx = ( + create_macos_ssl_context() if platform.system() == "Darwin" else None + ) + self._authenticate() + + @classmethod + def from_api_key(cls, api_key: str) -> "WebApiHandler": + """Build a handler from a GRAMPS_WEB_API_KEY-shaped string.""" + token, url = parse_api_key(api_key) + return cls(url, refresh_token=token) + + @classmethod + def from_env(cls, env_var: str = API_KEY_ENV_VAR) -> "WebApiHandler": + """ + Build a handler from an environment variable holding a + GRAMPS_WEB_API_KEY-shaped string. This is the SDK entry point: + ``client = WebApiHandler.from_env()``. + """ + api_key = os.environ.get(env_var) + if not api_key: + raise ValueError(f"{env_var} is not set") + return cls.from_api_key(api_key) + + @classmethod + def mint_api_key(cls, url: str, username: str, password: str) -> str: + """ + One-time username+password login that returns a GRAMPS_WEB_API_KEY + value for all future non-interactive use. This is the client-side + half of what a future "Generate SDK Key" UI button would automate + server-side; until that exists, this is how a key gets created at + all. + """ + handler = cls(url, username=username, password=password) + if not handler._refresh_token: + raise ValueError("Server did not return a refresh token") + return make_api_key(handler._refresh_token, handler.url) + + def _open(self, req: Request): + """Open ``req`` with this handler's SSL context and timeout.""" + return urlopen(req, context=self._ctx, timeout=TIMEOUT) + + @property + def access_token(self) -> str: + """Get the access token. Cached after first call unless refresh needed.""" + if not self._access_token: + self._authenticate() + remaining_time = self.get_access_token_remaining_time() + if remaining_time is not None and remaining_time < 60: + self._authenticate() + assert self._access_token # for type checker + return self._access_token + + def get_access_token_remaining_time(self) -> int | None: + """Get the remaining time of the access token in seconds.""" + if self._access_token is None: + return None + payload = decode_jwt_payload(self._access_token) + if "exp" not in payload: + return None + expires = payload["exp"] + now = time.time() + return int(expires - now) + + def _authenticate(self) -> None: + """Get a fresh access token, via whichever credential we hold.""" + if self._refresh_token: + self._refresh_access_token() + else: + self.fetch_token() + + def fetch_token(self, retry_on_rate_limit: bool = True) -> None: + """Fetch and store an access token via username+password.""" + LOG.debug("Fetching an access token from the server") + data = json.dumps({"username": self.username, "password": self.password}) + req = Request( + f"{self.url}/token/", + data=data.encode(), + headers={"Content-Type": "application/json", "User-Agent": "GrampsWebApiDb"}, + ) + try: + with self._open(req) as res: + res_json = json.load(res) + except HTTPError as exc: + if exc.code == 429 and retry_on_rate_limit: + sleep(RATE_LIMIT_BACKOFF) + return self.fetch_token(retry_on_rate_limit=False) + if "/api" not in self.url: + self.url = f"{self.url}/api" + return self.fetch_token(retry_on_rate_limit=retry_on_rate_limit) + raise + except (UnicodeDecodeError, json.JSONDecodeError): + if "/api" not in self.url: + self.url = f"{self.url}/api" + return self.fetch_token(retry_on_rate_limit=retry_on_rate_limit) + raise + self._access_token = res_json["access_token"] + # /token/ with username+password always includes a refresh token + # (TokenResource.post() calls get_tokens(..., include_refresh=True)). + if "refresh_token" in res_json: + self._refresh_token = res_json["refresh_token"] + + def _refresh_access_token(self, retry_on_rate_limit: bool = True) -> None: + """Trade the stored refresh token for a new access token.""" + LOG.debug("Refreshing access token from stored refresh token") + req = Request( + f"{self.url}/token/refresh/", + method="POST", + headers={ + "Authorization": f"Bearer {self._refresh_token}", + "User-Agent": "GrampsWebApiDb", + }, + ) + try: + with self._open(req) as res: + res_json = json.load(res) + except HTTPError as exc: + if exc.code == 429 and retry_on_rate_limit: + sleep(RATE_LIMIT_BACKOFF) + return self._refresh_access_token(retry_on_rate_limit=False) + if "/api" not in self.url: + self.url = f"{self.url}/api" + return self._refresh_access_token(retry_on_rate_limit=retry_on_rate_limit) + raise + except (UnicodeDecodeError, json.JSONDecodeError): + if "/api" not in self.url: + self.url = f"{self.url}/api" + return self._refresh_access_token(retry_on_rate_limit=retry_on_rate_limit) + raise + self._access_token = res_json["access_token"] + + def get_permissions(self) -> set[str]: + """Get the permissions of the current user.""" + return decode_jwt_payload(self.access_token).get("permissions", set()) + + @property + def hostname(self) -> str: + """Server hostname, e.g. "hadaly.duckdns.org" for a url of + "https://hadaly.duckdns.org/api".""" + return urlparse(self.url).hostname or self.url + + def get_current_username(self) -> str: + """Name of the user this handler is authenticated as. + + Set directly for a username+password login (mint_api_key()); the + refresh-token credential the normal from_env() path uses carries no + plaintext username (the access token's "sub" claim is a user id, + not a name -- see gramps-web-api's token.py), so it is resolved + once via GET /users/-/ (the "current user" alias) and cached here. + """ + if self.username is None: + data, _headers = self._get_json(f"{self.url}/users/-/") + self.username = data["name"] + return self.username + + def get_identity(self) -> str: + """"@" identifying the account+server this + handler authenticates as -- see grampswebapidb.py's + _check_identity(), which requires a Family Tree's own name to + match this before trusting its local mirror.""" + return f"{self.get_current_username()}@{self.hostname}" + + def _get_json(self, url: str, retry: bool = True) -> tuple[Any, dict]: + """GET ``url`` with the bearer token and return ``(body, headers)``.""" + req = Request( + url, + headers={ + "Authorization": f"Bearer {self.access_token}", + "User-Agent": "GrampsWebApiDb", + }, + ) + try: + with self._open(req) as res: + return json.load(res), dict(res.headers) + except HTTPError as exc: + if exc.code == 401 and retry: + # in case of 401, retry once with a new token + sleep(RATE_LIMIT_BACKOFF) # avoid immediately re-tripping the rate limit + self._authenticate() + return self._get_json(url, retry=False) + if exc.code == 429 and retry: + sleep(RATE_LIMIT_BACKOFF) + return self._get_json(url, retry=False) + raise + except (URLError, socket.timeout): + if retry: + sleep(1) + return self._get_json(url, retry=False) + raise + + def _get_binary(self, url: str, retry: bool = True) -> bytes: + """GET ``url`` with the bearer token and return the raw response + body, unlike _get_json() -- for endpoints that return a file + rather than a JSON document (see download_export()).""" + req = Request( + url, + headers={ + "Authorization": f"Bearer {self.access_token}", + "User-Agent": "GrampsWebApiDb", + }, + ) + try: + with self._open(req) as res: + return res.read() + except HTTPError as exc: + if exc.code == 401 and retry: + sleep(RATE_LIMIT_BACKOFF) + self._authenticate() + return self._get_binary(url, retry=False) + if exc.code == 429 and retry: + sleep(RATE_LIMIT_BACKOFF) + return self._get_binary(url, retry=False) + raise + except (URLError, socket.timeout): + if retry: + sleep(1) + return self._get_binary(url, retry=False) + raise + + def download_export(self, extension: str = "gramps") -> bytes: + """ + Download a full backup export of the tree from the server -- + by default a gzip-compressed Gramps XML file, the exact on-disk + shape Gramps' own ImportXml importer already reads (confirmed + against a live server: GET /exporters/gramps/file runs + synchronously and streams the file back, no task polling + needed). Used by grampswebapidb.py's WebApiDB._full_resync() to + rebuild the local mirror wholesale when the transaction-history + feed can't describe what changed -- see that method's own doc + comment on why. + """ + url = f"{self.url}/exporters/{extension}/file" + return self._get_binary(url) + + def get_transaction_history( + self, after: float = 0, page: int = 1, pagesize: int = 100 + ) -> tuple[list[dict[str, Any]], int]: + """ + Fetch one page of the server's transaction history committed + after ``after`` (a Unix timestamp), ascending by transaction id, + including the post-change raw object data. + + :returns: ``(transactions, total_count)``. ``total_count`` comes + from the ``X-Total-Count`` response header, so the caller can + tell whether more pages remain. + """ + params = { + "after": after, + "new": "1", + "sort": "id", + "page": page, + "pagesize": pagesize, + } + url = f"{self.url}/transactions/history/?{urlencode(params)}" + body, headers = self._get_json(url) + total_count = int(headers.get("X-Total-Count", len(body))) + return body, total_count + + def push_transaction( + self, + payload: list[dict[str, Any]], + retry: bool = True, + undo: bool = False, + ) -> None: + """ + POST a batch of local changes to /transactions/ (no force=1): the + server compares each item's "old" snapshot -- the local mirror's + state of the object *before* the local edit -- against its own + current data, and rejects the whole batch with HTTP 400 + ``{"error": {"message": "Object has changed"}}`` on any mismatch + (see gramps_webapi/api/tasks.py's process_transactions -> + old_unchanged()). That's a real, if coarse, optimistic-concurrency + check: it fires whenever the server-side object was edited (by + anyone) since the local mirror last synced, which is exactly what + a conflict is. Raised here as WebApiPushConflict so the caller + (grampswebapidb.py's transaction_commit) can tell "the server + rejected this because something changed underneath it" apart from + a network/auth failure. Actual merge resolution is still out of + scope -- the caller's response to a conflict is to resync from the + server, not to retry the push. + + ``undo=True`` sends the *same* payload a prior push_transaction() + call already sent, with ?undo=1: the server reverses it itself + (swaps old/new, add<->delete -- see + gramps_webapi/api/resources/util.py's reverse_transaction()) before + applying, so this is how grampswebapidb.py implements Undo without + having to compute the inverse payload locally. Redo is *not* a + variant of this -- it's just an ordinary push_transaction() call + with the original (forward) payload again. + """ + if not payload: + return + data = json.dumps(payload).encode() + url = f"{self.url}/transactions/" + if undo: + url += "?undo=1" + req = Request( + url, + data=data, + method="POST", + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {self.access_token}", + "User-Agent": "GrampsWebApiDb", + }, + ) + try: + with self._open(req) as res: + res.read() + except HTTPError as exc: + if exc.code == 401 and retry: + sleep(RATE_LIMIT_BACKOFF) + self._authenticate() + return self.push_transaction(payload, retry=False, undo=undo) + if exc.code == 429 and retry: + sleep(RATE_LIMIT_BACKOFF) + return self.push_transaction(payload, retry=False, undo=undo) + if exc.code == 400: + _raise_for_push_conflict(exc) + raise + except (URLError, socket.timeout): + if retry: + sleep(RATE_LIMIT_BACKOFF) + return self.push_transaction(payload, retry=False, undo=undo) + raise diff --git a/GrampsWebSync/adapters.py b/GrampsWebSync/adapters.py new file mode 100644 index 000000000..15fc009a6 --- /dev/null +++ b/GrampsWebSync/adapters.py @@ -0,0 +1,648 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2021-2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Production implementations of the :mod:`session` ports. + +Two task runners are provided rather than one. :class:`GLibTaskRunner` keeps a +step on the GTK main loop, which is mandatory for anything touching a Gramps +database: the sqlite backend binds a connection to its creating thread. +:class:`IoRunner` moves a step to a worker thread, which is where the network +calls belong -- they are the only part of a sync that can block indefinitely. + +:class:`ConfigCredentialStore` keeps one entry per ``(url, username)`` pair, so +each server carries its own sync baseline. +""" + +from __future__ import annotations + +import logging +import os +import threading +import time +from collections.abc import Callable +from dataclasses import dataclass +from typing import Any + +from gi.repository import GLib +from gramps.gen.config import config as configman +from gramps.gen.utils.file import media_path_full + +LOG = logging.getLogger("grampswebsync") + +#: Keys the pre-multi-server versions of the addon used. Still written, as a +#: mirror of the last-used entry, so that downgrading keeps working. +LEGACY_URL = "credentials.url" +LEGACY_USERNAME = "credentials.username" +LEGACY_TIMESTAMP = "credentials.timestamp" + +#: snapd interface granting access to ``org.freedesktop.secrets``. Declared by +#: the Gramps snap but manually connected, so it is off until the user says so. +SNAP_KEYRING_INTERFACE = "password-manager-service" + + +def normalize_url(url: str) -> str: + """Return ``url`` in the form used as a credential-store key. + + Normalizing here rather than in :class:`webapihandler.WebApiHandler` keeps a + stray trailing slash from looking like a different server, which would + otherwise cost the entry its sync baseline. + + :param url: The URL as typed or stored. + :returns: The URL without surrounding whitespace or trailing slashes. + """ + return url.strip().rstrip("/") + + +# ------------------------------------------------------------ +# +# Keyring +# +# ------------------------------------------------------------ +@dataclass(frozen=True) +class KeyringUnavailable: + """A keyring call that failed, for the view to report. + + :param detail: The underlying exception text, for logs and details views. + :param snap_command: The ``snap connect`` command that would fix it, when + running confined under snap; ``None`` elsewhere. + """ + + detail: str + snap_command: str | None = None + + +def snap_connect_command() -> str | None: + """Return the command connecting the keyring interface, under snap only. + + ``SNAP_INSTANCE_NAME`` rather than ``SNAP_NAME`` is what makes the command + correct under a parallel install. + + :returns: The command, or ``None`` when not running as a snap. + """ + if not os.environ.get("SNAP"): + return None + name = ( + os.environ.get("SNAP_INSTANCE_NAME") + or os.environ.get("SNAP_NAME") + or "gramps" + ) + return f"snap connect {name}:{SNAP_KEYRING_INTERFACE}" + + +class Keyring: + """The system keyring, degrading to unavailable instead of raising. + + Every call is guarded with a bare ``except Exception``. The failures seen in + practice do not derive from ``keyring.errors``: under snap confinement the + Secret Service backend raises ``jeepney.wrappers.DBusErrorResponse``, from a + transitive dependency, so catching the keyring package's own hierarchy is + not enough. + + After a failure the keyring is marked unavailable and no further calls are + attempted for the lifetime of this object. + """ + + def __init__(self) -> None: + self.unavailable: KeyringUnavailable | None = None + + def _module(self): + """Return the ``keyring`` module, or ``None`` if it cannot be used.""" + if self.unavailable is not None: + return None + try: + import keyring + except Exception as exc: # noqa: BLE001 -- absence is not an error here + LOG.warning("Keyring is not available: %s", exc) + self.unavailable = KeyringUnavailable(str(exc), snap_connect_command()) + return None + return keyring + + def _failed(self, action: str, exc: Exception) -> None: + """Record that ``action`` failed and stop using the keyring.""" + LOG.warning("Keyring %s failed: %s", action, exc) + self.unavailable = KeyringUnavailable(str(exc), snap_connect_command()) + + def get(self, service: str, username: str) -> str | None: + """Return the stored password, or ``None`` if it cannot be read.""" + keyring = self._module() + if keyring is None: + return None + try: + return keyring.get_password(service, username) + except Exception as exc: # noqa: BLE001 -- reported through `unavailable` + self._failed("read", exc) + return None + + def set(self, service: str, username: str, password: str) -> bool: + """Store ``password``. Returns whether it was actually stored.""" + keyring = self._module() + if keyring is None: + return False + try: + keyring.set_password(service, username, password) + except Exception as exc: # noqa: BLE001 -- reported through `unavailable` + self._failed("write", exc) + return False + return True + + def delete(self, service: str, username: str) -> bool: + """Remove a stored password. + + :returns: Whether the password is now gone. + """ + keyring = self._module() + if keyring is None: + return False + try: + keyring.delete_password(service, username) + except Exception as exc: # noqa: BLE001 -- absent entries raise too + # Deleting what was never there raises, and is harmless; a keyring + # that is actually broken still has the password afterwards. A read + # that fails proves nothing either way, so it counts as a failure. + before = self.unavailable + if self.get(service, username) is None and self.unavailable is before: + LOG.debug("Nothing to delete for %s: %s", username, exc) + return True + self._failed("delete", exc) + return False + return True + + +# ------------------------------------------------------------ +# +# Credential store +# +# ------------------------------------------------------------ +class ConfigCredentialStore: + """Server entries in the Gramps config file, passwords in the keyring. + + Each entry is keyed by ``(url, username)`` -- which identifies a tree, since + a Gramps Web account belongs to exactly one -- and carries its own + ``timestamp``, the baseline the diff uses. Per-entry baselines are why + switching servers no longer discards one. + """ + + def __init__( + self, + keyring: Keyring | None = None, + config: Any = None, + tree_id: str = "", + ) -> None: + """Initialize the store. + + :param keyring: Password storage. A real one is built if omitted. + :param config: An already-registered config manager. Tests pass one + pointed at a temporary directory so a run cannot write to the + user's own Gramps configuration. + :param tree_id: Identifier of the local family tree that is open, from + ``db.get_dbid()``. Entries record the tree they were synced from, + so that opening another one does not silently offer this tree's + server. Empty means the tree is unknown and no entry can match. + """ + self.keyring = keyring if keyring is not None else Keyring() + self.tree_id = tree_id or "" + self.config = ( + config if config is not None else configman.register_manager("webapisync") + ) + self.config.register(LEGACY_URL, "") + self.config.register(LEGACY_USERNAME, "") + self.config.register(LEGACY_TIMESTAMP, 0) + self.config.register("credentials.servers", []) + self.config.register("credentials.last_used", []) + self.config.load() + self._reconcile_legacy() + + # -------------------------------------------------------- + # Raw access + # -------------------------------------------------------- + def _servers(self) -> list[dict[str, Any]]: + """Return the stored entries, tolerating a corrupted config value. + + A value the config manager could not parse is stored as ``None`` rather + than falling back to the registered default, so the type has to be + checked rather than assumed. + """ + servers = self.config.get("credentials.servers") + if not isinstance(servers, list): + LOG.warning("Ignoring unreadable server list in config.") + return [] + return [entry for entry in servers if isinstance(entry, dict)] + + def _find( + self, servers: list[dict[str, Any]], url: str, username: str + ) -> dict[str, Any] | None: + """Return the entry for ``(url, username)``, or ``None``.""" + url = normalize_url(url) + for entry in servers: + if normalize_url(entry.get("url", "")) == url and ( + entry.get("username", "") == username + ): + return entry + return None + + def _last_used(self) -> tuple[str, str] | None: + """Return the ``(url, username)`` last synced, if any.""" + pair = self.config.get("credentials.last_used") + if isinstance(pair, list) and len(pair) == 2: + return str(pair[0]), str(pair[1]) + return None + + def _for_tree(self, servers: list[dict[str, Any]]) -> dict[str, Any] | None: + """Return the entry recorded against the tree that is open, if any. + + The last-used entry is preferred among several, which only arises if + one tree has been synced to more than one account. + """ + if not self.tree_id: + return None + matches = [ + entry for entry in servers if entry.get("tree_id") == self.tree_id + ] + if not matches: + return None + pair = self._last_used() + if pair is not None: + entry = self._find(matches, *pair) + if entry is not None: + return entry + return matches[0] + + def _current(self) -> dict[str, Any] | None: + """Return the entry to offer: this tree's, else the last used. + + Falling back to the last-used entry rather than to nothing keeps the + connect pane pre-filled for anyone whose entries predate tree ids. + """ + servers = self._servers() + entry = self._for_tree(servers) + if entry is not None: + return entry + pair = self._last_used() + if pair is not None: + entry = self._find(servers, *pair) + if entry is not None: + return entry + return servers[0] if len(servers) == 1 else None + + def _write(self, servers: list[dict[str, Any]]) -> None: + """Persist the entry list and the legacy mirror, then save.""" + self.config.set("credentials.servers", servers) + self._write_legacy_mirror(servers) + self.config.save() + + def _write_legacy_mirror(self, servers: list[dict[str, Any]]) -> None: + """Mirror the last-used entry into the pre-multi-server keys. + + An older version of the addon reads only those keys. Keeping them + current means a downgrade finds its credentials and its baseline where + it expects them, instead of resyncing from scratch. + """ + pair = self._last_used() + entry = self._find(servers, *pair) if pair is not None else None + if entry is None: + self.config.set(LEGACY_URL, "") + self.config.set(LEGACY_USERNAME, "") + self.config.set(LEGACY_TIMESTAMP, 0) + return + self.config.set(LEGACY_URL, entry.get("url", "")) + self.config.set(LEGACY_USERNAME, entry.get("username", "")) + self.config.set(LEGACY_TIMESTAMP, int(entry.get("timestamp", 0) or 0)) + + def _reconcile_legacy(self) -> None: + """Fold the legacy keys into the entry list. + + Covers both cases in one path: on first run after an upgrade there is no + matching entry and the legacy triple becomes one, and after a downgrade + and back the entry exists but an older version may have synced in the + meantime, so the later of the two baselines wins. + """ + legacy_url = normalize_url(self.config.get(LEGACY_URL) or "") + if not legacy_url: + return + legacy_username = self.config.get(LEGACY_USERNAME) or "" + legacy_timestamp = float(self.config.get(LEGACY_TIMESTAMP) or 0) + + servers = self._servers() + entry = self._find(servers, legacy_url, legacy_username) + if entry is None: + LOG.info("Migrating stored credentials to the server list.") + servers.append( + { + "url": legacy_url, + "username": legacy_username, + "timestamp": legacy_timestamp, + "remember_password": True, + } + ) + self.config.set("credentials.last_used", [legacy_url, legacy_username]) + elif legacy_timestamp > float(entry.get("timestamp", 0) or 0): + entry["timestamp"] = legacy_timestamp + else: + return + self.config.set("credentials.servers", servers) + self.config.save() + + # -------------------------------------------------------- + # CredentialStore protocol + # -------------------------------------------------------- + def get_url(self) -> str: + """Return the last-used server URL, for pre-filling the login page.""" + entry = self._current() + return entry.get("url", "") if entry else "" + + def get_username(self) -> str: + """Return the last-used user name.""" + entry = self._current() + return entry.get("username", "") if entry else "" + + def get_password(self) -> str | None: + """Return the last-used password, if one was stored and is readable.""" + entry = self._current() + if not entry or not entry.get("remember_password", True): + return None + url = entry.get("url", "") + username = entry.get("username", "") + if not url or not username: + return None + return self.keyring.get(url, username) + + def get_remember_password(self) -> bool: + """Whether the entry on offer is allowed to keep its password. + + :returns: The stored choice, defaulting to true for a server that has + not been seen before. + """ + entry = self._current() + if entry is None: + return True + return bool(entry.get("remember_password", True)) + + def get_timestamp(self, url: str, username: str) -> float: + """Return the sync baseline for one server. + + :param url: The server URL. + :param username: The account on that server. + :returns: The last successful sync time, or ``0`` if never synced. + """ + entry = self._find(self._servers(), url, username) + return float(entry.get("timestamp", 0) or 0) if entry else 0.0 + + def set_timestamp(self, url: str, username: str, timestamp: float) -> None: + """Record a successful sync against one server. + + This is also where an entry adopts the tree that is open. Merely + authenticating is deliberately not enough: connecting to the wrong + server by mistake would otherwise claim the tree before the user has + seen what the sync would do, and go on connecting there unprompted. + """ + servers = self._servers() + entry = self._find(servers, url, username) + if entry is None: + entry = { + "url": normalize_url(url), + "username": username, + "remember_password": True, + } + servers.append(entry) + entry["timestamp"] = timestamp + if self.tree_id: + self._claim_tree(servers, entry) + LOG.debug("Recording last successful sync at %s", timestamp) + self.config.set("credentials.last_used", [normalize_url(url), username]) + self._write(servers) + + def _claim_tree( + self, servers: list[dict[str, Any]], entry: dict[str, Any] + ) -> None: + """Record the open tree on ``entry``, and take it off every other one. + + Any entry that loses the claim also loses its baseline. + """ + # A tree has one current server, or the choice falls to list order. + # The old baseline no longer describes anything true, and keeping it + # would raise the comparison cutoff; too high a cutoff makes objects + # look deleted instead of added. + for other in servers: + if other is not entry and other.get("tree_id") == self.tree_id: + LOG.info( + "Tree moved away from %s; dropping its baseline.", + other.get("url"), + ) + del other["tree_id"] + other["timestamp"] = 0.0 + entry["tree_id"] = self.tree_id + + def save_credentials( + self, url: str, username: str, password: str, remember_password: bool = True + ) -> None: + """Persist one server entry, and its password if asked to. + + The entry itself is always stored: it carries the sync baseline, which + is not a credential, and discarding it would make every later run a cold + sync. ``remember_password`` governs the keyring only. + + :param url: The server URL, already sanitized by the caller. + :param username: The account name. + :param password: The password, stored only if ``remember_password``. + :param remember_password: Whether the password may go to the keyring. + """ + url = normalize_url(url) + servers = self._servers() + entry = self._find(servers, url, username) + if entry is None: + entry = {"url": url, "username": username, "timestamp": 0.0} + servers.append(entry) + entry["remember_password"] = remember_password + + if remember_password: + self.keyring.set(url, username, password) + else: + # Turning the setting off has to erase what is already stored, not + # merely stop writing, or it appears to do nothing. + self.keyring.delete(url, username) + + self.config.set("credentials.last_used", [url, username]) + self._write(servers) + + def forget(self, url: str, username: str) -> None: + """Remove one server entry entirely, keyring item included.""" + url = normalize_url(url) + servers = [ + entry + for entry in self._servers() + if not ( + normalize_url(entry.get("url", "")) == url + and entry.get("username", "") == username + ) + ] + self.keyring.delete(url, username) + if self._last_used() == (url, username): + self.config.set("credentials.last_used", []) + self._write(servers) + + def is_for_open_tree(self) -> bool: + """Whether the stored credentials belong to the tree that is open. + + Only then may a server be contacted without the user asking, since a + sync against the wrong tree proposes deleting both of them. + """ + return self._for_tree(self._servers()) is not None + + def is_from_another_tree(self) -> bool: + """Whether the credentials on offer were last synced from a different tree. + + Distinguished from merely having no tree recorded, which is what every + entry looks like until its first sync after upgrading, and which says + nothing either way. + """ + entry = self._current() + if entry is None or not self.tree_id: + return False + recorded = entry.get("tree_id", "") + return bool(recorded) and recorded != self.tree_id + + def keyring_error(self) -> KeyringUnavailable | None: + """Return the keyring failure to report, if one has occurred.""" + return self.keyring.unavailable + + +# ------------------------------------------------------------ +# +# Media +# +# ------------------------------------------------------------ +class GrampsMediaStore: + """Resolves media paths against the open Gramps database's media path. + + :param db: The local database whose media base path applies. + """ + + def __init__(self, db) -> None: + self.db = db + + def full_path(self, media: Any) -> str: + """Return the absolute path of ``media``'s file.""" + return media_path_full(self.db, media.get_path()) + + def exists(self, media: Any) -> bool: + """Whether ``media``'s file is present on disk.""" + return os.path.exists(self.full_path(media)) + + +# ------------------------------------------------------------ +# +# Task runners +# +# ------------------------------------------------------------ +def _post_to_main_loop(func: Callable[[], None]) -> None: + """Schedule ``func`` to run once on the GTK main loop.""" + + def once() -> bool: + func() + return False + + GLib.idle_add(once) + + +class GLibTaskRunner: + """Defers a task to the GTK main loop. + + For steps that touch a Gramps database. Those must not run on a worker + thread: the sqlite backend passes no ``check_same_thread=False`` and shares + one cursor, so a connection is usable only from the thread that created it. + They also drive Gramps progress through the GUI + :class:`gramps.gui.user.User`, which touches widgets. + + :func:`GLib.idle_add` keeps the work on the main loop while still letting the + caller return, so the view can paint the progress page first. + """ + + def run( + self, + func: Callable[[], Any], + on_success: Callable[[Any], None], + on_error: Callable[[BaseException], None], + ) -> None: + """Schedule ``func`` on the main loop and dispatch the outcome there.""" + + def once() -> bool: + try: + result = func() + except BaseException as exc: # noqa: BLE001 -- reported, not swallowed + on_error(exc) + else: + on_success(result) + return False # run once + + GLib.idle_add(once) + + def post(self, func: Callable[[], None]) -> None: + """Run ``func`` on the main loop.""" + _post_to_main_loop(func) + + +class IoRunner: + """Runs a task on a worker thread, dispatching the outcome on the main loop. + + For steps that only do network I/O. Those are where a sync spends most of + its wall-clock time and the only place it can block indefinitely, so moving + them off the main loop is what makes the window stay responsive and Cancel + actually work. Callbacks are marshalled back through + :func:`GLib.idle_add`, so listeners still run on the thread that owns GTK. + """ + + def run( + self, + func: Callable[[], Any], + on_success: Callable[[Any], None], + on_error: Callable[[BaseException], None], + ) -> None: + """Run ``func`` on a worker thread; call back on the main loop.""" + + def work() -> None: + try: + result = func() + except BaseException as exc: # noqa: BLE001 -- reported, not swallowed + # Handed straight on: `except ... as exc` unbinds the name when + # the block exits, and the callback runs later than that. + self._dispatch(on_error, exc) + else: + self._dispatch(on_success, result) + + threading.Thread(target=work, daemon=True, name="grampswebsync-io").start() + + @staticmethod + def _dispatch(callback: Callable[[Any], None], value: Any) -> None: + """Deliver ``value`` to ``callback`` on the main loop.""" + _post_to_main_loop(lambda: callback(value)) + + def post(self, func: Callable[[], None]) -> None: + """Run ``func`` on the main loop. + + Progress raised inside a network step arrives here, so that listeners + drawing widgets never run on the worker thread. + """ + _post_to_main_loop(func) + + +class SystemClock: + """The wall clock.""" + + def now(self) -> float: + """Return the current POSIX timestamp.""" + return time.time() diff --git a/GrampsWebSync/const.py b/GrampsWebSync/const.py index 83eb4c779..f51830f7e 100644 --- a/GrampsWebSync/const.py +++ b/GrampsWebSync/const.py @@ -65,4 +65,36 @@ MODE_BIDIRECTIONAL = 0 MODE_RESET_TO_LOCAL = 1 MODE_RESET_TO_REMOTE = 2 -MODE_MERGE = 3 \ No newline at end of file + +#: The modes offered, in the order they are presented. Display text lives in +#: the view, which is where translation happens. +SYNC_MODES = (MODE_BIDIRECTIONAL, MODE_RESET_TO_LOCAL, MODE_RESET_TO_REMOTE) + +#: Modes that discard one side's changes wholesale rather than propagating +#: them. The view warns about these instead of presenting them as equal-weight +#: peers of the default. +DESTRUCTIVE_MODES = frozenset({MODE_RESET_TO_LOCAL, MODE_RESET_TO_REMOTE}) + +#: The Gramps Web API major version this branch of the addon speaks. Each +#: Gramps release line pairs with one: Gramps 6.0, which this branch targets, +#: pairs with Gramps Web API 3. A server on a later major therefore needs a +#: later *Gramps*, not a later addon -- which is why the two version errors +#: give opposite advice. +#: +#: Stated rather than derived from the minimum below: "the only major we speak +#: is whichever the minimum happens to name" is an assumption that would hold +#: silently until it did not. +API_MAJOR = 3 + +#: Oldest Gramps Web API this addon will sync against, as ``(major, minor)``. +#: Checked at connect time so that an unsupported server says so, instead of +#: failing later in a way that reads as a problem with the user's account. +#: Raise the minor when a newer endpoint becomes a hard requirement; the major +#: is :data:`API_MAJOR` by definition. +MIN_API_VERSION = (API_MAJOR, 0) + +#: :data:`MIN_API_VERSION` as it is written out for the user. +MIN_API_VERSION_TEXT = f"{MIN_API_VERSION[0]}.{MIN_API_VERSION[1]}" + +#: :data:`API_MAJOR` as it is written out for the user. +API_MAJOR_TEXT = str(API_MAJOR) \ No newline at end of file diff --git a/GrampsWebSync/diffhandler.py b/GrampsWebSync/diffhandler.py index ca7e56adb..7e43c40da 100644 --- a/GrampsWebSync/diffhandler.py +++ b/GrampsWebSync/diffhandler.py @@ -48,7 +48,6 @@ MODE_BIDIRECTIONAL, MODE_RESET_TO_LOCAL, MODE_RESET_TO_REMOTE, - MODE_MERGE, OBJ_LST, Action, Actions, @@ -344,16 +343,6 @@ def changes_to_actions(changes, sync_mode: int) -> Actions: C_UPD_LOC: A_UPD_LOC, C_UPD_REM: A_UPD_LOC, } - elif sync_mode == MODE_MERGE: - change_to_action = { - C_UPD_BOTH: A_MRG_REM, - C_ADD_LOC: A_ADD_REM, - C_ADD_REM: A_ADD_LOC, - C_DEL_LOC: A_ADD_LOC, - C_DEL_REM: A_ADD_REM, - C_UPD_LOC: A_UPD_REM, - C_UPD_REM: A_UPD_LOC, - } else: raise ValueError(f"Invalid sync mode: {sync_mode}") actions = [] diff --git a/GrampsWebSync/grampswebsync.gpr.py b/GrampsWebSync/grampswebsync.gpr.py index e4f69720a..9d56e6ead 100644 --- a/GrampsWebSync/grampswebsync.gpr.py +++ b/GrampsWebSync/grampswebsync.gpr.py @@ -28,7 +28,7 @@ id="gramps_web_sync", name=_("Gramps Web Sync"), description=_("Synchronizes a local database with a Gramps Web instance."), - version = '1.3.11', + version = '1.5.1', gramps_target_version="6.0", status=STABLE, fname="grampswebsync.py", diff --git a/GrampsWebSync/grampswebsync.py b/GrampsWebSync/grampswebsync.py index 2bc090295..f7835fb16 100644 --- a/GrampsWebSync/grampswebsync.py +++ b/GrampsWebSync/grampswebsync.py @@ -1,6 +1,6 @@ # Gramps - a GTK+/GNOME based genealogy program # -# Copyright (C) 2021-2024 David Straub +# Copyright (C) 2021-2026 David Straub # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -17,51 +17,62 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -"""Gramps addon to synchronize with a Gramps Web server.""" +"""Gramps addon to synchronize with a Gramps Web server. + +Provides :class:`GrampsWebSyncTool`, a dialog presenting a +:class:`session.SyncSession` as four panes in a :class:`Gtk.Stack`. +:data:`PANE_FOR_STATE` maps each :class:`session.State` to the pane that +represents it and :func:`error_message` localizes a :class:`session.ErrorKind`. + +The synchronization itself lives in :mod:`session`, and everything the panes +render is prepared in :mod:`presentation`. +""" from __future__ import annotations import logging -import os -import threading -from collections.abc import Callable -from datetime import datetime -from typing import Any -from urllib.error import HTTPError, URLError -from urllib.parse import urlparse - -from const import ( - C_ADD_LOC, - C_ADD_REM, - C_DEL_LOC, - C_DEL_REM, - C_UPD_BOTH, - C_UPD_LOC, - C_UPD_REM, - MODE_BIDIRECTIONAL, - MODE_MERGE, - MODE_RESET_TO_LOCAL, - MODE_RESET_TO_REMOTE, - Actions, -) -from diffhandler import ( - WebApiSyncDiffHandler, - changes_to_actions, - has_local_actions, - has_remote_actions, +import time + +from adapters import ( + ConfigCredentialStore, + GLibTaskRunner, + GrampsMediaStore, + IoRunner, + SystemClock, ) -from gi.repository import GLib, Gtk -from gramps.gen.config import config as configman +from const import MODE_BIDIRECTIONAL, SYNC_MODES +from diffhandler import changes_to_actions +from gi.repository import GLib, Gtk, Pango from gramps.gen.const import GRAMPS_LOCALE as glocale -from gramps.gen.db import DbTxn -from gramps.gen.db.utils import import_as_dict -from gramps.gen.errors import HandleError -from gramps.gen.lib import Tag -from gramps.gen.utils.file import media_path_full from gramps.gui.dialog import QuestionDialog2 +from gramps.gui.display import display_url from gramps.gui.managedwindow import ManagedWindow from gramps.gui.plug.tool import BatchTool, ToolOptions -from webapihandler import WebApiHandler, transaction_to_json +from presentation import ( + ReviewModel, + build_review, + deletion_warning, + destination_label, + error_message, + context_lines, + format_last_synced, + insecure_warning, + is_insecure, + keyring_message, + media_label, + missing_both_notice, + mode_description, + mode_label, + outcome_summary, + sanitize_url, + state_label, + status_message, + transfer_message, + verb_label, + version_line, +) +from session import WORKING_STATES, State, SyncSession +from webapihandler import WebApiHandler assert glocale is not None # for type checker try: @@ -69,694 +80,594 @@ except ValueError: _trans = glocale.translation _ = _trans.gettext -ngettext = _trans.ngettext LOG = logging.getLogger("grampswebsync") - -def get_password(service: str, username: str) -> str | None: - """If keyring is installed, return the user's password or None.""" - LOG.debug("Retrieving password for user %s", username) - try: - import keyring - except ImportError: - LOG.warning("Keyring is not installed, cannot retrieve password.") - return None - return keyring.get_password(service, username) - - -def set_password(service: str, username: str, password: str) -> None: - """If keyring is installed, store the user's password.""" - try: - import keyring - except ImportError: - return None - LOG.debug("Storing password for user %s", username) - keyring.set_password(service, username, password) +#: Where the manual lives. Deliberately the English page rather than a +#: localized one: the site declares its translations as ``hreflang`` alternates +#: and renders a language switcher from them, so it always offers every +#: language it currently has. Anything this addon hardcoded would be a guess +#: that goes stale, and a wrong guess is a 404. +#: +#: ``gramps.gui.display.display_help`` must not be used to open it -- it +#: appends the UI locale to whatever it is given, full URLs included. +DOCUMENTATION_URL = "https://www.grampsweb.org/administration/sync/" + +#: Names of the stack's children. +PANE_CONNECT = "connect" +PANE_WORKING = "working" +PANE_REVIEW = "review" +PANE_RESULT = "result" + +#: The one place that knows how flow states correspond to panes. Four panes +#: replace the eight assistant pages this tool used to have; the states that +#: differ only in what work is running share the working pane, and both +#: terminal states share the result pane. +PANE_FOR_STATE: dict[State, str] = { + State.CONNECT: PANE_CONNECT, + State.CONNECTING: PANE_WORKING, + State.COMPARING: PANE_WORKING, + State.REVIEW: PANE_REVIEW, + State.APPLYING: PANE_WORKING, + State.TRANSFERRING: PANE_WORKING, + State.DONE: PANE_RESULT, + State.FAILED: PANE_RESULT, +} + +#: States in which the tool has begun writing. The server may not be swapped +#: underneath a run that has already committed something, and abandoning one +#: would leave no record of how far it got. +WRITING_STATES = (State.APPLYING, State.TRANSFERRING) + +#: Response ids for the buttons the dialog adds itself. +RESPONSE_CONNECT = 1 +RESPONSE_APPLY = 2 +RESPONSE_RETRY = 3 + +#: Names of the phase markers, as children of each row's marker stack. +MARK_DONE = "done" +MARK_ACTIVE = "active" +MARK_PENDING = "pending" + +#: Themed icon standing for a finished phase. A symbolic icon follows the +#: theme's foreground colour and its dark variant, which neither a text glyph +#: nor a bundled SVG would; the theme is also already a Gramps dependency. +DONE_ICON = "emblem-ok-symbolic" + +#: How often the working pane is refreshed. A progress bar in pulse mode only +#: moves when it is told to, so this is the pulse rate as well as the rate the +#: elapsed clock is checked at. +TICK_INTERVAL_MS = 120 + +#: Column of the review tree holding the object name: free text of unbounded +#: length, and the only one that gives way when the window is too narrow. +NAME_COLUMN = 1 + +#: How narrow the name column may get before the tree scrolls instead. +NAME_MIN_WIDTH = 180 + +#: How wide the progress bar is allowed to get. Left to fill the pane it +#: stretches the width of the window and reads as a divider rather than a bar. +PROGRESS_WIDTH = 380 + + +def _dim(text: str) -> str: + """Return markup rendering ``text`` as secondary.""" + return f"{GLib.markup_escape_text(text)}" + + +def _label(text: str = "", *, xalign: float = 0.0, wrap: bool = True) -> Gtk.Label: + """Return a left-aligned label with sensible wrapping defaults.""" + label = Gtk.Label(label=text) + label.set_xalign(xalign) + if wrap: + label.set_line_wrap(True) + label.set_max_width_chars(60) + return label +# ------------------------------------------------------------ +# +# The tool +# +# ------------------------------------------------------------ class GrampsWebSyncTool(BatchTool, ManagedWindow): - """Main class for the Gramps Web Sync tool.""" + """Dialog presenting a :class:`session.SyncSession` to the user.""" def __init__(self, dbstate, user, options_class, name, *args, **kwargs) -> None: - """Initialize GUI.""" + """Build the dialog and the session behind it.""" LOG.debug("Initializing Gramps Web Sync addon.") BatchTool.__init__(self, dbstate, user, options_class, name) + if self.fail: + # The user declined the undo-history warning; honour that instead + # of opening the dialog anyway. + LOG.debug("Undo history warning declined; not opening the tool.") + return ManagedWindow.__init__(self, user.uistate, [], self.__class__) self.dbstate = dbstate - self.callback = self.uistate.pulse_progressbar - - self.config = configman.register_manager("webapisync") - self.config.register("credentials.url", "") - self.config.register("credentials.username", "") - self.config.register("credentials.timestamp", 0) - self.config.load() - - self.assistant = Gtk.Assistant() - self.set_window(self.assistant, None, _("Gramps Web Sync")) - self.setup_configs("interface.webapisync", 780, 600) - - self.assistant.connect("close", self.do_close) - self.assistant.connect("cancel", self.do_close) - self.assistant.connect("apply", self.apply) - self.assistant.connect("prepare", self.prepare) - - self.intro = IntroductionPage(self.assistant) - self.add_page(self.intro, Gtk.AssistantPageType.INTRO, _("Introduction")) - - self.url = self.config.get("credentials.url") - self.username = self.config.get("credentials.username") - self.password = self.get_password() - self.loginpage = LoginPage( - self.assistant, - url=self.url, - username=self.username, - password=self.password, + self._timer_id: int | None = None + self._phase_started = time.monotonic() + + self.credentials = ConfigCredentialStore(tree_id=dbstate.db.get_dbid()) + self.session = SyncSession( + db=dbstate.db, + user=self._user, + backend_factory=self._make_backend, + credentials=self.credentials, + media=GrampsMediaStore(dbstate.db), + runner=GLibTaskRunner(), + io_runner=IoRunner(), + clock=SystemClock(), + listener=self, ) - self.add_page(self.loginpage, Gtk.AssistantPageType.CONTENT, _("Login")) - self.diff_progress_page = DiffProgressPage(self.assistant) - self.add_page( - self.diff_progress_page, - Gtk.AssistantPageType.PROGRESS, - _("Progress Information"), - ) - - self.confirmation = ConfirmationPage(self.assistant) - self.add_page( - self.confirmation, Gtk.AssistantPageType.CONFIRM, _("Final confirmation") + self._build_window() + self.show() + self._start() + + # -------------------------------------------------------- + # Window construction + # -------------------------------------------------------- + def _build_window(self) -> None: + """Assemble the dialog: context strip, pane stack, footer, buttons.""" + self.dialog = Gtk.Dialog() + self.set_window(self.dialog, None, _("Gramps Web Sync")) + # A new key deliberately. The old one holds whatever size suited the + # eight-page assistant, and a window that shares almost nothing with it + # should not inherit a geometry chosen for the other one. + self.setup_configs("interface.grampswebsync", 820, 640) + + content = self.dialog.get_content_area() + content.set_spacing(0) + + self.context = ContextStrip(on_change_server=self._on_change_server) + content.pack_start(self.context, False, False, 0) + content.pack_start( + Gtk.Separator(orientation=Gtk.Orientation.HORIZONTAL), False, False, 0 ) - self.sync_progress_page = SyncProgressPage(self.assistant) - self.add_page( - self.sync_progress_page, - Gtk.AssistantPageType.PROGRESS, - _("Summary"), + self.stack = Gtk.Stack() + self.stack.set_transition_type(Gtk.StackTransitionType.NONE) + self.stack.set_border_width(12) + self.connect_pane = ConnectPane( + on_changed=self._on_connect_fields_changed, on_forget=self._on_forget ) - - self.file_confirmation = FileConfirmationPage(self.assistant) - self.add_page( - self.file_confirmation, - Gtk.AssistantPageType.CONFIRM, - _("Media Files"), + self.working_pane = WorkingPane() + self.review_pane = ReviewPane() + self.result_pane = ResultPane() + self.stack.add_named(self.connect_pane, PANE_CONNECT) + self.stack.add_named(self.working_pane, PANE_WORKING) + self.stack.add_named(self.review_pane, PANE_REVIEW) + self.stack.add_named(self.result_pane, PANE_RESULT) + content.pack_start(self.stack, True, True, 0) + + self.version_label = _label(wrap=False) + self.version_label.set_margin_start(12) + self.version_label.set_margin_bottom(6) + self.version_label.set_markup(_dim(version_line(None))) + content.pack_start(self.version_label, False, False, 0) + + self._build_buttons() + self.dialog.connect("response", self._on_response) + + def _build_buttons(self) -> None: + """Add every button once; visibility follows the state.""" + self.button_cancel = self.dialog.add_button( + _("_Cancel"), Gtk.ResponseType.CANCEL ) - - self.file_progress_page = FileProgressPage(self.assistant) - self.add_page( - self.file_progress_page, - Gtk.AssistantPageType.PROGRESS, - _("Progress Information"), + self.button_close = self.dialog.add_button(_("_Close"), Gtk.ResponseType.CLOSE) + self.button_retry = self.dialog.add_button(_("_Try again"), RESPONSE_RETRY) + self.button_connect = self.dialog.add_button(_("C_onnect"), RESPONSE_CONNECT) + self.button_apply = self.dialog.add_button(_("_Apply"), RESPONSE_APPLY) + for button in self._buttons(): + button.set_can_default(True) + button.set_no_show_all(True) + button.hide() + + def _buttons(self) -> tuple[Gtk.Button, ...]: + """Return every button in the action area.""" + return ( + self.button_cancel, + self.button_close, + self.button_retry, + self.button_connect, + self.button_apply, ) - self.conclusion = ConclusionPage(self.assistant) - self.add_page(self.conclusion, Gtk.AssistantPageType.SUMMARY, _("Summary")) - - self.show() - self.assistant.set_forward_page_func(self.forward_page, None) - - self._api: WebApiHandler | None = None - - self.db1 = dbstate.db - self.db2 = None - self._closing = False - self._download_timestamp = 0 - self._changes: Actions | None = None - self._sync: WebApiSyncDiffHandler | None = None - self.files_missing_local: list[tuple[str, str]] = [] - self.files_missing_remote: list[tuple[str, str]] = [] - self.uploaded: dict[str, bool] = {} - self.downloaded: dict[str, bool] = {} - - @property - def api(self) -> WebApiHandler: - if self._api is None: - raise ValueError("No WebApiHandler found") # shouldn't happen! - return self._api - - @property - def sync(self) -> WebApiSyncDiffHandler: - if self._sync is None: - raise ValueError("No WebApiSyncDiffHandler found") # shouldn't happen! - return self._sync - - @property - def changes(self) -> Actions: - if self._changes is None: - raise ValueError("No change actions found") # shouldn't happen! - return self._changes - def build_menu_names(self, obj): # type: ignore """Override :class:`.ManagedWindow` method.""" return (_("Gramps Web Sync"), None) - def do_close(self, assistant): - """Close the assistant.""" - LOG.debug("Closing Gramps Web Sync addon.") - self._closing = True - if self.db2 is not None: - LOG.debug("Closing in-memory remote database.") - self.db2.close() - self.db2 = None - # Clear the diff handler which holds references to both db1 and db2 - self._sync = None - self._changes = None - position = self.window.get_position() # crock - self.assistant.hide() - self.window.move(position[0], position[1]) - self.close() - - def forward_page(self, page, data): - """Specify the next page to be displayed.""" - LOG.debug(f"Moving to next page from page {page}.") - if self.conclusion.error: - LOG.debug("Skipping to last page due to error.") - return 7 - if page == 2 and self._changes is not None and len(self.changes) == 0: - LOG.debug("Skipping to media sync as databases are in sync.") - return 4 - if page == 5 and self.conclusion.unchanged: - LOG.debug("Skipping to last page as media files are in sync.") - return 7 - return page + 1 - - def add_page(self, page, page_type, title=""): - """Add a page to the assistant.""" - page.show_all() - self.assistant.append_page(page) - self.assistant.set_page_title(page, title) - self.assistant.set_page_type(page, page_type) - - def handle_done_syncing_dbs(self): - """Handle the completion of syncing the databases.""" - self.save_timestamp() - self.sync_progress_page.handle_done_syncing_dbs() - self.files_missing_local = self.get_missing_files_local() - self.assistant.next_page() - - def prepare(self, assistant, page): - """Run page preparation code.""" - page.update_complete() - if page == self.diff_progress_page: - # Clear any previous login error when starting fresh - self.loginpage.clear_error() - - # Try to connect and authenticate - self.save_credentials() - url, username, password = self.get_credentials() - if not self.test_connection(url, username, password): - # Connection failed, go back to login page - self.assistant.set_current_page(1) # Login page index - return None - - if "ViewPrivate" not in self.api.get_permissions(): - self.loginpage.show_error( - _( - "Your user does not have sufficient server permissions to use sync." - ) - ) - self.assistant.set_current_page(1) # Go back to login page - return None - - self.diff_progress_page.label.set_text(_("Fetching remote data...")) - t = threading.Thread(target=self.async_compare_dbs) - t.start() - elif page == self.confirmation: - self.confirmation.prepare(self.changes) - elif page == self.sync_progress_page: - self.assistant.commit() # just erases the visited page history - actions = changes_to_actions(self.changes, self.confirmation.sync_mode) - self.sync_progress_page.prepare(actions) - if len(actions) == 0: - self.handle_done_syncing_dbs() - else: - try: - self.commit_all_actions(actions) - except Exception as e: - self.handle_error( - _("Unexpected error while applying changes.") + f" {e}" - ) - - # now, get missing media files - elif page == self.file_confirmation: - if self.files_missing_local: - LOG.debug( - "The following media files are missing on the local side: %s", - ", ".join([gramps_id for gramps_id, _ in self.files_missing_local]), - ) - else: - LOG.debug("No files missing locally.") - self.files_missing_remote = self.get_missing_files_remote() - if self.files_missing_remote: - LOG.debug( - "The following media files are missing on the remote side: %s", - ", ".join( - [gramps_id for gramps_id, _ in self.files_missing_remote] - ), - ) - else: - LOG.debug("No files missing remotely.") - if not self.files_missing_local and not self.files_missing_remote: - self.handle_files_unchanged() - else: - self.file_confirmation.prepare( - self.files_missing_local, self.files_missing_remote - ) - elif page == self.file_progress_page: - self.file_progress_page.prepare( - self.files_missing_local, self.files_missing_remote - ) - t = threading.Thread(target=self.async_transfer_media) - t.start() - elif page == self.conclusion: - if self.conclusion.error: - pass - elif self.conclusion.unchanged: - text = _("Media files are in sync.") - self.conclusion.label.set_text(text) - LOG.info("Media files are in sync.") - else: - text = "" - if self.downloaded: - ok = sum([b for gid, b in self.downloaded.items()]) - nok = sum([not b for gid, b in self.downloaded.items()]) - if ok: - text += _("Successfully downloaded %s media files.") % ok - text += " " - if nok: - text += _("Encountered %s errors during download.") % nok - text += " " - if self.uploaded: - ok = sum([b for gid, b in self.uploaded.items()]) - nok = sum([not b for gid, b in self.uploaded.items()]) - if ok: - text += _("Successfully uploaded %s media files.") % ok - text += " " - if nok: - text += _("Encountered %s errors during upload.") % nok - self.conclusion.label.set_text(text) - - self.conclusion.set_complete() - - def test_connection(self, url: str, username: str, password: str) -> bool: - """Test the connection and authentication. Return True if successful.""" - try: - # Try to create API handler - self._api = WebApiHandler(url, username, password, None) - - # Test the connection by making a simple API call - self.api.get_permissions() - return True - - except HTTPError as exc: - if exc.code == 401: - self.loginpage.show_error( - _("Authentication failed. Please check your username and password.") - ) - elif exc.code == 403: - self.loginpage.show_error( - _("Access forbidden. Please check username and password.") - ) - elif exc.code == 404: - self.loginpage.show_error( - _("GrampsWeb service not found. Please check the URL.") - ) - elif exc.code == 429: - self.loginpage.show_error( - _("Too many requests, please try again in a few seconds.") - ) - elif exc.code == 503: - self.loginpage.show_error(_("GrampsWeb tree is disabled.")) - else: - self.loginpage.show_error( - _("Server error %s. Please check your connection.") % exc.code - ) - return False - except URLError: - self.loginpage.show_error( - _( - "Connection failed. Please check the URL and your internet connection." - ) - ) - return False - except ValueError: - self.loginpage.show_error( - _("Invalid server response. Please check the URL.") + def _make_backend(self, url: str, username: str, password: str) -> WebApiHandler: + """Build the real Web API handler. Injected into the session.""" + return WebApiHandler(url, username, password, None) + + # -------------------------------------------------------- + # Lifecycle + # -------------------------------------------------------- + def _start(self) -> None: + """Show the stored server, and connect if it belongs to the open tree. + + Where the tree cannot be established the credentials are still offered, + but the user presses Connect. + """ + # Connecting unprompted is only safe for the tree the entry was synced + # from: against another one the two share nothing, every object falls + # the wrong side of the baseline, and a bidirectional run proposes + # deleting both trees. + url = self.credentials.get_url() + username = self.credentials.get_username() + password = self.credentials.get_password() or "" + self.connect_pane.set_credentials(url, username, password) + self.connect_pane.set_notices(self._connect_notices()) + self.connect_pane.set_can_forget(bool(url)) + self.connect_pane.set_remember_password( + self.credentials.get_remember_password() + ) + self._refresh_password_storage() + if url and username and password and self.credentials.is_for_open_tree(): + self._submit() + else: + self._render(self.session.state) + + def clean_up(self) -> None: + """Release the session and stop the clock when the window goes away.""" + self._stop_timer() + self.session.cancel() + super().clean_up() + + def _on_response(self, _dialog, response: int) -> None: + """Act on a button in the action area.""" + if response == Gtk.ResponseType.DELETE_EVENT: + # ManagedWindow already closed us from its own delete-event + # handler; acting again would only warn about a double close. + return + if response == RESPONSE_CONNECT: + self._submit() + elif response == RESPONSE_APPLY: + self.session.confirm( + self.review_pane.sync_mode, self.review_pane.transfer_media ) - return False - except Exception as e: - self.loginpage.show_error(_("Unexpected error: %s") % str(e)) - return False - - def handle_files_unchanged(self): - self.conclusion.unchanged = True - self.assistant.next_page() - - def apply(self, assistant): - """Apply the changes.""" - page_number = assistant.get_current_page() - page = assistant.get_nth_page(page_number) - if page == self.confirmation: - pass - elif page == self.file_confirmation: - pass - - def download_files(self): - """Download media files missing locally.""" - if not self.files_missing_local: + elif response == RESPONSE_RETRY: + self.session.retry() + else: + LOG.debug("Closing Gramps Web Sync addon (response=%s).", response) + self.close() + + def _on_change_server(self, _button) -> None: + """Stop whatever is running and return to the connect pane. + + Reachable while connecting and comparing, so that switching servers + does not first cost a whole download of the one being left. + """ + self.session.abandon() + + def _on_forget(self, _button) -> None: + """Remove the stored server, after asking. + + The password can be retyped; the baseline cannot be recovered, and + losing it turns the next run into a full comparison. That is worth a + confirmation. + """ + url = self.connect_pane.url.get_text() + username = self.connect_pane.username.get_text() + question = QuestionDialog2( + _("Forget this server?"), + _( + "The address, user name and password stored for this server " + "will be removed, along with the record of when this family " + "tree last synchronized with it. The next synchronization " + "will compare the two trees from scratch." + ), + _("Forget"), + _("Cancel"), + parent=self.window, + ) + if not question.run(): return - res = {} - for gramps_id, handle in self.files_missing_local: - LOG.debug("Downloading file %s", gramps_id) - self.downloaded[gramps_id] = self._download_file(handle) - self._update_file_progress() - return res - - def _update_file_progress(self): - """Update the file progress bars.""" - self.file_progress_page.update_progress( - self.files_missing_local, - self.files_missing_remote, - self.downloaded, - self.uploaded, + LOG.info("Forgetting the stored server.") + self.credentials.forget(url, username) + self.connect_pane.set_credentials("", "", "") + # Also drops the session's copy of the connection, which the context + # strip and the version footer are rendered from. + self.session.abandon() + + def _submit(self) -> None: + """Hand what the connect pane holds to the session.""" + url = sanitize_url(self.connect_pane.url.get_text()) + self.connect_pane.set_url(url) + self.session.submit_credentials( + url, + self.connect_pane.username.get_text(), + self.connect_pane.password.get_text(), + self.connect_pane.remember_password, ) - # force updating progress bar + + # -------------------------------------------------------- + # SessionListener + # -------------------------------------------------------- + def on_state_changed(self, state: State) -> None: + """Follow the session to the pane representing ``state``.""" + self._render(state) + + def on_progress(self, kind: str, fraction: float) -> None: + """Render a progress update from the session.""" + detail = transfer_message(kind) + if detail: + self.working_pane.set_detail(detail) + self.working_pane.set_fraction(fraction) + self._pump() + + def on_status(self, stage: str) -> None: + """Render a status update from the session.""" + self.working_pane.set_detail(status_message(stage)) + self._pump() + + @staticmethod + def _pump() -> None: + """Redraw now. + + The steps that touch a database run on the main loop, so without this + the pane would not repaint until the whole step finished. + """ while Gtk.events_pending(): Gtk.main_iteration() - def _download_file(self, handle): - """Download a single media file.""" - try: - obj = self.db1.get_media_from_handle(handle) - except HandleError: - self.handle_error(_("Error accessing media object.")) - return False - path = media_path_full(self.db1, obj.get_path()) - try: - return self.api.download_media_file(handle=handle, path=path) - except Exception as e: - LOG.warning(f"Failed to download media file {obj.gramps_id}: {e}") - return False - - def upload_files(self): - """Upload media files missing remotely.""" - if not self.files_missing_remote: - return - res = {} - for gramps_id, handle in self.files_missing_remote: - LOG.debug("Uploading file %s", gramps_id) - self.uploaded[gramps_id] = self._upload_file(handle) - self._update_file_progress() - return res - - def _upload_file(self, handle): - """Upload a single media file.""" - try: - obj = self.db1.get_media_from_handle(handle) - except HandleError: - self.handle_error(_("Error accessing media object.")) - return - path = media_path_full(self.db1, obj.get_path()) - return self.api.upload_media_file(handle=handle, path=path) - - def get_password(self): - """Get a stored password.""" - url = self.config.get("credentials.url") - username = self.config.get("credentials.username") - if not url or not username: - return None - return get_password(url, username) - - def handle_error(self, message): - """Handle an error message during sync.""" - LOG.warning(message) - self.conclusion.error = True - self.assistant.next_page() - self.conclusion.label.set_text(message) - self.conclusion.set_complete() - - def handle_unchanged(self): - """Return a message if nothing has changed.""" - self.save_timestamp() - self.assistant.next_page() - - def async_compare_dbs(self): - """Download the remote data and import it to an in-memory database.""" - # store timestamp just before downloading the XML - self._download_timestamp = datetime.now().timestamp() - GLib.idle_add(self.get_diff_actions) - - def get_diff_actions(self) -> None: - """Download the remote data, import it and compare it to local.""" - if self._closing: - return - LOG.info("Downloading Gramps XML file.") - path = self.handle_server_errors(self.api.download_xml) - if path is None: - return - LOG.debug(f"The file name of the downloaded file is: {path}") - LOG.debug("Importing Gramps XML file.") - db2 = import_as_dict(str(path), self._user) - if db2 is None: - self.handle_error(_("Failed importing downloaded XML file.")) - return - LOG.debug("Successfully imported Gramps XML file.") - path.unlink() # delete temporary file - self.db2 = db2 - self.diff_progress_page.label.set_text(_("Comparing local and remote data...")) - LOG.info("Comparing local and remote data...") - timestamp = self.config.get("credentials.timestamp") or None - from datetime import datetime - - LOG.debug( - "Loading last sync timestamp from config: %s (%s)", - timestamp, - ( - datetime.fromtimestamp(timestamp).strftime("%Y-%m-%d %H:%M:%S %Z") - if timestamp - else "None" - ), + # -------------------------------------------------------- + # Rendering + # -------------------------------------------------------- + def _render(self, state: State) -> None: + """Show the pane for ``state`` and bring the rest of the shell in line.""" + self._prepare_pane(state) + self.stack.set_visible_child_name(PANE_FOR_STATE[state]) + self._update_buttons(state) + self._update_context(state) + self._update_timer(state) + self.version_label.set_markup(_dim(version_line(self.session.api_version))) + + def _prepare_pane(self, state: State) -> None: + """Fill the pane for ``state`` with what the session now holds.""" + if state is State.CONNECT: + error = self.session.login_error + if error is None: + self.connect_pane.clear_error() + else: + self.connect_pane.show_error( + error_message(error.kind, error.detail) + ) + self.connect_pane.set_notices(self._connect_notices()) + self.connect_pane.set_can_forget(bool(self.credentials.get_url())) + self._refresh_password_storage() + elif state in WORKING_STATES: + self.working_pane.set_state(state) + elif state is State.REVIEW: + self.review_pane.prepare(self.session) + else: + self.result_pane.prepare(self.session) + # A keyring write happens after a successful connect, so its + # failure can land once the user has left the connect pane. + problem = self.credentials.keyring_error() + if problem is not None: + self.result_pane.show_notice(keyring_message(problem)) + + def _update_buttons(self, state: State) -> None: + """Show the buttons that make sense in ``state``, and pick the default.""" + terminal = state in (State.DONE, State.FAILED) + self.button_cancel.set_visible(not terminal) + self.button_close.set_visible(terminal) + self.button_retry.set_visible( + state is State.FAILED and self.session.can_retry ) - self._sync = WebApiSyncDiffHandler( - self.db1, self.db2, user=self._user, last_synced=timestamp + self.button_connect.set_visible(state is State.CONNECT) + self.button_apply.set_visible(state is State.REVIEW) + if state is State.CONNECT: + self._on_connect_fields_changed() + self.button_connect.grab_default() + elif state is State.REVIEW: + self.button_apply.grab_default() + + def _update_context(self, state: State) -> None: + """Say which tree is being synced, and when it last was.""" + url = self.session.url or self.credentials.get_url() + username = self.session.username or self.credentials.get_username() + title, subtitle = context_lines( + url, + username, + self.session.tree_name, + format_last_synced(self.credentials.get_timestamp(url, username)), ) - self._changes = self.sync.get_changes() - self.diff_progress_page.label.set_text("") - self.diff_progress_page.set_complete() - if len(self.changes) == 0: - LOG.info("Databases are in sync.") - self.handle_unchanged() + self.context.update(title, subtitle) + self.context.set_busy(state in WRITING_STATES) + + def _connect_notices(self) -> list[str]: + """Return everything worth saying on the connect pane, in order. + + An unusable keyring is reported rather than swallowed: without it the + password field is simply empty every run and nothing explains why. + """ + notices = [] + if self.credentials.is_from_another_tree(): + notices.append( + _( + "These credentials were last used with a different family " + "tree. Check the server before continuing." + ) + ) + problem = self.credentials.keyring_error() + if problem is not None: + notices.append(keyring_message(problem)) + return notices + + def _refresh_password_storage(self) -> None: + """Offer to remember the password only where that can be honoured.""" + problem = self.credentials.keyring_error() + self.connect_pane.set_keyring_available( + problem is None, "" if problem is None else keyring_message(problem) + ) + + def _on_connect_fields_changed(self) -> None: + """Keep the Connect button in step with the entries.""" + self.button_connect.set_sensitive(self.connect_pane.complete) + + # -------------------------------------------------------- + # Elapsed time + # -------------------------------------------------------- + def _update_timer(self, state: State) -> None: + """Run a one-second clock for as long as a phase is running. + + A progress bar that can only pulse -- which is what the server's task + endpoint gives us for most of an apply -- reads as a hang without one. + """ + if state in WORKING_STATES: + self._phase_started = time.monotonic() + self.working_pane.set_elapsed(0) + if self._timer_id is None: + self._timer_id = GLib.timeout_add(TICK_INTERVAL_MS, self._tick) else: - self.assistant.next_page() + self._stop_timer() - def async_transfer_media(self): - """Upload/download media files.""" - GLib.idle_add(self._async_transfer_media) + def _stop_timer(self) -> None: + """Stop the elapsed-time clock, if it is running.""" + if self._timer_id is not None: + GLib.source_remove(self._timer_id) + self._timer_id = None - def _async_transfer_media(self): - """Upload/download media files.""" - if self._closing: - return - self.handle_server_errors(self.download_files) - if self.conclusion.error: - return - self.handle_server_errors(self.upload_files) - if self.conclusion.error: - return - self.file_progress_page.set_complete() - self.assistant.next_page() - - def handle_server_errors(self, callback: Callable, *args) -> None: - """Handle server errors while executing a function.""" - try: - return callback(*args) - except HTTPError as exc: - if exc.code == 401: - self.handle_error(_("Server authorization error.")) - elif exc.code == 403: - self.handle_error( - _("Server authorization error: insufficient permissions.") - ) - elif exc.code == 404: - self.handle_error(_("Error: URL not found.")) - elif exc.code == 409: - self.handle_error( - _( - "Unable to synchronize changes to server: objects have been modified." - ) - ) - else: - self.handle_error(_("Error %s while connecting to server.") % exc.code) - return None - except URLError: - self.handle_error(_("URL error while connecting to server.")) - return None - except ValueError as exc: - self.handle_error( - f"{_('Unable to synchronize changes to server.')} ({exc})" - ) - return None - - def save_credentials(self) -> None: - """Save the login credentials.""" - url = self.loginpage.url.get_text() - url = self.sanitize_url(url) - if url is None: - self.handle_error("No URL provided") - return - username = self.loginpage.username.get_text() - password = self.loginpage.password.get_text() - if url != self.config.get("credentials.url"): - # if URL changed, clear last sync timestamp - self.config.set("credentials.timestamp", 0) - self.config.set("credentials.url", url) - self.config.set("credentials.username", username) - set_password(url, username, password) - self.config.save() - - def sanitize_url(self, url: str) -> str | None: - """Warn if http and prepend https if missing.""" - parsed_url = urlparse(url) - if parsed_url.scheme == "": - # if no httpX given, prepend https! - url = f"https://{url}" - elif parsed_url.scheme == "http": - question = QuestionDialog2( - _("Continue without transport encryption?"), - _( - "You have specified a URL with http scheme. " - "If you continue, your password will be sent " - "in clear text over the network. " - "Use only for local testing!" - ), - _("Continue with HTTP"), - _("Use HTTPS"), - parent=self.window, - ) - if not question.run(): - return url.replace("http", "https") - return url + def _tick(self) -> bool: + """Animate the progress bar and update the elapsed-time readout.""" + self.working_pane.pulse() + self.working_pane.set_elapsed(int(time.monotonic() - self._phase_started)) + return True - def get_credentials(self): - """Get a tuple of URL, username, and password.""" - return ( - self.config.get("credentials.url"), - self.config.get("credentials.username"), - self.loginpage.password.get_text(), - ) - def commit_all_actions(self, actions: Actions) -> None: - """Commit all changes to the databases.""" - LOG.info("Committing all changes to the databases.") - msg = "Apply Gramps Web Sync changes" - with DbTxn(msg, self.sync.db1) as trans1: - with DbTxn(msg, self.sync.db2) as trans2: - if has_local_actions(actions): - LOG.debug("Committing changes to local database.") - else: - LOG.debug("No changes to apply to local database.") - self.sync.commit_actions(actions, trans1, trans2) - self.sync_progress_page.handle_local_sync_complete(actions) - # force the sync for all modes: the server-side "object has changed" - # check compares against the XML-round-tripped object, which often - # differs from the live server object due to serialization artifacts, - # causing false-positive 409 conflicts even when no real concurrent - # edit has occurred. - force = True - lang = self.api.get_lang() - payload = transaction_to_json(trans2, lang) - GLib.idle_add(self.async_commit_actions_to_remote, payload, force) - - def async_commit_actions_to_remote( - self, payload: dict[str, "Any"], force: bool - ) -> None: - """Commit all changes to the remote database.""" - GLib.idle_add(self._async_commit_actions_to_remote, payload, force) - - def _async_commit_actions_to_remote( - self, payload: dict[str, "Any"], force: bool - ) -> None: - """Upload/download media files.""" - if self._closing: - return - LOG.debug("Committing changes to remote database.") - self.handle_server_errors( - self.api.commit, - payload, - force, - self.sync_progress_page.update_api_progress, - ) - if self.conclusion.error: - return - self.handle_done_syncing_dbs() - - def save_timestamp(self): - """Save last sync timestamp.""" - # self.config.set("credentials.timestamp", self._download_timestamp) - timestamp = datetime.now().timestamp() - LOG.debug( - "Saving current time stamp (%s) as last successful sync time (%s).", - timestamp, - datetime.fromtimestamp(timestamp).strftime("%Y-%m-%d %H:%M:%S %Z"), - ) - self.config.set("credentials.timestamp", timestamp) - self.config.save() +# ------------------------------------------------------------ +# +# Shell widgets +# +# ------------------------------------------------------------ +class ContextStrip(Gtk.Box): + """Names the tree being synced, where from, and when it last was. - def get_missing_files_local(self) -> list[tuple[str, str]]: - """Get a list of media files missing locally.""" - return [ - (media.gramps_id, media.handle) - for media in self.db1.iter_media() - if not os.path.exists(media_path_full(self.db1, media.get_path())) - ] + The sync baseline governs the entire conflict classification, and until now + nothing in the interface revealed it, or even which server was about to be + written to -- let alone which tree on it. - def get_missing_files_remote(self): - """Get a list of media files missing remotely.""" - missing_files = self.handle_server_errors(self.api.get_missing_files) or [] - return [(media["gramps_id"], media["handle"]) for media in missing_files] + :param on_change_server: Called when the user wants a different server. + """ + def __init__(self, on_change_server) -> None: + Gtk.Box.__init__(self, orientation=Gtk.Orientation.HORIZONTAL, spacing=12) + self.set_border_width(12) -class Page(Gtk.Box): - """Page base class.""" + text = Gtk.Box(orientation=Gtk.Orientation.VERTICAL) + self.server_label = _label(wrap=False) + self.server_label.set_ellipsize(Pango.EllipsizeMode.MIDDLE) + self.synced_label = _label(wrap=False) + text.pack_start(self.server_label, False, False, 0) + text.pack_start(self.synced_label, False, False, 0) + self.pack_start(text, True, True, 0) - def __init__(self, assistant: Gtk.Assistant): - """Initialize self.""" - Gtk.Box.__init__(self, orientation=Gtk.Orientation.VERTICAL) - self.assistant = assistant - self._complete = False + self.change_button = Gtk.Button(label=_("Change server…")) + self.change_button.set_valign(Gtk.Align.CENTER) + self.change_button.connect("clicked", on_change_server) + self.pack_start(self.change_button, False, False, 0) - def set_complete(self): - """Set as complete.""" - self._complete = True - self.update_complete() + def update(self, title: str, subtitle: str) -> None: + """Show what is being synced, and where from. - @property - def complete(self): - return self._complete + :param title: The remote tree's name once known, else the account. + :param subtitle: The line below it. + """ + self.server_label.set_markup(f"{GLib.markup_escape_text(title)}") + self.synced_label.set_markup(_dim(subtitle)) - def update_complete(self): - """Set the current page's complete status.""" - page_number = self.assistant.get_current_page() - current_page = self.assistant.get_nth_page(page_number) - if current_page is not None: - self.assistant.set_page_complete(current_page, self.complete) + def set_busy(self, busy: bool) -> None: + """Block a server switch while a sync is running.""" + self.change_button.set_sensitive(not busy) -class IntroductionPage(Page): - """A page containing introductory text.""" +class ConnectPane(Gtk.Box): + """Server URL, user name and password, plus what used to be the intro page. - def __init__(self, assistant): - super().__init__(assistant) - label = Gtk.Label(label=self.__get_intro_text()) - label.set_line_wrap(True) - label.set_use_markup(True) - label.set_max_width_chars(60) + :param on_changed: Called whenever an entry changes. + :param on_forget: Called when the user asks to remove the stored server. + """ - self.pack_start(label, False, False, 0) - self._complete = True + def __init__(self, on_changed, on_forget) -> None: + Gtk.Box.__init__(self, orientation=Gtk.Orientation.VERTICAL, spacing=12) + self._on_changed = on_changed - def __get_intro_text(self): + grid = Gtk.Grid() + grid.set_row_spacing(6) + grid.set_column_spacing(12) + self.pack_start(grid, False, False, 0) + + self.url = self._entry(grid, _("Server URL:"), 0) + self.url.set_input_purpose(Gtk.InputPurpose.URL) + self.username = self._entry(grid, _("Username:"), 1) + self.password = self._entry(grid, _("Password:"), 2) + self.password.set_visibility(False) + self.password.set_input_purpose(Gtk.InputPurpose.PASSWORD) + + self.remember_check = Gtk.CheckButton(label=_("Remember password")) + self.remember_check.set_active(True) + self.pack_start(self.remember_check, False, False, 0) + + self.scheme_label = self._hidden_label() + self.pack_start(self.scheme_label, False, False, 0) + self.error_label = self._hidden_label() + self.error_label.get_style_context().add_class("error") + self.pack_start(self.error_label, False, False, 0) + self.notice_label = self._hidden_label() + self.pack_start(self.notice_label, False, False, 0) + + self.forget_button = Gtk.Button(label=_("Forget this server")) + self.forget_button.set_halign(Gtk.Align.START) + self.forget_button.connect("clicked", on_forget) + self.pack_start(self.forget_button, False, False, 0) + + self.pack_start(self._about(), False, False, 0) + + def _entry(self, grid: Gtk.Grid, text: str, row: int) -> Gtk.Entry: + """Add one labelled entry to ``grid`` and return it.""" + grid.attach(_label(text, wrap=False), 0, row, 1, 1) + entry = Gtk.Entry() + entry.set_hexpand(True) + entry.set_activates_default(True) + entry.connect("changed", self._on_entry_changed) + grid.attach(entry, 1, row, 1, 1) + return entry + + @staticmethod + def _hidden_label() -> Gtk.Label: + """Return a label that stays hidden until it has something to say.""" + label = _label() + label.set_no_show_all(True) + label.hide() + return label + + def _about(self) -> Gtk.Expander: + """Return the collapsed introduction, with a link to the wiki page. + + Four paragraphs of preconditions matter on first use and are friction + on run fifty, so they fold away instead of occupying a page of their own. + """ + expander = Gtk.Expander(label=_("About this tool")) + box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6) + box.set_margin_top(6) + box.pack_start(_label(self._intro_text()), False, False, 0) + help_button = Gtk.Button(label=_("Open the online manual")) + help_button.set_halign(Gtk.Align.START) + help_button.connect( + "clicked", lambda *_a: display_url(DOCUMENTATION_URL) + ) + box.pack_start(help_button, False, False, 0) + expander.add(box) + return expander + + @staticmethod + def _intro_text() -> str: """Return the introductory text.""" return _( "This tool allows to synchronize the currently opened " @@ -773,397 +684,474 @@ def __get_intro_text(self): "Tool instead." ) + # -------------------------------------------------------- + # Contents + # -------------------------------------------------------- + def set_credentials(self, url: str, username: str, password: str) -> None: + """Pre-fill the entries from the credential store.""" + self.url.set_text(url or "") + self.username.set_text(username or "") + self.password.set_text(password or "") -class LoginPage(Page): - """A page to provide server credentials.""" - - def __init__(self, assistant, url, username, password): - super().__init__(assistant) - self.set_spacing(12) - - grid = Gtk.Grid() - grid.set_row_spacing(6) - grid.set_column_spacing(6) - self.add(grid) - - label = Gtk.Label(label=_("Server URL: ")) - grid.attach(label, 0, 0, 1, 1) - self.url = Gtk.Entry() - if url: + @property + def remember_password(self) -> bool: + """Whether the user is willing to have the password stored.""" + return self.remember_check.get_active() + + def set_remember_password(self, remember: bool) -> None: + """Reflect the choice stored for this server.""" + self.remember_check.set_active(remember) + + def set_keyring_available(self, available: bool, reason: str = "") -> None: + """Withdraw the offer when there is nowhere to store a password. + + Left checked but inert, the box would promise something that cannot + happen; the reason goes on the tooltip so the state is explicable. + """ + self.remember_check.set_sensitive(available) + if not available: + self.remember_check.set_active(False) + self.remember_check.set_tooltip_text(reason or None) + + def set_can_forget(self, can_forget: bool) -> None: + """Offer removal only when there is something stored to remove.""" + self.forget_button.set_sensitive(can_forget) + + def set_url(self, url: str) -> None: + """Show the URL that will actually be used. + + The scheme is completed before connecting, and leaving the entry + showing something else would misreport what the tool just did. + """ + if self.url.get_text() != url: self.url.set_text(url) - self.url.set_hexpand(True) - self.url.set_input_purpose(Gtk.InputPurpose.URL) - grid.attach(self.url, 1, 0, 1, 1) - - label = Gtk.Label(label=_("Username: ")) - grid.attach(label, 0, 1, 1, 1) - self.username = Gtk.Entry() - if username: - self.username.set_text(username) - self.username.set_hexpand(True) - grid.attach(self.username, 1, 1, 1, 1) - - label = Gtk.Label(label=_("Password: ")) - grid.attach(label, 0, 2, 1, 1) - self.password = Gtk.Entry() - if password: - self.password.set_text(password) - self.password.set_hexpand(True) - self.password.set_visibility(False) - self.password.set_input_purpose(Gtk.InputPurpose.PASSWORD) - grid.attach(self.password, 1, 2, 1, 1) - # Error message label - initially hidden - self.error_label = Gtk.Label() - self.error_label.set_line_wrap(True) - self.error_label.set_max_width_chars(60) - self.error_label.get_style_context().add_class("error") - self.error_label.set_no_show_all(True) # Don't show when show_all() is called - self.error_label.hide() - grid.attach(self.error_label, 0, 3, 2, 1) + @property + def complete(self) -> bool: + """Whether all three fields have something in them.""" + return bool( + self.url.get_text() + and self.username.get_text() + and self.password.get_text() + ) - # Connect entry change events - self.url.connect("changed", self.on_entry_changed) - self.username.connect("changed", self.on_entry_changed) - self.password.connect("changed", self.on_entry_changed) + def show_error(self, message: str) -> None: + """Display an error. - def show_error(self, message: str): - """Display an error message on the login page.""" - self.error_label.set_markup(f"Error: {message}") + The message is escaped: it can carry server or exception text, and an + unescaped ``&`` or ``<`` would break the markup or swallow the message. + """ + label = GLib.markup_escape_text(_("Error:")) + self.error_label.set_markup( + f"{label} {GLib.markup_escape_text(message)}" + ) self.error_label.show() - self.update_complete() - def clear_error(self): + def clear_error(self) -> None: """Clear any displayed error message.""" self.error_label.hide() - self.update_complete() - @property - def complete(self): - url = self.url.get_text() - username = self.username.get_text() - password = self.password.get_text() - if url and username and password: - return True - return False - - def on_entry_changed(self, widget): - """Handle changes to entry fields.""" - # Clear error when user starts typing - if self.error_label.get_visible(): - self.clear_error() - self.update_complete() - - -class DiffProgressPage(Page): - """A progress page.""" - - def __init__(self, assistant): - super().__init__(assistant) - label = Gtk.Label(label="") - label.set_line_wrap(True) - label.set_use_markup(True) - label.set_max_width_chars(60) - self.label = label - self.pack_start(self.label, False, False, 0) + def set_notices(self, messages: list[str]) -> None: + """Display non-fatal notices, or hide the label when there are none.""" + if not messages: + self.notice_label.hide() + return + self.notice_label.set_markup( + "\n".join( + f"{GLib.markup_escape_text(message)}" for message in messages + ) + ) + self.notice_label.show() + + def _on_entry_changed(self, _widget) -> None: + """Clear a stale error, warn about http, and report the change on.""" + self.clear_error() + if is_insecure(self.url.get_text()): + self.scheme_label.set_markup( + f"{GLib.markup_escape_text(_('Warning:'))} " + f"{GLib.markup_escape_text(insecure_warning())}" + ) + self.scheme_label.show() + else: + self.scheme_label.hide() + self._on_changed() + + +class PhaseRow: + """One line of the working pane's phase list: a marker and a name. + + The marker is a themed symbolic icon or a spinner rather than a character, + so it neither depends on the interface font carrying the glyph nor stays + the wrong colour in a dark theme. + + :param state: The phase this row stands for. + """ + + def __init__(self, state: State) -> None: + self.marker = Gtk.Stack() + self.marker.set_valign(Gtk.Align.CENTER) + self.spinner = Gtk.Spinner() + self.marker.add_named(Gtk.Box(), MARK_PENDING) + self.marker.add_named(self.spinner, MARK_ACTIVE) + self.marker.add_named(self._done_icon(), MARK_DONE) + self.label = _label(state_label(state), wrap=False) + # A stack has no visible child until its children are shown, and would + # then ignore being told which one to display. + self.marker.show_all() + self.set_marker(MARK_PENDING) + + @staticmethod + def _done_icon() -> Gtk.Widget: + """Return the finished marker, falling back if the theme lacks it.""" + if Gtk.IconTheme.get_default().has_icon(DONE_ICON): + return Gtk.Image.new_from_icon_name(DONE_ICON, Gtk.IconSize.MENU) + return Gtk.Label(label="\u2713") + + def set_marker(self, marker: str) -> None: + """Show this row as pending, running or finished. + + :param marker: One of the ``MARK_*`` names. + """ + self.marker.set_visible_child_name(marker) + if marker == MARK_ACTIVE: + self.spinner.start() + else: + self.spinner.stop() + name = GLib.markup_escape_text(self.label.get_text()) + self.label.set_markup( + f"{name}" if marker == MARK_ACTIVE else name + ) -class ConfirmationPage(Page): - """Page showing the differences before applying them.""" +class WorkingPane(Gtk.Box): + """A phase list, a detail line, a progress bar and an elapsed-time clock. + + One pane covers every long-running stage. The phase list is what tells the + user where in the run they are; the clock is what distinguishes slow from + stuck while the progress bar can only pulse. + """ + + def __init__(self) -> None: + Gtk.Box.__init__(self, orientation=Gtk.Orientation.VERTICAL, spacing=24) + self.set_valign(Gtk.Align.CENTER) + # Filling the width would space the phase list and the bar far apart. + self.set_halign(Gtk.Align.CENTER) + + #: Whether the running phase has reported a real fraction. Until it + #: does the bar is pulsed; afterwards pulsing would fight the value. + self._measurable = False + self._elapsed_text = "" + + self._rows: dict[State, PhaseRow] = {} + phases = Gtk.Grid() + phases.set_row_spacing(8) + phases.set_column_spacing(12) + for index, state in enumerate(WORKING_STATES): + row = PhaseRow(state) + phases.attach(row.marker, 0, index, 1, 1) + phases.attach(row.label, 1, index, 1, 1) + self._rows[state] = row + self.pack_start(phases, False, False, 0) + + # Grouped, so the detail line reads as belonging to the bar below it + # rather than to the phase list above. + progress = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6) + self.detail_label = _label(xalign=0.5) + progress.pack_start(self.detail_label, False, False, 0) + self.progressbar = Gtk.ProgressBar() + self.progressbar.set_size_request(PROGRESS_WIDTH, -1) + progress.pack_start(self.progressbar, False, False, 0) + self.elapsed_label = _label(xalign=0.5, wrap=False) + progress.pack_start(self.elapsed_label, False, False, 0) + self.pack_start(progress, False, False, 0) + + def set_state(self, state: State) -> None: + """Mark ``state`` as the phase now running. + + Phases the run skipped are marked done rather than left pending: they + are behind the user either way, and a list that never fills in reads as + something having gone wrong. + """ + current = WORKING_STATES.index(state) + for index, phase in enumerate(WORKING_STATES): + if index < current: + mark = MARK_DONE + elif index == current: + mark = MARK_ACTIVE + else: + mark = MARK_PENDING + self._rows[phase].set_marker(mark) + self._measurable = False + self.progressbar.set_fraction(0) + + def set_detail(self, text: str) -> None: + """Show what the running phase is doing right now.""" + self.detail_label.set_text(text) + + def set_fraction(self, fraction: float) -> None: + """Advance the progress bar, or pulse it when nothing is measurable.""" + if fraction >= 0: + self._measurable = True + self.progressbar.set_fraction(min(fraction, 1.0)) + else: + self.progressbar.pulse() - def __init__(self, assistant): - super().__init__(assistant) - self.sync_mode = MODE_BIDIRECTIONAL - self.store = Gtk.TreeStore(str, str) + def pulse(self) -> None: + """Advance the bar while the running phase reports no fraction. - # tree view - self.tree_view = Gtk.TreeView(model=self.store) + Downloading the remote tree reports none at all, so without a caller + on a timer the bar moved one step and then stood still for the longest + phase of the run. + """ + if not self._measurable: + self.progressbar.pulse() - for i, col in enumerate(["ID", "Content"]): - renderer = Gtk.CellRendererText() - column = Gtk.TreeViewColumn(col, renderer, text=i) - self.tree_view.append_column(column) + def set_elapsed(self, seconds: int) -> None: + """Show how long the current phase has been running.""" + text = "" if seconds < 3 else f"{seconds // 60}:{seconds % 60:02d}" + if text != self._elapsed_text: + self._elapsed_text = text + self.elapsed_label.set_markup(_dim(text)) - # scrolled window - scrolled_window = Gtk.ScrolledWindow() - scrolled_window.add(self.tree_view) - self.sync_label = Gtk.Label() - self.sync_label.set_text("Sync mode") +class ReviewPane(Gtk.Box): + """The one screen that matters: what will happen, and to which tree. - # Box for radio buttons - self.radio_box = Gtk.Box(orientation=Gtk.Orientation.HORIZONTAL, spacing=10) + The list is regenerated from the selected mode, so it shows the *actions* + that mode produces rather than the raw differences. Media transfers are a + checkbox here rather than a second confirmation page of their own. - # Radio buttons - option_name = _("Bidirectional Synchronization") - self.radio_button1 = Gtk.RadioButton.new_with_label_from_widget( - None, option_name - ) - self.radio_button1.connect( - "toggled", self.on_radio_button_toggled, MODE_BIDIRECTIONAL - ) - self.radio_box.pack_start(self.radio_button1, False, False, 0) + """ - option_name = _("Reset remote to local") - self.radio_button2 = Gtk.RadioButton.new_from_widget(self.radio_button1) - self.radio_button2.set_label(option_name) - self.radio_button2.connect( - "toggled", self.on_radio_button_toggled, MODE_RESET_TO_LOCAL + def __init__(self) -> None: + Gtk.Box.__init__(self, orientation=Gtk.Orientation.VERTICAL, spacing=12) + self._session: SyncSession | None = None + self.sync_mode = MODE_BIDIRECTIONAL + + self.mode_box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=3) + self.mode_box.set_no_show_all(True) + self.mode_box.pack_start( + _label(_("Sync mode:"), wrap=False), False, False, 0 ) - self.radio_box.pack_start(self.radio_button2, False, False, 0) + first = None + for mode in SYNC_MODES: + if first is None: + button = Gtk.RadioButton.new_with_label_from_widget( + None, mode_label(mode) + ) + first = button + else: + button = Gtk.RadioButton.new_with_label_from_widget( + first, mode_label(mode) + ) + button.connect("toggled", self._on_mode_toggled, mode) + self.mode_box.pack_start(button, False, False, 0) + self.description_label = _label() + self.description_label.set_margin_start(24) + self.mode_box.pack_start(self.description_label, False, False, 0) + self.pack_start(self.mode_box, False, False, 0) + + self.warning_label = _label() + self.warning_label.set_no_show_all(True) + self.warning_label.hide() + self.pack_start(self.warning_label, False, False, 0) + + self.store = Gtk.TreeStore(str, str, str) + self.tree_view = Gtk.TreeView(model=self.store) + for index, title in enumerate( + (_("Change"), _("Name"), _("ID")) + ): + renderer = Gtk.CellRendererText() + column = Gtk.TreeViewColumn(title, renderer, text=index) + column.set_resizable(True) + if index == NAME_COLUMN: + # An ellipsizing renderer reports a minimum width of almost + # nothing, so only the column that should absorb the shortfall + # gets one. Setting it on all three let every column collapse + # to its minimum, and cut off the group headings, which sit in + # column 0 and are longer than any leaf value in it. + renderer.set_property("ellipsize", Pango.EllipsizeMode.END) + column.set_expand(True) + column.set_min_width(NAME_MIN_WIDTH) + self.tree_view.append_column(column) + self.scrolled = Gtk.ScrolledWindow() + self.scrolled.set_policy(Gtk.PolicyType.AUTOMATIC, Gtk.PolicyType.AUTOMATIC) + self.scrolled.set_shadow_type(Gtk.ShadowType.IN) + self.scrolled.set_no_show_all(True) + self.scrolled.add(self.tree_view) + self.pack_start(self.scrolled, True, True, 0) + + self.media_check = Gtk.CheckButton(label="") + self.media_check.set_active(True) + self.media_check.set_no_show_all(True) + self.pack_start(self.media_check, False, False, 0) + + self.media_notice = _label() + self.media_notice.set_margin_start(24) + self.media_notice.set_no_show_all(True) + self.media_notice.hide() + self.pack_start(self.media_notice, False, False, 0) - option_name = _("Reset local to remote") - self.radio_button3 = Gtk.RadioButton.new_from_widget(self.radio_button1) - self.radio_button3.set_label(option_name) - self.radio_button3.connect( - "toggled", self.on_radio_button_toggled, MODE_RESET_TO_REMOTE + @property + def transfer_media(self) -> bool: + """Whether the user wants the missing media files moved.""" + return self.media_check.get_active() + + def prepare(self, session: SyncSession) -> None: + """Render what ``session`` found, for the mode now selected.""" + self._session = session + # A run with nothing but media to move has no mode to choose and no + # object list to show, so neither is offered. + self.mode_box.set_visible(bool(session.changes)) + self.scrolled.set_visible(bool(session.changes)) + self.tree_view.show_all() + self._render_media(session) + self._render_changes() + + def _on_mode_toggled(self, button: Gtk.RadioButton, mode: int) -> None: + """Re-render the list, because the mode reinterprets every row.""" + if not button.get_active(): + return + self.sync_mode = mode + self._render_changes() + + def _render_changes(self) -> None: + """Rebuild the tree from the actions the selected mode produces.""" + self.store.clear() + self.description_label.set_text(mode_description(self.sync_mode)) + session = self._session + if session is None or not session.changes: + self.warning_label.hide() + return + actions = changes_to_actions(session.changes, self.sync_mode) + model = build_review(actions, session.db1, session.db2) + self._fill(model) + self._render_warning(model) + + def _fill(self, model: ReviewModel) -> None: + """Write ``model`` into the tree store and expand the headings.""" + for destination in model.destinations: + parent = self.store.append( + None, [destination_label(destination.where, destination.count), "", ""] + ) + for group in destination.groups: + node = self.store.append( + parent, [verb_label(group.verb, group.count), "", ""] + ) + for row in group.rows: + self.store.append(node, [row.type_label, row.name, row.gramps_id]) + self.tree_view.expand_all() + + def _render_warning(self, model: ReviewModel) -> None: + """Show what will be deleted, if anything will be.""" + text = deletion_warning(model) + if not text: + self.warning_label.hide() + return + self.warning_label.set_markup( + f"{GLib.markup_escape_text(_('Warning:'))} " + f"{GLib.markup_escape_text(text)}" ) - self.radio_box.pack_start(self.radio_button3, False, False, 0) - - option_name = _("Merge") - self.radio_button4 = Gtk.RadioButton.new_from_widget(self.radio_button1) - self.radio_button4.set_label(option_name) - self.radio_button4.connect("toggled", self.on_radio_button_toggled, MODE_MERGE) - self.radio_box.pack_start(self.radio_button4, False, False, 0) - - # Box to hold the label and radio buttons - self.label_radio_box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=5) - self.label_radio_box.pack_start(self.sync_label, False, False, 0) - self.label_radio_box.pack_start(self.radio_box, False, False, 0) - - self.outer_box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=10) - self.outer_box.pack_start(scrolled_window, True, True, 0) - self.outer_box.pack_start(self.label_radio_box, False, False, 10) - - self.pack_start(self.outer_box, True, True, 0) - - def on_radio_button_toggled(self, button, name): - """Callback for radio buttons setting sync mode.""" - if button.get_active(): - self.sync_mode = int(name) - - def prepare(self, changes: Actions): - """Convert the changes list to a tree store.""" - change_labels = { - _("Local changes"): { - _("Added"): C_ADD_LOC, - _("Deleted"): C_DEL_LOC, - _("Modified"): C_UPD_LOC, - }, - _("Remote changes"): { - _("Added"): C_ADD_REM, - _("Deleted"): C_DEL_REM, - _("Modified"): C_UPD_REM, - }, - _("Simultaneous changes"): {_("Modified"): C_UPD_BOTH}, - } - - for label1, v1 in change_labels.items(): - iter1 = self.store.append(None, [label1, ""]) - for label2, change_type in v1.items(): - rows = [] - for change in changes: - _type, handle, class_name, obj1, obj2 = change - if _type == change_type: - if obj1 is not None: - if class_name == "Tag": - assert isinstance(obj1, Tag) # for type checker - gid = obj1.name - else: - gid = obj1.gramps_id - else: - assert obj2 # for type checker - if class_name == "Tag": - assert isinstance(obj2, Tag) # for type checker - gid = obj2.name - else: - gid = obj2.gramps_id - obj_details = [class_name, gid] - rows.append(obj_details) - if rows: - label2 = f"{label2} ({len(rows)})" - iter2 = self.store.append(iter1, [label2, ""]) - for row in rows: - self.store.append(iter2, row) - - # expand first level - for i, row in enumerate(self.store): - self.tree_view.expand_row(Gtk.TreePath(i), False) - - self.set_complete() - - -class SyncProgressPage(Page): - """Page showing database sync progress.""" - - def __init__(self, assistant): - super().__init__(assistant) - label = Gtk.Label(label="") - label.set_line_wrap(True) - label.set_use_markup(True) - label.set_max_width_chars(60) - self.label = label - self.pack_start(self.label, False, False, 0) - - self.label_progressbar_api = Gtk.Label(label="") - self.label_progressbar_api.set_margin_top(50) - self.pack_start(self.label_progressbar_api, False, False, 20) - - self.progressbar_api = Gtk.ProgressBar() - self.pack_start(self.progressbar_api, False, False, 20) - - media_label = Gtk.Label(label=_("Fetching information about media files...")) - media_label.set_line_wrap(True) - media_label.set_use_markup(True) - media_label.set_max_width_chars(60) - self.media_label = media_label - self.media_label.set_margin_top(50) - self.pack_start(self.media_label, False, False, 0) - - def update_api_progress(self, progress: float) -> None: - """Update the progress bar for the API transaction endpoint.""" - if progress >= 0: - self.progressbar_api.set_fraction(progress) - else: - self.progressbar_api.pulse() - # force updating progress bar - while Gtk.events_pending(): - Gtk.main_iteration() + self.warning_label.show() - def prepare(self, actions: Actions): - if len(actions) == 0: - self.label.set_text(_("Both trees are the same.")) - self.label_progressbar_api.hide() - self.progressbar_api.hide() + def _render_media(self, session: SyncSession) -> None: + """Offer the media transfer, and name the files nothing can be done for.""" + if session.has_missing_files: + self.media_check.set_label( + media_label(len(session.missing_local), len(session.missing_remote)) + ) + self.media_check.show() else: - self.media_label.hide() - if has_local_actions(actions): - self.label.set_text(_("Applying changes to local database ...")) + self.media_check.hide() + if session.missing_both: + self.media_notice.set_markup( + _dim(missing_both_notice(len(session.missing_both))) + ) + self.media_notice.show() else: - self.label.set_text(_("No changes to apply to local database.")) - if has_remote_actions(actions): - self.label_progressbar_api.show() - self.label_progressbar_api.set_text( - _("Applying changes to remote database ...") + self.media_notice.hide() + + +class ResultPane(Gtk.Box): + """Reports the outcome: a title, the error if there was one, and a summary.""" + + def __init__(self) -> None: + Gtk.Box.__init__(self, orientation=Gtk.Orientation.VERTICAL, spacing=12) + self.set_valign(Gtk.Align.CENTER) + + self.title_label = _label(xalign=0.5) + self.pack_start(self.title_label, False, False, 0) + + self.message_label = _label(xalign=0.5) + self.pack_start(self.message_label, False, False, 0) + + self.summary_label = _label(xalign=0.5) + self.pack_start(self.summary_label, False, False, 0) + + self.notice_label = _label(xalign=0.5) + self.notice_label.set_no_show_all(True) + self.notice_label.hide() + self.pack_start(self.notice_label, False, False, 0) + + self.details = Gtk.Expander(label=_("Details")) + self.details.set_no_show_all(True) + self.details.hide() + details_box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6) + self.details_label = _label() + self.details_label.set_selectable(True) + details_box.pack_start(self.details_label, False, False, 0) + copy_button = Gtk.Button(label=_("Copy")) + copy_button.set_halign(Gtk.Align.START) + copy_button.connect("clicked", self._on_copy) + details_box.pack_start(copy_button, False, False, 0) + self.details.add(details_box) + self.pack_start(self.details, False, False, 0) + + self._details_text = "" + + def prepare(self, session: SyncSession) -> None: + """Render the outcome of ``session``.""" + error = session.error + if error is None: + self.title_label.set_markup( + f"{GLib.markup_escape_text(_('Synchronization complete'))}" + "" ) - self.progressbar_api.show() + self.message_label.set_text("") + self._set_details("") else: - self.label_progressbar_api.set_text( - _("No changes to apply to remote database.") + self.title_label.set_markup( + f"{GLib.markup_escape_text(_('Synchronization failed'))}" + "" ) - self.progressbar_api.hide() - - def handle_local_sync_complete(self, actions: Actions) -> None: - """Handle completion of local sync.""" - if not has_local_actions(actions): - return - self.label.set_text(_("Successfully applied changes to local database.")) - - def handle_done_syncing_dbs(self) -> None: - """Handle completion of syncing the databases.""" - self.media_label.show() - - -class FileConfirmationPage(Page): - """File sync confirmation page.""" - - def __init__(self, assistant): - super().__init__(assistant) - self.store = Gtk.TreeStore(str) - - # tree view - self.tree_view = Gtk.TreeView(model=self.store) - - for i, col in enumerate(["ID"]): - renderer = Gtk.CellRendererText() - column = Gtk.TreeViewColumn(col, renderer, text=i) - self.tree_view.append_column(column) - - # scrolled window - scrolled_window = Gtk.ScrolledWindow() - scrolled_window.add(self.tree_view) - - self.pack_start(scrolled_window, True, True, 0) - - def prepare(self, missing_local, missing_remote): - iter_local = self.store.append(None, [_("Missing locally")]) - for gramps_id, handle in missing_local: - self.store.append(iter_local, [gramps_id]) - iter_remote = self.store.append(None, [_("Missing remotely")]) - for gramps_id, handle in missing_remote: - self.store.append(iter_remote, [gramps_id]) - - # expand first level - for i, row in enumerate(self.store): - self.tree_view.expand_row(Gtk.TreePath(i), False) - - self.set_complete() - - -class FileProgressPage(Page): - """A file progress page.""" - - def __init__(self, assistant): - """Initialize page.""" - super().__init__(assistant) - self.label1 = Gtk.Label(label="Media file download") - self.pack_start(self.label1, False, False, 20) - - self.progressbar1 = Gtk.ProgressBar() - self.pack_start(self.progressbar1, False, False, 20) - - self.label2 = Gtk.Label(label="Media file upload") - self.pack_start(self.label2, False, False, 20) + self.message_label.set_text(error_message(error.kind, error.detail)) + self._set_details( + f"{error.kind.name}: {error.detail}" + if error.detail + else error.kind.name + ) + self.summary_label.set_text(outcome_summary(session)) + + def show_notice(self, message: str) -> None: + """Show a non-fatal notice alongside the outcome.""" + self.notice_label.set_markup(f"{GLib.markup_escape_text(message)}") + self.notice_label.show() + + def _set_details(self, text: str) -> None: + """Offer the raw failure text, for pasting into a bug report.""" + self._details_text = text + if text: + self.details_label.set_text(text) + self.details.show() + else: + self.details.hide() - self.progressbar2 = Gtk.ProgressBar() - self.pack_start(self.progressbar2, False, False, 20) + def _on_copy(self, _button) -> None: + """Put the details on the clipboard.""" + from gi.repository import Gdk - def prepare(self, files_missing_local, files_missing_remote): - """Prepare.""" - n_down = len(files_missing_local) - if not n_down: - self.label1.hide() - self.progressbar1.hide() - else: - self.label1.show() - self.progressbar1.show() - self.label1.set_text(_("Downloading %s media file(s)") % n_down) - n_up = len(files_missing_remote) - if not n_up: - self.label2.hide() - self.progressbar2.hide() - else: - self.label2.show() - self.progressbar2.show() - self.label2.set_text(_("Uploading %s media file(s)") % n_up) - - def update_progress( - self, files_missing_local, files_missing_remote, downloaded, uploaded - ): - """Update the progress bar.""" - n_down = len(files_missing_local) - n_up = len(files_missing_remote) - i_down = len(downloaded) - i_up = len(uploaded) - if n_down: - self.progressbar1.set_fraction(i_down / n_down) - if n_up: - self.progressbar2.set_fraction(i_up / n_up) - - -class ConclusionPage(Page): - """The conclusion page.""" - - def __init__(self, assistant): - super().__init__(assistant) - self.error = False - self.unchanged = False - label = Gtk.Label(label="") - label.set_line_wrap(True) - label.set_use_markup(True) - label.set_max_width_chars(60) - self.label = label - self.pack_start(self.label, False, False, 0) + clipboard = Gtk.Clipboard.get(Gdk.SELECTION_CLIPBOARD) + clipboard.set_text(self._details_text, -1) class GrampsWebSyncOptions(ToolOptions): diff --git a/GrampsWebSync/po/template.pot b/GrampsWebSync/po/template.pot index 19c2b8ea3..459feaf4f 100644 --- a/GrampsWebSync/po/template.pot +++ b/GrampsWebSync/po/template.pot @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2025-10-11 21:37+0200\n" +"POT-Creation-Date: 2026-08-06 07:24+0200\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -16,9 +16,10 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" +"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n" -#: GrampsWebSync/grampswebsync.gpr.py:29 GrampsWebSync/grampswebsync.py:116 -#: GrampsWebSync/grampswebsync.py:209 +#: GrampsWebSync/grampswebsync.gpr.py:29 GrampsWebSync/grampswebsync.py:204 +#: GrampsWebSync/grampswebsync.py:269 msgid "Gramps Web Sync" msgstr "" @@ -26,172 +27,456 @@ msgstr "" msgid "Synchronizes a local database with a Gramps Web instance." msgstr "" -#: GrampsWebSync/grampswebsync.py:125 -msgid "Introduction" +#: GrampsWebSync/presentation.py:126 +msgid "unknown" +msgstr "" + +#: GrampsWebSync/presentation.py:128 +#, python-format +msgid "Gramps Web Sync %(addon)s, Web API %(api)s" +msgstr "" + +#: GrampsWebSync/presentation.py:132 +#, python-format +msgid "Gramps Web Sync %s" +msgstr "" + +#: GrampsWebSync/presentation.py:164 +msgid "" +"This URL uses http, so your password will be sent in clear text. Use only " +"for local testing." +msgstr "" + +#: GrampsWebSync/presentation.py:182 +#, python-format +msgid "" +"The system keyring could not be used. Snap confinement blocks access until " +"you run: %s" +msgstr "" + +#: GrampsWebSync/presentation.py:186 +msgid "" +"The system keyring could not be used. You will need to enter your password " +"each time." +msgstr "" + +#: GrampsWebSync/presentation.py:203 +msgid "Authentication failed. Please check your username and password." +msgstr "" + +#: GrampsWebSync/presentation.py:206 +msgid "Access forbidden. Please check username and password." msgstr "" -#: GrampsWebSync/grampswebsync.py:136 -msgid "Login" +#: GrampsWebSync/presentation.py:208 +msgid "GrampsWeb service not found. Please check the URL." msgstr "" -#: GrampsWebSync/grampswebsync.py:142 GrampsWebSync/grampswebsync.py:168 -msgid "Progress Information" +#: GrampsWebSync/presentation.py:210 +msgid "Too many requests, please try again in a few seconds." msgstr "" -#: GrampsWebSync/grampswebsync.py:147 -msgid "Final confirmation" +#: GrampsWebSync/presentation.py:212 +msgid "GrampsWeb tree is disabled." msgstr "" -#: GrampsWebSync/grampswebsync.py:154 GrampsWebSync/grampswebsync.py:172 -msgid "Summary" +#: GrampsWebSync/presentation.py:214 +msgid "Connection failed. Please check the URL and your internet connection." msgstr "" -#: GrampsWebSync/grampswebsync.py:161 -msgid "Media Files" +#: GrampsWebSync/presentation.py:217 +msgid "Invalid server response. Please check the URL." msgstr "" -#: GrampsWebSync/grampswebsync.py:264 +#: GrampsWebSync/presentation.py:220 msgid "Your user does not have sufficient server permissions to use sync." msgstr "" -#: GrampsWebSync/grampswebsync.py:269 -msgid "Fetching remote data..." +#: GrampsWebSync/presentation.py:222 +msgid "Failed importing downloaded XML file." msgstr "" -#: GrampsWebSync/grampswebsync.py:285 +#: GrampsWebSync/presentation.py:224 +msgid "Unable to synchronize changes to server: objects have been modified." +msgstr "" + +#: GrampsWebSync/presentation.py:226 msgid "Unexpected error while applying changes." msgstr "" -#: GrampsWebSync/grampswebsync.py:323 +#: GrampsWebSync/presentation.py:228 +msgid "" +"The family tree was modified while the changes were being reviewed. Nothing " +"has been applied. Please compare again." +msgstr "" + +#: GrampsWebSync/presentation.py:235 +#, python-format +msgid "" +"This server runs Gramps Web API %(found)s, but synchronization needs " +"%(needed)s or newer. Please update the server." +msgstr "" + +#: GrampsWebSync/presentation.py:239 +#, python-format +msgid "" +"This server did not report a Gramps Web API version, so it is too old to " +"synchronize with. Version %s or newer is needed." +msgstr "" + +#: GrampsWebSync/presentation.py:244 +#, python-format +msgid "" +"This server runs Gramps Web API %(found)s. This version of Gramps works with " +"Gramps Web API %(major)s, so synchronizing with this server needs a newer " +"version of Gramps." +msgstr "" + +#: GrampsWebSync/presentation.py:250 +msgid "" +"This server is not configured to use a background task queue. " +"Synchronization needs one: without it, applying changes times out before the " +"server has finished. Please enable it on the server." +msgstr "" + +#: GrampsWebSync/presentation.py:256 +#, python-format +msgid "The server could not apply the changes: %s" +msgstr "" + +#: GrampsWebSync/presentation.py:258 +#, python-format +msgid "Server error %s. Please check your connection." +msgstr "" + +#: GrampsWebSync/presentation.py:260 GrampsWebSync/presentation.py:261 +#, python-format +msgid "Unexpected error: %s" +msgstr "" + +#: GrampsWebSync/presentation.py:267 +msgid "Signing in…" +msgstr "" + +#: GrampsWebSync/presentation.py:268 +msgid "Downloading the remote family tree…" +msgstr "" + +#: GrampsWebSync/presentation.py:269 +msgid "Comparing the two family trees…" +msgstr "" + +#: GrampsWebSync/presentation.py:270 +msgid "Checking which media files are missing…" +msgstr "" + +#: GrampsWebSync/presentation.py:271 +msgid "Applied the changes to this computer." +msgstr "" + +#: GrampsWebSync/presentation.py:272 +msgid "Sending the changes to the server…" +msgstr "" + +#: GrampsWebSync/presentation.py:284 +msgid "Downloading media files…" +msgstr "" + +#: GrampsWebSync/presentation.py:285 +msgid "Uploading media files…" +msgstr "" + +#: GrampsWebSync/presentation.py:293 +msgid "Connect" +msgstr "" + +#: GrampsWebSync/presentation.py:294 +msgid "Compare" +msgstr "" + +#: GrampsWebSync/presentation.py:295 +msgid "Apply changes" +msgstr "" + +#: GrampsWebSync/presentation.py:296 +msgid "Transfer media files" +msgstr "" + +#: GrampsWebSync/presentation.py:304 +msgid "Bidirectional synchronization" +msgstr "" + +#: GrampsWebSync/presentation.py:305 +msgid "Reset the server to match this computer" +msgstr "" + +#: GrampsWebSync/presentation.py:306 +msgid "Reset this computer to match the server" +msgstr "" + +#: GrampsWebSync/presentation.py:319 +msgid "" +"Changes from both sides are combined. Objects edited in both places are " +"merged." +msgstr "" + +#: GrampsWebSync/presentation.py:323 +msgid "" +"The server is made to match this computer. Anything changed only on the " +"server is discarded." +msgstr "" + +#: GrampsWebSync/presentation.py:327 +msgid "" +"This computer is made to match the server. Anything changed only here is " +"discarded." +msgstr "" + +#: GrampsWebSync/presentation.py:574 +#, python-format +msgid "Will change on this computer (%s object)" +msgid_plural "Will change on this computer (%s objects)" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:582 +#, python-format +msgid "Will change on the server (%s object)" +msgid_plural "Will change on the server (%s objects)" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:593 +#, python-format +msgid "Add %s object" +msgid_plural "Add %s objects" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:594 +#, python-format +msgid "Update %s object" +msgid_plural "Update %s objects" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:595 +#, python-format +msgid "Merge %s object" +msgid_plural "Merge %s objects" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:596 +#, python-format +msgid "Delete %s object" +msgid_plural "Delete %s objects" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:615 +#, python-format +msgid "%s object will be deleted on this computer." +msgid_plural "%s objects will be deleted on this computer." +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:624 +#, python-format +msgid "%s object will be deleted on the server." +msgid_plural "%s objects will be deleted on the server." +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:636 +#, python-format +msgid "%(down)s to download, %(up)s to upload" +msgstr "" + +#: GrampsWebSync/presentation.py:642 +#, python-format +msgid "Also transfer %(total)s media file (%(counts)s)" +msgid_plural "Also transfer %(total)s media files (%(counts)s)" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:658 +#, python-format +msgid "%s media file is missing on both sides and cannot be transferred." +msgid_plural "" +"%s media files are missing on both sides and cannot be transferred." +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:681 +#, python-format +msgid "Applied %s change." +msgid_plural "Applied %s changes." +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:684 +msgid "Both trees are already in sync." +msgstr "" + +#: GrampsWebSync/presentation.py:689 msgid "Media files are in sync." msgstr "" -#: GrampsWebSync/grampswebsync.py:332 +#: GrampsWebSync/presentation.py:708 #, python-format msgid "Successfully downloaded %s media file." msgid_plural "Successfully downloaded %s media files." msgstr[0] "" msgstr[1] "" -#: GrampsWebSync/grampswebsync.py:335 +#: GrampsWebSync/presentation.py:717 #, python-format msgid "Encountered %s error during download." msgid_plural "Encountered %s errors during download." msgstr[0] "" msgstr[1] "" -#: GrampsWebSync/grampswebsync.py:341 +#: GrampsWebSync/presentation.py:727 #, python-format msgid "Successfully uploaded %s media file." msgid_plural "Successfully uploaded %s media files." msgstr[0] "" msgstr[1] "" -#: GrampsWebSync/grampswebsync.py:344 +#: GrampsWebSync/presentation.py:736 #, python-format msgid "Encountered %s error during upload." msgid_plural "Encountered %s errors during upload." msgstr[0] "" msgstr[1] "" -#: GrampsWebSync/grampswebsync.py:362 -msgid "Authentication failed. Please check your username and password." +#: GrampsWebSync/presentation.py:768 +msgid "No server configured" msgstr "" -#: GrampsWebSync/grampswebsync.py:366 -msgid "Access forbidden. Please check username and password." +#: GrampsWebSync/presentation.py:769 +#, python-format +msgid "%(user)s on %(server)s" msgstr "" -#: GrampsWebSync/grampswebsync.py:370 -msgid "GrampsWeb service not found. Please check the URL." +#: GrampsWebSync/presentation.py:774 +#, python-format +msgid "%(where)s — %(synced)s" msgstr "" -#: GrampsWebSync/grampswebsync.py:374 -msgid "Too many requests, please try again in a few seconds." +#: GrampsWebSync/presentation.py:788 +msgid "Never synced" msgstr "" -#: GrampsWebSync/grampswebsync.py:378 -msgid "GrampsWeb tree is disabled." +#: GrampsWebSync/presentation.py:792 +msgid "Last synced just now" msgstr "" -#: GrampsWebSync/grampswebsync.py:382 +#: GrampsWebSync/presentation.py:795 #, python-format -msgid "Server error %s. Please check your connection." +msgid "Last synced %s minute ago" +msgid_plural "Last synced %s minutes ago" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:801 +#, python-format +msgid "Last synced %s hour ago" +msgid_plural "Last synced %s hours ago" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:806 +#, python-format +msgid "Last synced %s day ago" +msgid_plural "Last synced %s days ago" +msgstr[0] "" +msgstr[1] "" + +#: GrampsWebSync/presentation.py:807 +#, python-format +msgid "Last synced on %s" msgstr "" -#: GrampsWebSync/grampswebsync.py:387 -msgid "Connection failed. Please check the URL and your internet connection." +#: GrampsWebSync/grampswebsync.py:246 +msgid "_Cancel" msgstr "" -#: GrampsWebSync/grampswebsync.py:391 -msgid "Invalid server response. Please check the URL." +#: GrampsWebSync/grampswebsync.py:248 +msgid "_Close" msgstr "" -#: GrampsWebSync/grampswebsync.py:394 -#, python-format -msgid "Unexpected error: %s" +#: GrampsWebSync/grampswebsync.py:249 +msgid "_Try again" msgstr "" -#: GrampsWebSync/grampswebsync.py:438 GrampsWebSync/grampswebsync.py:459 -msgid "Error accessing media object." +#: GrampsWebSync/grampswebsync.py:250 +msgid "C_onnect" msgstr "" -#: GrampsWebSync/grampswebsync.py:500 -msgid "Failed importing downloaded XML file." +#: GrampsWebSync/grampswebsync.py:251 +msgid "_Apply" msgstr "" -#: GrampsWebSync/grampswebsync.py:505 -msgid "Comparing local and remote data..." +#: GrampsWebSync/grampswebsync.py:345 +msgid "Forget this server?" msgstr "" -#: GrampsWebSync/grampswebsync.py:537 -msgid "Server authorization error." +#: GrampsWebSync/grampswebsync.py:347 +msgid "" +"The address, user name and password stored for this server will be removed, " +"along with the record of when this family tree last synchronized with it. " +"The next synchronization will compare the two trees from scratch." msgstr "" -#: GrampsWebSync/grampswebsync.py:540 -msgid "Server authorization error: insufficient permissions." +#: GrampsWebSync/grampswebsync.py:352 +msgid "Forget" msgstr "" -#: GrampsWebSync/grampswebsync.py:543 -msgid "Error: URL not found." +#: GrampsWebSync/grampswebsync.py:353 +msgid "Cancel" msgstr "" -#: GrampsWebSync/grampswebsync.py:547 -msgid "Unable to synchronize changes to server: objects have been modified." +#: GrampsWebSync/grampswebsync.py:480 +msgid "" +"These credentials were last used with a different family tree. Check the " +"server before continuing." msgstr "" -#: GrampsWebSync/grampswebsync.py:551 -#, python-format -msgid "Error %s while connecting to server." +#: GrampsWebSync/grampswebsync.py:556 +msgid "Change server…" msgstr "" -#: GrampsWebSync/grampswebsync.py:554 -msgid "URL error while connecting to server." +#: GrampsWebSync/grampswebsync.py:591 +msgid "Server URL:" msgstr "" -#: GrampsWebSync/grampswebsync.py:557 -msgid "Unable to synchronize changes to server." +#: GrampsWebSync/grampswebsync.py:593 +msgid "Username:" msgstr "" -#: GrampsWebSync/grampswebsync.py:585 -msgid "Continue without transport encryption?" +#: GrampsWebSync/grampswebsync.py:594 +msgid "Password:" msgstr "" -#: GrampsWebSync/grampswebsync.py:587 -msgid "" -"You have specified a URL with http scheme. If you continue, your password " -"will be sent in clear text over the network. Use only for local testing!" +#: GrampsWebSync/grampswebsync.py:598 +msgid "Remember password" msgstr "" -#: GrampsWebSync/grampswebsync.py:592 -msgid "Continue with HTTP" +#: GrampsWebSync/grampswebsync.py:610 +msgid "Forget this server" msgstr "" -#: GrampsWebSync/grampswebsync.py:593 -msgid "Use HTTPS" +#: GrampsWebSync/grampswebsync.py:641 +msgid "About this tool" msgstr "" -#: GrampsWebSync/grampswebsync.py:711 +#: GrampsWebSync/grampswebsync.py:645 +msgid "Open the online manual" +msgstr "" + +#: GrampsWebSync/grampswebsync.py:658 msgid "" "This tool allows to synchronize the currently opened family tree with a " "remote family tree served by Gramps Web.\n" @@ -207,105 +492,42 @@ msgid "" "option to make manual modifications, use the Import Merge Tool instead." msgstr "" -#: GrampsWebSync/grampswebsync.py:738 -msgid "Server URL: " -msgstr "" - -#: GrampsWebSync/grampswebsync.py:747 -msgid "Username: " -msgstr "" - -#: GrampsWebSync/grampswebsync.py:755 -msgid "Password: " -msgstr "" - -#: GrampsWebSync/grampswebsync.py:847 -msgid "Bidirectional Synchronization" -msgstr "" - -#: GrampsWebSync/grampswebsync.py:856 -msgid "Reset remote to local" -msgstr "" - -#: GrampsWebSync/grampswebsync.py:864 -msgid "Reset local to remote" -msgstr "" - -#: GrampsWebSync/grampswebsync.py:872 -msgid "Merge" -msgstr "" - -#: GrampsWebSync/grampswebsync.py:897 -msgid "Local changes" +#: GrampsWebSync/grampswebsync.py:729 +msgid "Error:" msgstr "" -#: GrampsWebSync/grampswebsync.py:898 GrampsWebSync/grampswebsync.py:903 -msgid "Added" +#: GrampsWebSync/grampswebsync.py:756 GrampsWebSync/grampswebsync.py:1012 +msgid "Warning:" msgstr "" -#: GrampsWebSync/grampswebsync.py:899 GrampsWebSync/grampswebsync.py:904 -msgid "Deleted" +#: GrampsWebSync/grampswebsync.py:901 +msgid "Sync mode:" msgstr "" -#: GrampsWebSync/grampswebsync.py:900 GrampsWebSync/grampswebsync.py:905 -#: GrampsWebSync/grampswebsync.py:907 -msgid "Modified" +#: GrampsWebSync/grampswebsync.py:929 +msgid "Change" msgstr "" -#: GrampsWebSync/grampswebsync.py:902 -msgid "Remote changes" +#: GrampsWebSync/grampswebsync.py:929 +msgid "Name" msgstr "" -#: GrampsWebSync/grampswebsync.py:907 -msgid "Simultaneous changes" +#: GrampsWebSync/grampswebsync.py:929 +msgid "ID" msgstr "" -#: GrampsWebSync/grampswebsync.py:964 -msgid "Fetching information about media files..." +#: GrampsWebSync/grampswebsync.py:1056 +msgid "Details" msgstr "" -#: GrampsWebSync/grampswebsync.py:985 -msgid "Both trees are the same." +#: GrampsWebSync/grampswebsync.py:1063 +msgid "Copy" msgstr "" -#: GrampsWebSync/grampswebsync.py:991 -msgid "Applying changes to local database ..." +#: GrampsWebSync/grampswebsync.py:1077 +msgid "Synchronization complete" msgstr "" -#: GrampsWebSync/grampswebsync.py:993 -msgid "No changes to apply to local database." -msgstr "" - -#: GrampsWebSync/grampswebsync.py:996 -msgid "Applying changes to remote database ..." -msgstr "" - -#: GrampsWebSync/grampswebsync.py:999 -msgid "No changes to apply to remote database." -msgstr "" - -#: GrampsWebSync/grampswebsync.py:1006 -msgid "Successfully applied changes to local database." -msgstr "" - -#: GrampsWebSync/grampswebsync.py:1035 -msgid "Missing locally" -msgstr "" - -#: GrampsWebSync/grampswebsync.py:1038 -msgid "Missing remotely" -msgstr "" - -#: GrampsWebSync/grampswebsync.py:1076 -#, python-format -msgid "Downloading %s media file" -msgid_plural "Downloading %s media files" -msgstr[0] "" -msgstr[1] "" - #: GrampsWebSync/grampswebsync.py:1084 -#, python-format -msgid "Uploading %s media file" -msgid_plural "Uploading %s media files" -msgstr[0] "" -msgstr[1] "" +msgid "Synchronization failed" +msgstr "" diff --git a/GrampsWebSync/presentation.py b/GrampsWebSync/presentation.py new file mode 100644 index 000000000..fd78f7bef --- /dev/null +++ b/GrampsWebSync/presentation.py @@ -0,0 +1,807 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Everything the interface needs that does not need GTK. + +Two kinds of thing live here: the wording, and the shaping of session data into +something a widget can render row by row. Neither touches a widget, so both can +be tested without a display, and the rule for where a new string goes is simply +whether it needs GTK. + +:func:`build_review` is the important one. The confirmation list used to show +*changes* -- what differs between the two trees -- while the user was choosing +a *mode* that decides what to do about them, so under "Reset remote to local" +rows filed under "Added" were in fact about to be deleted from the server. It +takes actions instead, grouped by the database they will change. +""" + +from __future__ import annotations + +import logging +import os +import time +from collections import defaultdict +from dataclasses import dataclass +from typing import TYPE_CHECKING +from urllib.parse import urlparse + +from const import ( + A_ADD_LOC, + A_ADD_REM, + A_DEL_LOC, + A_DEL_REM, + A_MRG_REM, + A_UPD_LOC, + A_UPD_REM, + API_MAJOR_TEXT, + MIN_API_VERSION_TEXT, + MODE_BIDIRECTIONAL, + MODE_RESET_TO_LOCAL, + MODE_RESET_TO_REMOTE, + Actions, +) +from gramps.gen.const import GRAMPS_LOCALE as glocale +from session import ( + STATUS_COMPARING, + STATUS_CONNECTING, + STATUS_FETCHING, + STATUS_LOCAL_APPLIED, + STATUS_PUSHING, + STATUS_SCANNING_MEDIA, + ErrorKind, + State, +) + +if TYPE_CHECKING: + # Imported for annotations only: `adapters` pulls in GTK, and this module + # stays importable without it. + from adapters import KeyringUnavailable + from session import SyncSession + +LOG = logging.getLogger("grampswebsync") + +assert glocale is not None # for type checker +try: + _trans = glocale.get_addon_translator(__file__) +except ValueError: + _trans = glocale.translation +_ = _trans.gettext +ngettext = _trans.ngettext + +#: Gramps' own catalogue, for terms it already translates in every language. +#: Reusing them keeps object class names off the addon's translators' plate. +_core = glocale.translation.gettext + +#: The tool's plugin id, as registered in ``grampswebsync.gpr.py``. +PLUGIN_ID = "gramps_web_sync" + +#: How long a note excerpt may get before it is cut short. +NOTE_EXCERPT = 60 + + +# ------------------------------------------------------------ +# +# Versions +# +# ------------------------------------------------------------ +def addon_version() -> str: + """Return the addon's registered version. + + :returns: The version, or ``""`` if the plugin registry cannot supply it. + """ + try: + from gramps.gen.plug import PluginRegister + + plugin = PluginRegister.get_instance().get_plugin(PLUGIN_ID) + except Exception as exc: # noqa: BLE001 -- a version is never worth a crash + LOG.debug("Cannot read the addon version: %s", exc) + return "" + return getattr(plugin, "version", "") or "" + + +def version_line(api_version: str | None) -> str: + """Return the footer naming this addon and the server it is talking to. + + Shown so that a bug report says which build produced it, and so that a + version mismatch is visible before it turns into a failure. + + :param api_version: The server's Gramps Web API version, once known. + :returns: A one-line summary. + """ + addon = addon_version() or _("unknown") + if api_version: + return _("Gramps Web Sync %(addon)s, Web API %(api)s") % { + "addon": addon, + "api": api_version, + } + return _("Gramps Web Sync %s") % addon + + +# ------------------------------------------------------------ +# +# URLs +# +# ------------------------------------------------------------ +def sanitize_url(url: str) -> str: + """Return the URL to actually use for a server. + + Only completes what the user typed. The warning about plain http is a + separate question, answered by :func:`is_insecure` as the URL is edited, + rather than by a modal fired once the user has already moved on. + + :param url: The URL as typed. + :returns: The URL with a scheme. + """ + url = url.strip() + if url and urlparse(url).scheme == "": + return f"https://{url}" + return url + + +def is_insecure(url: str) -> bool: + """Whether ``url`` would send the password in clear text.""" + return urlparse(url.strip()).scheme == "http" + + +def insecure_warning() -> str: + """Return the notice shown while an http URL is in the entry.""" + return _( + "This URL uses http, so your password will be sent in clear text. " + "Use only for local testing." + ) + + +# ------------------------------------------------------------ +# +# Errors, status and other wording the view shows verbatim +# +# ------------------------------------------------------------ +def keyring_message(problem: KeyringUnavailable) -> str: + """Return the localized notice for an unusable keyring. + + :param problem: What the keyring reported. + :returns: A message suitable for display. + """ + if problem.snap_command: + return _( + "The system keyring could not be used. Snap confinement blocks " + "access until you run: %s" + ) % problem.snap_command + return _( + "The system keyring could not be used. " + "You will need to enter your password each time." + ) + + +def error_message(kind: ErrorKind, detail: str = "") -> str: + """Return the localized message for an error kind. + + Translation lives here rather than in :mod:`session` so the flow logic can + be asserted on stable enum values instead of translated prose. + + :param kind: The classification recorded by the session. + :param detail: Optional extra context, e.g. an HTTP status. + :returns: A message suitable for display. + """ + messages = { + ErrorKind.AUTH_FAILED: _( + "Authentication failed. Please check your username and password." + ), + ErrorKind.FORBIDDEN: _( + "Access forbidden. Please check username and password." + ), + ErrorKind.NOT_FOUND: _("GrampsWeb service not found. Please check the URL."), + ErrorKind.RATE_LIMITED: _( + "Too many requests, please try again in a few seconds." + ), + ErrorKind.TREE_DISABLED: _("GrampsWeb tree is disabled."), + ErrorKind.CONNECTION_FAILED: _( + "Connection failed. Please check the URL and your internet connection." + ), + ErrorKind.INVALID_RESPONSE: _( + "Invalid server response. Please check the URL." + ), + ErrorKind.INSUFFICIENT_PERMISSIONS: _( + "Your user does not have sufficient server permissions to use sync." + ), + ErrorKind.XML_IMPORT_FAILED: _("Failed importing downloaded XML file."), + ErrorKind.CONFLICT: _( + "Unable to synchronize changes to server: objects have been modified." + ), + ErrorKind.APPLY_FAILED: _("Unexpected error while applying changes."), + ErrorKind.STALE_LOCAL_DATA: _( + "The family tree was modified while the changes were being " + "reviewed. Nothing has been applied. Please compare again." + ), + } + if kind is ErrorKind.SERVER_TOO_OLD: + if detail: + return _( + "This server runs Gramps Web API %(found)s, but synchronization " + "needs %(needed)s or newer. Please update the server." + ) % {"found": detail, "needed": MIN_API_VERSION_TEXT} + return _( + "This server did not report a Gramps Web API version, so it is too " + "old to synchronize with. Version %s or newer is needed." + ) % MIN_API_VERSION_TEXT + if kind is ErrorKind.SERVER_TOO_NEW: + return _( + "This server runs Gramps Web API %(found)s. This version of Gramps " + "works with Gramps Web API %(major)s, so synchronizing with this " + "server needs a newer version of Gramps." + ) % {"found": detail, "major": API_MAJOR_TEXT} + if kind is ErrorKind.SERVER_NO_TASK_QUEUE: + return _( + "This server is not configured to use a background task queue. " + "Synchronization needs one: without it, applying changes times " + "out before the server has finished. Please enable it on the " + "server." + ) + if kind is ErrorKind.SERVER_TASK_FAILED: + return _("The server could not apply the changes: %s") % detail + if kind is ErrorKind.SERVER_ERROR: + return _("Server error %s. Please check your connection.") % detail + if kind is ErrorKind.UNEXPECTED: + return _("Unexpected error: %s") % detail + return messages.get(kind, _("Unexpected error: %s") % detail) + + +def status_message(stage: str) -> str: + """Return the detail line shown while a step runs.""" + messages = { + STATUS_CONNECTING: _("Signing in…"), + STATUS_FETCHING: _("Downloading the remote family tree…"), + STATUS_COMPARING: _("Comparing the two family trees…"), + STATUS_SCANNING_MEDIA: _("Checking which media files are missing…"), + STATUS_LOCAL_APPLIED: _("Applied the changes to this computer."), + STATUS_PUSHING: _("Sending the changes to the server…"), + } + return messages.get(stage, "") + + +def transfer_message(kind: str) -> str: + """Return the detail line for a media transfer in progress. + + :param kind: The progress channel, as reported by the session. + :returns: The message, or ``""`` for a channel that is not a transfer. + """ + messages = { + "download": _("Downloading media files…"), + "upload": _("Uploading media files…"), + } + return messages.get(kind, "") + + +def state_label(state: State) -> str: + """Return the name of one phase in the working pane's list.""" + labels = { + State.CONNECTING: _("Connect"), + State.COMPARING: _("Compare"), + State.APPLYING: _("Apply changes"), + State.TRANSFERRING: _("Transfer media files"), + } + return labels.get(state, "") + + +def mode_label(mode: int) -> str: + """Return the name of a sync mode.""" + labels = { + MODE_BIDIRECTIONAL: _("Bidirectional synchronization"), + MODE_RESET_TO_LOCAL: _("Reset the server to match this computer"), + MODE_RESET_TO_REMOTE: _("Reset this computer to match the server"), + } + return labels.get(mode, "") + + +def mode_description(mode: int) -> str: + """Return the one-line explanation of a sync mode. + + With per-object selection out of scope the mode is the user's only control, + so each option has to say what it will do. + """ + descriptions = { + MODE_BIDIRECTIONAL: _( + "Changes from both sides are combined. Objects edited in both " + "places are merged." + ), + MODE_RESET_TO_LOCAL: _( + "The server is made to match this computer. Anything changed only " + "on the server is discarded." + ), + MODE_RESET_TO_REMOTE: _( + "This computer is made to match the server. Anything changed only " + "here is discarded." + ), + } + return descriptions.get(mode, "") + + +# ------------------------------------------------------------ +# +# Describing objects +# +# ------------------------------------------------------------ +def object_type_label(obj_type: str) -> str: + """Return the localized name of a Gramps object class.""" + return _core(obj_type) + + +def object_id(obj, obj_type: str) -> str: + """Return the Gramps ID of ``obj``. + + :returns: The ID, or ``""`` for a tag, which has none. + """ + if obj_type == "Tag": + return "" + return getattr(obj, "gramps_id", "") or "" + + +def _describe(obj, obj_type: str, db) -> str: + """Return a human-readable name for one object, without guarding.""" + if obj_type == "Person": + from gramps.gen.display.name import displayer as name_displayer + + return name_displayer.display(obj) + if obj_type == "Family": + if db is None: + return "" + from gramps.gen.utils.db import family_name + + return family_name(obj, db) + if obj_type == "Place": + if db is not None: + from gramps.gen.display.place import displayer as place_displayer + + return place_displayer.display(db, obj) + return obj.get_title() + if obj_type == "Event": + return obj.get_description() or str(obj.get_type()) + if obj_type == "Media": + return obj.get_description() or os.path.basename(obj.get_path() or "") + if obj_type == "Note": + text = " ".join((obj.get() or "").split()) + return text[:NOTE_EXCERPT] + "…" if len(text) > NOTE_EXCERPT else text + if obj_type == "Source": + return obj.get_title() + if obj_type == "Citation": + return obj.get_page() + if obj_type == "Repository": + return obj.get_name() + if obj_type == "Tag": + return obj.get_name() + return "" + + +def describe_object(obj, obj_type: str, db=None) -> str: + """Return a human-readable name for one object. + + A row identifying only a class and an ID -- "Person / I0123" -- tells the + user nothing about what is being changed, which is the whole point of the + review. + + :param obj: The object to describe. + :param obj_type: Its Gramps class name. + :param db: The database it came from, needed for the classes whose name is + assembled from other objects. Omitting it degrades those to ``""``. + :returns: The name, or ``""`` if none could be derived. + """ + if obj is None: + return "" + try: + return (_describe(obj, obj_type, db) or "").strip() + except Exception as exc: # noqa: BLE001 -- a label is never worth a crash + LOG.debug("Cannot describe %s: %s", obj_type, exc) + return "" + + +# ------------------------------------------------------------ +# +# The review model +# +# ------------------------------------------------------------ +#: Which database an action writes to. +LOCAL = "local" +REMOTE = "remote" + +VERB_ADD = "add" +VERB_UPDATE = "update" +VERB_MERGE = "merge" +VERB_DELETE = "delete" + +#: Verbs in the order they are listed, deletions last so the destructive part +#: of a group reads as the exception rather than the headline. +VERB_ORDER = (VERB_ADD, VERB_UPDATE, VERB_MERGE, VERB_DELETE) + +#: What each action does, as ``(destination, verb)``. A merge writes the +#: combined object to both databases, so it has two effects and appears twice. +ACTION_EFFECTS: dict[str, tuple[tuple[str, str], ...]] = { + A_ADD_LOC: ((LOCAL, VERB_ADD),), + A_UPD_LOC: ((LOCAL, VERB_UPDATE),), + A_DEL_LOC: ((LOCAL, VERB_DELETE),), + A_ADD_REM: ((REMOTE, VERB_ADD),), + A_UPD_REM: ((REMOTE, VERB_UPDATE),), + A_DEL_REM: ((REMOTE, VERB_DELETE),), + A_MRG_REM: ((LOCAL, VERB_MERGE), (REMOTE, VERB_MERGE)), +} + + +@dataclass(frozen=True) +class ObjectRow: + """One object as it appears in the review list. + + :param type_label: The localized class name, e.g. "Person". + :param name: What the object is called, or ``""`` if it has no name. + :param gramps_id: Its Gramps ID, or ``""`` for a tag. + """ + + type_label: str + name: str + gramps_id: str + + +@dataclass(frozen=True) +class ActionGroup: + """The objects one verb applies to, within one destination. + + :param verb: One of the ``VERB_*`` constants. + :param rows: The objects, sorted. + """ + + verb: str + rows: tuple[ObjectRow, ...] + + @property + def count(self) -> int: + """How many objects this verb applies to.""" + return len(self.rows) + + +@dataclass(frozen=True) +class Destination: + """Everything that will change in one of the two databases. + + :param where: :data:`LOCAL` or :data:`REMOTE`. + :param groups: The verbs applying there, in :data:`VERB_ORDER`. + """ + + where: str + groups: tuple[ActionGroup, ...] + + @property + def count(self) -> int: + """How many objects will change here.""" + return sum(group.count for group in self.groups) + + +@dataclass(frozen=True) +class ReviewModel: + """What a run will do, grouped for display. + + :param destinations: The databases that will change, local first. A + database nothing happens to is left out. + :param local_deletions: How many objects will be removed locally. + :param remote_deletions: How many will be removed on the server. + """ + + destinations: tuple[Destination, ...] + local_deletions: int + remote_deletions: int + + @property + def deletes(self) -> bool: + """Whether anything will be removed on either side.""" + return bool(self.local_deletions or self.remote_deletions) + + +def _row_key(row: ObjectRow) -> tuple[str, str, str]: + """Sort rows by type, then name, then ID, so a list never reshuffles.""" + return (row.type_label, row.name, row.gramps_id) + + +def build_review(actions: Actions, db1=None, db2=None) -> ReviewModel: + """Group actions by the database they change and what they do to it. + + :param actions: The actions the selected sync mode produced. + :param db1: The local database, for naming local objects. + :param db2: The remote database, for naming objects only it has. + :returns: The grouped model. + """ + buckets: dict[tuple[str, str], list[ObjectRow]] = defaultdict(list) + deletions = {LOCAL: 0, REMOTE: 0} + + for typ, _handle, obj_type, obj1, obj2 in actions: + effects = ACTION_EFFECTS.get(typ) + if effects is None: + LOG.warning("Not showing unknown action type %s", typ) + continue + obj, db = (obj1, db1) if obj1 is not None else (obj2, db2) + row = ObjectRow( + type_label=object_type_label(obj_type), + name=describe_object(obj, obj_type, db), + gramps_id=object_id(obj, obj_type) if obj is not None else "", + ) + for where, verb in effects: + buckets[(where, verb)].append(row) + if verb == VERB_DELETE: + deletions[where] += 1 + + destinations = [] + for where in (LOCAL, REMOTE): + groups = tuple( + ActionGroup(verb, tuple(sorted(buckets[(where, verb)], key=_row_key))) + for verb in VERB_ORDER + if buckets.get((where, verb)) + ) + if groups: + destinations.append(Destination(where, groups)) + + return ReviewModel( + destinations=tuple(destinations), + local_deletions=deletions[LOCAL], + remote_deletions=deletions[REMOTE], + ) + + +# ------------------------------------------------------------ +# +# Wording +# +# ------------------------------------------------------------ +def destination_label(where: str, count: int) -> str: + """Return the heading for one destination. + + Phrased as what will happen rather than as where a difference was found, + since that is the question the review answers. + """ + if where == LOCAL: + return ( + ngettext( + "Will change on this computer (%s object)", + "Will change on this computer (%s objects)", + count, + ) + % count + ) + return ( + ngettext( + "Will change on the server (%s object)", + "Will change on the server (%s objects)", + count, + ) + % count + ) + + +def verb_label(verb: str, count: int) -> str: + """Return the heading for one group of objects within a destination.""" + labels = { + VERB_ADD: ngettext("Add %s object", "Add %s objects", count), + VERB_UPDATE: ngettext("Update %s object", "Update %s objects", count), + VERB_MERGE: ngettext("Merge %s object", "Merge %s objects", count), + VERB_DELETE: ngettext("Delete %s object", "Delete %s objects", count), + } + return labels.get(verb, "%s") % count + + +def deletion_warning(model: ReviewModel) -> str: + """Return the warning shown above a run that removes data, if it does. + + Derived from the actions rather than from the selected mode: a plain + bidirectional sync propagates deletions too, and the user deserves the same + warning when it does. + + :param model: The review model for the selected mode. + :returns: The warning, or ``""`` when nothing will be deleted. + """ + parts = [] + if model.local_deletions: + parts.append( + ngettext( + "%s object will be deleted on this computer.", + "%s objects will be deleted on this computer.", + model.local_deletions, + ) + % model.local_deletions + ) + if model.remote_deletions: + parts.append( + ngettext( + "%s object will be deleted on the server.", + "%s objects will be deleted on the server.", + model.remote_deletions, + ) + % model.remote_deletions + ) + return " ".join(parts) + + +def media_label(n_download: int, n_upload: int) -> str: + """Return the label for the media transfer checkbox.""" + total = n_download + n_upload + counts = _("%(down)s to download, %(up)s to upload") % { + "down": n_download, + "up": n_upload, + } + return ( + ngettext( + "Also transfer %(total)s media file (%(counts)s)", + "Also transfer %(total)s media files (%(counts)s)", + total, + ) + % {"total": total, "counts": counts} + ) + + +def missing_both_notice(count: int) -> str: + """Return the note about files that exist on neither side. + + Called out before the transfer rather than after it: these used to be + attempted in both directions and reported as two failures each. + """ + return ( + ngettext( + "%s media file is missing on both sides and cannot be transferred.", + "%s media files are missing on both sides and cannot be transferred.", + count, + ) + % count + ) + + +def outcome_summary(session: SyncSession) -> str: + """Describe what a run actually did, to both trees and to the media files. + + Composed for a failed run as well as a successful one, so the caller can + show it alongside an error. + + :param session: The finished session. + :returns: One or more sentences describing the outcome. + """ + parts = [] + applied = len(session.actions) + # Reported first and unconditionally: left to a fallback, a run with + # nothing to do was described entirely by the media sentence below. + if applied: + parts.append( + ngettext("Applied %s change.", "Applied %s changes.", applied) % applied + ) + elif session.error is None: + parts.append(_("Both trees are already in sync.")) + transfer = transfer_summary(session) + if transfer: + parts.append(transfer) + elif not session.missing_both and session.error is None: + parts.append(_("Media files are in sync.")) + if session.missing_both: + parts.append(missing_both_notice(len(session.missing_both))) + return " ".join(parts) + + +def transfer_summary(session: SyncSession) -> str: + """Summarize how many media files moved, and how many failed. + + :param session: The session whose transfers to report. + :returns: The summary, or ``""`` if nothing was transferred. + """ + # Spelled out per call site rather than looked up from a table: xgettext + # extracts literals, and a table would leave these out of the catalogue. + parts = [] + ok, nok = _tally(session.downloaded) + if ok: + parts.append( + ngettext( + "Successfully downloaded %s media file.", + "Successfully downloaded %s media files.", + ok, + ) + % ok + ) + if nok: + parts.append( + ngettext( + "Encountered %s error during download.", + "Encountered %s errors during download.", + nok, + ) + % nok + ) + ok, nok = _tally(session.uploaded) + if ok: + parts.append( + ngettext( + "Successfully uploaded %s media file.", + "Successfully uploaded %s media files.", + ok, + ) + % ok + ) + if nok: + parts.append( + ngettext( + "Encountered %s error during upload.", + "Encountered %s errors during upload.", + nok, + ) + % nok + ) + return " ".join(parts) + + +def _tally(outcomes: dict[str, bool]) -> tuple[int, int]: + """Return how many transfers succeeded and how many did not.""" + ok = sum(1 for succeeded in outcomes.values() if succeeded) + return ok, len(outcomes) - ok + + +def context_lines( + url: str, username: str, tree_name: str = "", last_synced: str = "" +) -> tuple[str, str]: + """Return the two lines naming what is being synced, and how current it is. + + The remote tree's own name is the heading once the server has reported it. + It carries more than the address does where the address cannot distinguish + anything: a hosted deployment serves many trees from one URL, and only the + account differs. + + :param url: The server being synced with. + :param username: The account on it. + :param tree_name: What the server calls its tree, once known. + :param last_synced: The already-formatted last-sync phrase. + :returns: The heading and the line below it. + """ + if not url or not username: + return _("No server configured"), "" + where = _("%(user)s on %(server)s") % {"user": username, "server": url} + if not tree_name: + return where, last_synced + if not last_synced: + return tree_name, where + return tree_name, _("%(where)s — %(synced)s") % { + "where": where, + "synced": last_synced, + } + + +def format_last_synced(timestamp: float, now: float | None = None) -> str: + """Describe when a server was last synced with. + + :param timestamp: The stored baseline, or ``0`` if there is none. + :param now: The current time; the wall clock if omitted. + :returns: A phrase for the context strip. + """ + if not timestamp: + return _("Never synced") + now = time.time() if now is None else now + minutes = int(max(now - timestamp, 0) // 60) + if minutes < 1: + return _("Last synced just now") + if minutes < 60: + return ( + ngettext("Last synced %s minute ago", "Last synced %s minutes ago", minutes) + % minutes + ) + hours = minutes // 60 + if hours < 24: + return ( + ngettext("Last synced %s hour ago", "Last synced %s hours ago", hours) + % hours + ) + days = hours // 24 + if days < 30: + return ngettext("Last synced %s day ago", "Last synced %s days ago", days) % days + return _("Last synced on %s") % time.strftime("%x", time.localtime(timestamp)) diff --git a/GrampsWebSync/session.py b/GrampsWebSync/session.py new file mode 100644 index 000000000..c111c972c --- /dev/null +++ b/GrampsWebSync/session.py @@ -0,0 +1,1158 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2021-2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Headless sync session for the Gramps Web Sync addon. + +:class:`SyncSession` runs a synchronization against a Gramps Web server, +progressing through the stages in :class:`State`. Callers drive it with +:meth:`~SyncSession.submit_credentials` and :meth:`~SyncSession.confirm`, and +observe it through a :class:`SessionListener`. A failed run can be resumed with +:meth:`~SyncSession.retry`. + +Collaborators are supplied as ports: :class:`Backend`, +:class:`CredentialStore`, :class:`MediaStore`, :class:`TaskRunner` and +:class:`Clock`. Failures are recorded as a :class:`SyncError` carrying an +:class:`ErrorKind`; callers are responsible for localizing them. + +Each stage is split into the part that touches a database and the part that +talks to the network, because only the latter may leave the main loop -- see +:class:`adapters.IoRunner`. :data:`Step` names the pieces so that a retry can +resume at the one that failed rather than redoing the work before it. +""" + +from __future__ import annotations + +import logging +from collections.abc import Callable +from dataclasses import dataclass +from enum import Enum, auto +from pathlib import Path +from typing import Any, Protocol + +from const import API_MAJOR, MIN_API_VERSION, MODE_BIDIRECTIONAL, Actions +from diffhandler import ( + WebApiSyncDiffHandler, + changes_to_actions, + has_local_actions, + has_remote_actions, +) +from gramps.gen.db import DbTxn +from gramps.gen.db.utils import import_as_dict +from gramps.gen.errors import HandleError +from webapihandler import ServerTaskFailed, parse_version, transaction_to_json + +LOG = logging.getLogger("grampswebsync") + +#: Transaction description recorded in both databases' undo history. +TXN_MSG = "Apply Gramps Web Sync changes" + +#: Server permission required to run a sync at all. +REQUIRED_PERMISSION = "ViewPrivate" + +#: A media transfer, resolved to a local path before it leaves the main loop. +Transfers = list[tuple[str, str, str]] + +#: Stages reported through :meth:`SessionListener.on_status`. +STATUS_CONNECTING = "connecting" +STATUS_FETCHING = "fetching" +STATUS_COMPARING = "comparing" +STATUS_SCANNING_MEDIA = "scanning_media" +STATUS_LOCAL_APPLIED = "local_applied" +STATUS_PUSHING = "pushing" + + +# ------------------------------------------------------------ +# +# States and errors +# +# ------------------------------------------------------------ +class State(Enum): + """The stages of a sync run, in the order they occur.""" + + CONNECT = auto() # waiting for the user to supply credentials + CONNECTING = auto() # authenticating against the server + COMPARING = auto() # downloading, diffing and checking media + REVIEW = auto() # waiting for the user to confirm what will happen + APPLYING = auto() # writing both databases + TRANSFERRING = auto() # moving media files + DONE = auto() + FAILED = auto() + + +#: The stages the view renders as a progress list, in order. Every other state +#: waits for the user or reports an outcome. +WORKING_STATES = ( + State.CONNECTING, + State.COMPARING, + State.APPLYING, + State.TRANSFERRING, +) + + +class Step(Enum): + """The resumable pieces of a run. + + A stage that both touches a database and talks to the network is two of + these, so that a retry after, say, a dropped connection while pushing does + not re-apply the local half that already succeeded. + + Connecting is absent: it fails back to :attr:`State.CONNECT` for the user to + correct rather than ending the run, so there is nothing to resume. + """ + + FETCH = auto() # download the remote XML (network) + DIFF = auto() # import it and compare (database) + SCAN_MEDIA = auto() # ask which files the server lacks (network) + APPLY_LOCAL = auto() # write the local half (database) + PUSH_REMOTE = auto() # send the remote half (network) + TRANSFER = auto() # move media files (network) + + +class ErrorKind(Enum): + """Classification of a failure, independent of its localized wording.""" + + AUTH_FAILED = auto() # HTTP 401 + FORBIDDEN = auto() # HTTP 403 + NOT_FOUND = auto() # HTTP 404 + RATE_LIMITED = auto() # HTTP 429 + TREE_DISABLED = auto() # HTTP 503 + CONFLICT = auto() # HTTP 409 + SERVER_ERROR = auto() + SERVER_TASK_FAILED = auto() + CONNECTION_FAILED = auto() + INVALID_RESPONSE = auto() + INSUFFICIENT_PERMISSIONS = auto() + SERVER_TOO_OLD = auto() + SERVER_TOO_NEW = auto() + SERVER_NO_TASK_QUEUE = auto() + XML_IMPORT_FAILED = auto() + APPLY_FAILED = auto() + STALE_LOCAL_DATA = auto() + UNEXPECTED = auto() + + +@dataclass(frozen=True) +class SyncError: + """A failure recorded by the session. + + :param kind: The classification. + :param detail: Untranslated detail, e.g. an HTTP status or exception text. + """ + + kind: ErrorKind + detail: str = "" + + +class XmlImportFailed(Exception): + """The downloaded Gramps XML could not be imported.""" + + +class ApplyFailed(Exception): + """Applying the confirmed actions to the databases raised.""" + + +class StaleLocalData(Exception): + """The local tree changed between the comparison and the commit.""" + + +# Login and mid-sync failures read a few status codes differently. +_LOGIN_HTTP_ERRORS: dict[int, ErrorKind] = { + 401: ErrorKind.AUTH_FAILED, + 403: ErrorKind.FORBIDDEN, + 404: ErrorKind.NOT_FOUND, + 429: ErrorKind.RATE_LIMITED, + 503: ErrorKind.TREE_DISABLED, +} + +_SYNC_HTTP_ERRORS: dict[int, ErrorKind] = { + 401: ErrorKind.AUTH_FAILED, + 403: ErrorKind.FORBIDDEN, + 404: ErrorKind.NOT_FOUND, + 409: ErrorKind.CONFLICT, +} + + +def classify_http_error(exc: Any, *, login: bool) -> SyncError: + """Classify an :class:`HTTPError` into a :class:`SyncError`. + + :param exc: The raised error. + :param login: Whether this happened while establishing the connection. + :returns: The corresponding :class:`SyncError`. + """ + table = _LOGIN_HTTP_ERRORS if login else _SYNC_HTTP_ERRORS + kind = table.get(exc.code, ErrorKind.SERVER_ERROR) + return SyncError(kind, str(exc.code)) + + +def api_version_problem(version: str | None) -> ErrorKind | None: + """Classify a server's API version against what this addon speaks. + + :param version: The server's ``gramps_webapi`` version, as reported. + :returns: The problem, or ``None`` if the version is usable. An absent or + unreadable version counts as :attr:`ErrorKind.SERVER_TOO_OLD`. + """ + if not version: + return ErrorKind.SERVER_TOO_OLD + try: + major, _minor = parse_version(version) + except ValueError: + LOG.warning("Server reported an unreadable API version: %s", version) + return ErrorKind.SERVER_TOO_OLD + if major > API_MAJOR: + return ErrorKind.SERVER_TOO_NEW + # An older major is below the minimum too, so one comparison covers both + # an out-of-date major and an out-of-date minor within the current one. + if (major, _minor) < MIN_API_VERSION: + return ErrorKind.SERVER_TOO_OLD + return None + + +# ------------------------------------------------------------ +# +# Ports +# +# ------------------------------------------------------------ +class Backend(Protocol): + """What the session needs from a Gramps Web server. + + Implemented by :class:`webapihandler.WebApiHandler`. + """ + + def get_permissions(self) -> set[str]: ... + + def get_api_version(self) -> str | None: ... + + def has_task_queue(self) -> bool: ... + + def get_tree_name(self) -> str: ... + + def get_lang(self) -> str | None: ... + + def download_xml(self) -> Path: ... + + def commit( + self, + payload: list[dict[str, Any]], + force: bool = True, + progress_callback: Callable | None = None, + ) -> None: ... + + def get_missing_files(self) -> list[dict[str, Any]]: ... + + def download_media_file(self, handle: str, path: str) -> bool: ... + + def upload_media_file(self, handle: str, path: str) -> bool: ... + + +class CredentialStore(Protocol): + """Persistence for server credentials and per-server sync baselines.""" + + def get_url(self) -> str: ... + + def get_username(self) -> str: ... + + def get_password(self) -> str | None: ... + + def get_timestamp(self, url: str, username: str) -> float: ... + + def set_timestamp(self, url: str, username: str, timestamp: float) -> None: ... + + def save_credentials( + self, url: str, username: str, password: str, remember_password: bool = True + ) -> None: ... + + +class MediaStore(Protocol): + """Access to local media files belonging to the local database.""" + + def full_path(self, media: Any) -> str: ... + + def exists(self, media: Any) -> bool: ... + + +class TaskRunner(Protocol): + """Runs a potentially slow callable and reports the outcome back.""" + + def run( + self, + func: Callable[[], Any], + on_success: Callable[[Any], None], + on_error: Callable[[BaseException], None], + ) -> None: ... + + def post(self, func: Callable[[], None]) -> None: ... + + +class Clock(Protocol): + """Source of the current time.""" + + def now(self) -> float: ... + + +class SessionListener(Protocol): + """Receives session state changes, status and progress updates.""" + + def on_state_changed(self, state: State) -> None: ... + + def on_progress(self, kind: str, fraction: float) -> None: ... + + def on_status(self, stage: str) -> None: ... + + +@dataclass(frozen=True) +class Connection: + """What one connect attempt learned about a server. + + Carried as one value because the whole attempt happens on a worker thread + and only its result crosses back to the main loop. + + :param backend: The connected handler. + :param api_version: The Gramps Web API version it reports, if any. + :param task_queue: Whether it runs transactions in the background. + :param tree_name: What the server calls the tree it serves. + :param permissions: What the authenticated account may do. + """ + + backend: Backend + api_version: str | None + task_queue: bool + tree_name: str + permissions: set[str] + + +# ------------------------------------------------------------ +# +# Transition table +# +# ------------------------------------------------------------ +def next_state(state: State, session: SyncSession) -> State: + """Return the state that follows ``state``. + + :param state: The state being left. + :param session: The session, consulted for the branch conditions. + :returns: The next state. + """ + if session.error is not None: + return State.FAILED + if state is State.CONNECT: + return State.CONNECTING + if state is State.CONNECTING: + return State.COMPARING + if state is State.COMPARING: + # Nothing to decide means nothing to show: a run that finds the trees + # identical and no file to move reports that and stops. + return State.REVIEW if session.has_review_content else State.DONE + if state is State.REVIEW: + return State.APPLYING + if state is State.APPLYING: + return State.TRANSFERRING if session.will_transfer else State.DONE + if state is State.TRANSFERRING: + return State.DONE + return state + + +#: Which state a retry of each step returns to while it runs. +STATE_FOR_STEP: dict[Step, State] = { + Step.FETCH: State.COMPARING, + Step.DIFF: State.COMPARING, + Step.SCAN_MEDIA: State.COMPARING, # overridden by _scan_origin + Step.APPLY_LOCAL: State.APPLYING, + Step.PUSH_REMOTE: State.APPLYING, + Step.TRANSFER: State.TRANSFERRING, +} + + +# ------------------------------------------------------------ +# +# SyncSession +# +# ------------------------------------------------------------ +class SyncSession: + """Drives one synchronization run against a Gramps Web server.""" + + def __init__( + self, + db, + user, + backend_factory: Callable[[str, str, str], Backend], + credentials: CredentialStore, + media: MediaStore, + runner: TaskRunner, + clock: Clock, + listener: SessionListener | None = None, + io_runner: TaskRunner | None = None, + ) -> None: + """Initialize the session. + + :param db: The local (currently open) Gramps database. + :param user: A :class:`gramps.gen.user.User` for import/diff progress. + :param backend_factory: Builds a :class:`Backend` from url, username + and password. + :param credentials: Where credentials and sync baselines live. + :param media: Access to local media files. + :param runner: Executes steps that touch a database, on the main loop. + :param clock: Supplies the time recorded as the last successful sync. + :param listener: Optional observer of state and progress. + :param io_runner: Executes network steps. Defaults to ``runner``, which + keeps everything on one thread -- useful in tests. + """ + self.db1 = db + self.db2 = None + self._user = user + self._backend_factory = backend_factory + self.credentials = credentials + self.media = media + self.runner = runner + self.io_runner = io_runner if io_runner is not None else runner + self.clock = clock + self.listener = listener + + self.state: State = State.CONNECT + self.error: SyncError | None = None + #: Set when connecting fails. Recoverable, unlike :attr:`error`. + self.login_error: SyncError | None = None + #: Which step failed, so :meth:`retry` can resume at the right place. + self.failed_in: Step | None = None + + self.url: str = "" + self.username: str = "" + self.password: str = "" + self.remember_password: bool = True + #: The server's Gramps Web API version, once it has reported one. + self.api_version: str | None = None + #: What the server calls the tree it serves, once it has said. + self.tree_name: str = "" + + self.backend: Backend | None = None + self.sync: WebApiSyncDiffHandler | None = None + self.changes: Actions = [] + self.actions: Actions = [] + self.sync_mode: int = MODE_BIDIRECTIONAL + #: Whether the confirmed run should also move media files. + self.transfer_media: bool = True + + self.missing_local: list[tuple[str, str]] = [] + self.missing_remote: list[tuple[str, str]] = [] + #: Media absent on both sides. Neither transfer can supply these, so + #: they are reported up front rather than as two failures each. + self.missing_both: list[tuple[str, str]] = [] + self.downloaded: dict[str, bool] = {} + self.uploaded: dict[str, bool] = {} + + #: Held between the two halves of the apply stage, because a retry + #: after a failed push must not re-run the local commit. Everything + #: else a step hands to its successor travels as an argument. + self._payload: list[dict[str, Any]] | None = None + #: Which stage the current media scan was scheduled from. + self._scan_origin: State = State.COMPARING + #: Bumped by :meth:`abandon`. Callbacks scheduled under an earlier + #: value belong to a run the user has walked away from, and are + #: dropped rather than allowed to drive the session on. + self._run_id = 0 + self._closing = False + + # -------------------------------------------------------- + # Observable state + # -------------------------------------------------------- + @property + def has_missing_files(self) -> bool: + """Whether any media file can actually be transferred either way.""" + return bool(self.missing_local or self.missing_remote) + + @property + def has_review_content(self) -> bool: + """Whether there is anything for the user to confirm. + + Media missing on both sides does not count: nothing can be done about + those, so they are reported with the outcome rather than as a decision. + """ + return bool(self.changes or self.has_missing_files) + + @property + def will_transfer(self) -> bool: + """Whether the confirmed run still has media files to move.""" + return self.transfer_media and self.has_missing_files + + @property + def has_local_actions(self) -> bool: + """Whether the pending actions touch the local database.""" + return has_local_actions(self.actions) + + @property + def has_remote_actions(self) -> bool: + """Whether the pending actions touch the remote database.""" + return has_remote_actions(self.actions) + + @property + def can_retry(self) -> bool: + """Whether :meth:`retry` has a step to resume.""" + return self.state is State.FAILED and self.failed_in is not None + + # -------------------------------------------------------- + # Internals + # -------------------------------------------------------- + def _goto(self, state: State) -> None: + """Enter ``state`` and notify the listener.""" + LOG.debug("Sync session: %s -> %s", self.state.name, state.name) + self.state = state + if self.listener is not None: + self.listener.on_state_changed(state) + + def _advance(self) -> None: + """Move to whatever :func:`next_state` says comes next.""" + self._goto(next_state(self.state, self)) + + def _fail(self, error: SyncError, step: Step | None = None) -> None: + """Record a terminal failure and move to :attr:`State.FAILED`. + + The remote database is deliberately kept open: a retry that had to + re-download and re-diff it would throw away the most expensive part of + the run for what is usually a transient network problem. + """ + LOG.warning("Sync failed: %s (%s)", error.kind.name, error.detail) + self.error = error + self.failed_in = step + self._goto(State.FAILED) + + def _progress(self, kind: str, fraction: float) -> None: + """Forward a progress update to the listener, if any.""" + if self.listener is not None: + self.listener.on_progress(kind, fraction) + + def _status(self, stage: str) -> None: + """Forward a status update to the listener, if any.""" + if self.listener is not None: + self.listener.on_status(stage) + + def _progress_from_worker(self, kind: str, fraction: float) -> None: + """Report progress raised inside a network step. + + Marshalled onto the main loop, since the listener draws widgets and the + step is running on a worker thread. + """ + self.io_runner.post(lambda: self._progress(kind, fraction)) + + def _status_from_worker(self, stage: str) -> None: + """Report a status update raised inside a network step.""" + self.io_runner.post(lambda: self._status(stage)) + + def _classify(self, exc: BaseException, *, login: bool = False) -> SyncError: + """Turn an exception raised by a port into a :class:`SyncError`.""" + # Imported here so the module stays importable without urllib present. + from urllib.error import HTTPError, URLError + + if isinstance(exc, XmlImportFailed): + return SyncError(ErrorKind.XML_IMPORT_FAILED) + if isinstance(exc, StaleLocalData): + return SyncError(ErrorKind.STALE_LOCAL_DATA) + if isinstance(exc, ServerTaskFailed): + return SyncError(ErrorKind.SERVER_TASK_FAILED, str(exc)) + if isinstance(exc, ApplyFailed): + return SyncError(ErrorKind.APPLY_FAILED, str(exc)) + if isinstance(exc, HTTPError): + return classify_http_error(exc, login=login) + if isinstance(exc, URLError): + return SyncError(ErrorKind.CONNECTION_FAILED, str(exc.reason)) + # A read that times out raises this directly, not wrapped in URLError. + if isinstance(exc, TimeoutError): + return SyncError(ErrorKind.CONNECTION_FAILED, str(exc)) + if isinstance(exc, ValueError): + kind = ErrorKind.INVALID_RESPONSE if login else ErrorKind.SERVER_ERROR + return SyncError(kind, str(exc)) + return SyncError(ErrorKind.UNEXPECTED, str(exc)) + + def _run(self, step: Step, func, on_success) -> None: + """Schedule ``step`` on the runner that is allowed to execute it. + + Network steps go to a worker thread; database steps stay on the main + loop, because a Gramps sqlite connection belongs to the thread that + created it. + """ + runner = ( + self.io_runner + if step + in (Step.FETCH, Step.PUSH_REMOTE, Step.SCAN_MEDIA, Step.TRANSFER) + else self.runner + ) + runner.run( + func, + self._guarded(on_success), + self._guarded(lambda exc: self._on_step_error(exc, step)), + ) + + def _guarded(self, callback: Callable[[Any], None]) -> Callable[[Any], None]: + """Wrap ``callback`` so a run the user has left cannot resume itself. + + :param callback: What to call if the run is still the current one. + :returns: The wrapped callback. + """ + run_id = self._run_id + + def guarded(value: Any) -> None: + if run_id == self._run_id: + callback(value) + else: + LOG.debug("Dropping a callback from an abandoned run.") + + return guarded + + # -------------------------------------------------------- + # Intents + # -------------------------------------------------------- + def submit_credentials( + self, + url: str, + username: str, + password: str, + remember_password: bool = True, + ) -> None: + """Connect, authenticate, then download and diff the remote tree. + + Runs off the main loop. Anything the user can correct -- a wrong + password, an unsupported server, an account without the required + permission -- returns to :attr:`State.CONNECT` with + :attr:`login_error` set rather than ending the run, and credentials + are stored only once the server has accepted them. + + :param url: Server URL, already sanitized by the caller. + :param username: Login name. + :param password: Password. + :param remember_password: Whether the password may be stored. + """ + self.login_error = None + # These describe the server, and the view displays them. Left set, + # they would label this attempt with the previous server's identity. + self.backend = None + self.api_version = None + self.tree_name = "" + self.url = url + self.username = username + self.password = password + self.remember_password = remember_password + self._goto(State.CONNECTING) + self.io_runner.run( + self._connect, + self._guarded(self._on_connected), + self._guarded(self._on_connect_error), + ) + + def confirm(self, sync_mode: int, transfer_media: bool = True) -> None: + """Accept what the review pane showed and carry it out. + + :param sync_mode: One of the ``MODE_*`` constants from :mod:`const`. + :param transfer_media: Whether to also move the missing media files. + """ + self.sync_mode = sync_mode + self.transfer_media = transfer_media + self._goto(State.APPLYING) + self._run(Step.APPLY_LOCAL, self._apply_local, self._on_local_applied) + + def retry(self) -> None: + """Resume a failed run at the step that failed. + + Everything before that step is left alone: the remote tree is still + downloaded and diffed, and a local commit that already succeeded is not + repeated. + """ + step = self.failed_in + if step is None: + return + LOG.info("Retrying sync from %s.", step.name) + self.error = None + self.failed_in = None + self._goto( + self._scan_origin if step is Step.SCAN_MEDIA else STATE_FOR_STEP[step] + ) + if step is Step.FETCH: + self._run(Step.FETCH, self._fetch_xml, self._on_fetched) + elif step is Step.DIFF: + self._start_compare() + elif step is Step.APPLY_LOCAL: + self._run(Step.APPLY_LOCAL, self._apply_local, self._on_local_applied) + elif step is Step.PUSH_REMOTE: + self._run(Step.PUSH_REMOTE, self._push_remote, self._on_applied) + elif step is Step.SCAN_MEDIA: + self._start_media_scan() + elif step is Step.TRANSFER: + self._start_transfer() + + def abandon(self) -> None: + """Stop the current run and return to the connect pane. + + Clears what identifies the connection along with the run, so nothing + left on screen describes the server being walked away from. Callbacks + already in flight are dropped. Does nothing once writing has begun, + which would otherwise leave no record of what got through. + """ + if self.state in (State.APPLYING, State.TRANSFERRING): + LOG.debug("Not abandoning a run that has started writing.") + return + LOG.info("Abandoning the run at %s.", self.state.name) + self._run_id += 1 + self._release_remote() + self.backend = None + self.url = "" + self.username = "" + self.password = "" + self.api_version = None + self.tree_name = "" + self.error = None + self.login_error = None + self.failed_in = None + self.changes = [] + self.actions = [] + self.missing_local = [] + self.missing_remote = [] + self.missing_both = [] + self.downloaded = {} + self.uploaded = {} + self._payload = None + self._goto(State.CONNECT) + + def cancel(self) -> None: + """Abandon the run and release the in-memory remote database.""" + self._closing = True + self._release_remote() + + def _release_remote(self) -> None: + """Close the downloaded remote database, if one is open.""" + if self.db2 is not None: + self.db2.close() + self.db2 = None + self.sync = None # holds references to both databases + + # -------------------------------------------------------- + # Connecting + # -------------------------------------------------------- + def _connect(self) -> Connection: + """Authenticate and read what the server can do. Network only.""" + self._status_from_worker(STATUS_CONNECTING) + backend = self._backend_factory(self.url, self.username, self.password) + return Connection( + backend=backend, + api_version=backend.get_api_version(), + task_queue=backend.has_task_queue(), + tree_name=backend.get_tree_name(), + permissions=backend.get_permissions(), + ) + + def _on_connected(self, connection: Connection) -> None: + """Store the credentials and start comparing, back on the main loop.""" + if self._closing: + return + self.api_version = connection.api_version + self.tree_name = connection.tree_name + problem = self._reject_server(connection) + if problem is not None: + self.backend = None + self.login_error = problem + self._goto(State.CONNECT) + return + self.backend = connection.backend + self.credentials.save_credentials( + self.url, self.username, self.password, self.remember_password + ) + self._start_compare() + + def _on_connect_error(self, exc: BaseException) -> None: + """Return to the connect pane with something the user can act on.""" + if self._closing: + return + self.backend = None + self.login_error = self._classify(exc, login=True) + self._goto(State.CONNECT) + + @staticmethod + def _reject_server(connection: Connection) -> SyncError | None: + """Return why this server cannot be synced with, or ``None``. + + Ordered from the server outwards. The version comes first because an + API too old to report a permission claim yields an empty set, which + would otherwise be reported as a problem with the user's account; what + the server is configured to do comes before what the account may do, + for the same reason. + """ + version_problem = api_version_problem(connection.api_version) + if version_problem is not None: + return SyncError(version_problem, connection.api_version or "") + if not connection.task_queue: + return SyncError(ErrorKind.SERVER_NO_TASK_QUEUE) + if REQUIRED_PERMISSION not in connection.permissions: + return SyncError(ErrorKind.INSUFFICIENT_PERMISSIONS) + return None + + # -------------------------------------------------------- + # Comparison + # -------------------------------------------------------- + def _start_compare(self) -> None: + """Enter the comparison stage and fetch the remote tree.""" + self._goto(State.COMPARING) + self._run(Step.FETCH, self._fetch_xml, self._on_fetched) + + def _fetch_xml(self) -> Path | None: + """Download the remote tree as Gramps XML. Network only. + + :returns: Where the export was written, for the next step. + """ + if self._closing: + return None + assert self.backend is not None + LOG.info("Downloading Gramps XML file.") + self._status_from_worker(STATUS_FETCHING) + path = self.backend.download_xml() + LOG.debug("Downloaded XML to %s", path) + return path + + def _on_fetched(self, path: Path | None) -> None: + """Import and diff, back on the main loop.""" + if self._closing or path is None: + return + self._run( + Step.DIFF, lambda: self._import_and_diff(path), self._on_compared + ) + + def _import_and_diff(self, path: Path) -> None: + """Import the downloaded XML and diff it against the local tree. + + Runs on the main loop: ``import_as_dict`` builds an in-memory sqlite + database on the calling thread, and ``diff_dbs`` reads the local one, + which belongs to the main thread. + + :param path: The export downloaded by :meth:`_fetch_xml`. It is + consumed here, which is why retrying this step downloads again. + """ + if self._closing: + return + try: + db2 = import_as_dict(str(path), self._user) + finally: + path.unlink(missing_ok=True) + if db2 is None: + raise XmlImportFailed() + self._release_remote() + self.db2 = db2 + + LOG.info("Comparing local and remote data.") + self._status(STATUS_COMPARING) + last_synced = self.credentials.get_timestamp(self.url, self.username) or None + self.sync = WebApiSyncDiffHandler( + self.db1, self.db2, user=self._user, last_synced=last_synced + ) + self.changes = self.sync.get_changes() + + def _on_compared(self, _result: Any) -> None: + """Check the media state too, so the review can present both at once.""" + if self._closing: + return + if not self.changes: + LOG.info("Databases are in sync.") + self.credentials.set_timestamp(self.url, self.username, self.clock.now()) + self._start_media_scan() + + # -------------------------------------------------------- + # Applying + # -------------------------------------------------------- + def _apply_local(self) -> None: + """Write the local half of the sync and build the remote payload. + + Runs on the main loop: both halves are prepared inside Gramps + transactions against databases owned by this thread. + """ + if self._closing: + return + assert self.backend is not None and self.sync is not None + self.actions = changes_to_actions(self.changes, self.sync_mode) + self._payload = [] + if not self.actions: + return + + self._assert_local_unchanged() + + LOG.info("Committing %s actions.", len(self.actions)) + try: + with DbTxn(TXN_MSG, self.sync.db1) as trans1: + with DbTxn(TXN_MSG, self.sync.db2) as trans2: + self.sync.commit_actions(self.actions, trans1, trans2) + lang = self.backend.get_lang() + self._payload = transaction_to_json(trans2, lang) + except StaleLocalData: + raise + except Exception as exc: + raise ApplyFailed(str(exc)) from exc + + if self.has_local_actions: + self._status(STATUS_LOCAL_APPLIED) + + def _assert_local_unchanged(self) -> None: + """Verify the local tree still matches what the comparison saw. + + The comparison captured object snapshots, and the user may have gone on + editing the tree while reviewing them -- the tool does not block the + main window. Committing those snapshots would silently overwrite any + edit made in between, so the run stops instead and re-compares. + + :raises StaleLocalData: If any affected object changed or appeared. + """ + for _typ, handle, obj_type, obj1, _obj2 in self.actions: + method = self.db1.method("get_%s_from_handle", obj_type) + if method is None: + continue + try: + current = method(handle) + except HandleError: + current = None + if obj1 is None: + # Absent locally when compared; anything here now is new. + if current is not None: + raise StaleLocalData(f"{obj_type} {handle} was added locally") + elif current is None: + raise StaleLocalData(f"{obj_type} {handle} was deleted locally") + elif current.change != obj1.change: + raise StaleLocalData(f"{obj_type} {handle} was modified locally") + + def _on_local_applied(self, _result: Any) -> None: + """Push the remote half, off the main loop.""" + if self._closing: + return + self._run(Step.PUSH_REMOTE, self._push_remote, self._on_applied) + + def _push_remote(self) -> None: + """Send the remote half of the sync to the server. Network only.""" + if self._closing: + return + assert self.backend is not None + if not self._payload: + return + self._status_from_worker(STATUS_PUSHING) + # Always force: the server compares against the XML-round-tripped + # object, which differs from the live one through serialization + # artifacts alone, yielding spurious 409s. + self.backend.commit( + self._payload, + True, + lambda fraction: self._progress_from_worker("api", fraction), + ) + self._payload = [] + + def _on_applied(self, _result: Any) -> None: + """Record the sync time and move on to the media files.""" + if self._closing: + return + self.credentials.set_timestamp(self.url, self.username, self.clock.now()) + if self._media_objects_changed(): + self._start_media_scan() + return + self._finish_or_transfer() + + def _media_objects_changed(self) -> bool: + """Whether the applied actions added or removed media objects. + + Their files are absent on whichever side just received them, so the + scan taken before the review no longer describes what to transfer and + has to be repeated. Most runs touch no media object at all and are + spared the second round trip. + """ + return any(action[2] == "Media" for action in self.actions) + + # -------------------------------------------------------- + # Media + # -------------------------------------------------------- + def _start_media_scan(self) -> None: + """Ask the server which files it lacks, off the main loop. + + The stage the scan was scheduled from is recorded, because a scan + happens both before the review and again after an apply that touched + media objects, and the two carry on differently. It also survives a + failure, which would otherwise leave a retry no way back. + """ + self._scan_origin = self.state + self._run(Step.SCAN_MEDIA, self._fetch_remote_missing, self._on_media_scanned) + + def _fetch_remote_missing(self) -> list[dict[str, Any]]: + """Return the server's list of media objects with no file. Network only.""" + if self._closing: + return [] + assert self.backend is not None + self._status_from_worker(STATUS_SCANNING_MEDIA) + return self.backend.get_missing_files() or [] + + def _on_media_scanned(self, remote: list[dict[str, Any]] | None) -> None: + """Combine the server's answer with a local scan, back on the main loop.""" + if self._closing: + return + self._scan_media(remote or []) + if self._scan_origin is State.APPLYING: + self._finish_or_transfer() + return + self._advance() + + def _finish_or_transfer(self) -> None: + """Enter whatever follows the apply stage, and start its work.""" + self._advance() + if self.state is State.TRANSFERRING: + self._start_transfer() + + def _scan_media(self, remote: list[dict[str, Any]]) -> None: + """Classify media objects by which side has the file on disk. + + Fills three disjoint lists of ``(gramps_id, handle)``: + :attr:`missing_local` (file on the server, not here), + :attr:`missing_remote` (file here, not on the server) and + :attr:`missing_both` (file on neither). + + :param remote: Media objects the server reports no file for. + """ + on_disk_here: set[str] = set() + absent_here: dict[str, str] = {} + for media in self.db1.iter_media(): + if self.media.exists(media): + on_disk_here.add(media.handle) + else: + absent_here[media.handle] = media.gramps_id + absent_on_server = { + media["handle"]: media["gramps_id"] for media in remote + } + + # `on_disk_here` is built only from media objects in db1, so a handle + # missing from it means either that db1 has the object but not its + # file, or that db1 has no such object yet. An upload is possible in + # neither case. + self.missing_remote = sorted( + (gid, h) for h, gid in absent_on_server.items() if h in on_disk_here + ) + self.missing_both = sorted( + (gid, h) for h, gid in absent_on_server.items() if h not in on_disk_here + ) + self.missing_local = sorted( + (gid, h) for h, gid in absent_here.items() if h not in absent_on_server + ) + if self.missing_both: + LOG.warning( + "%s media file(s) are missing on both sides.", len(self.missing_both) + ) + + def _resolve_transfers(self) -> tuple[Transfers, Transfers]: + """Turn the missing-file lists into paths, on the main loop. + + The transfer itself runs on a worker thread and must not touch the + local database, so every handle is resolved to a path first. Files a + previous attempt already moved are left out, so a retry after a dropped + connection resumes rather than starting over. + + :returns: The downloads and uploads still to do. + """ + downloads: Transfers = [] + uploads: Transfers = [] + for gramps_id, handle in self.missing_local: + path = self._path_for(handle) + if path is None: + self.downloaded[gramps_id] = False + elif not self.downloaded.get(gramps_id): + downloads.append((gramps_id, handle, path)) + for gramps_id, handle in self.missing_remote: + path = self._path_for(handle) + if path is None: + self.uploaded[gramps_id] = False + elif not self.uploaded.get(gramps_id): + uploads.append((gramps_id, handle, path)) + return downloads, uploads + + def _start_transfer(self) -> None: + """Resolve paths on the main loop, then transfer off it.""" + downloads, uploads = self._resolve_transfers() + self._run( + Step.TRANSFER, + lambda: self._transfer(downloads, uploads), + self._on_transferred, + ) + + def _path_for(self, handle: str) -> str | None: + """Return the local path of a media object, or ``None`` if unusable.""" + try: + obj = self.db1.get_media_from_handle(handle) + except HandleError: + LOG.warning("Cannot access media object %s", handle) + return None + return self.media.full_path(obj) + + def _transfer(self, downloads: Transfers, uploads: Transfers) -> None: + """Download then upload media files. Network only. + + Both loops check for cancellation between files, so closing the window + stops the transfer rather than waiting for it to finish. + + :param downloads: Files to fetch, as ``(gramps_id, handle, path)``. + :param uploads: Files to send, in the same form. + """ + if self._closing: + return + assert self.backend is not None + for index, (gramps_id, handle, path) in enumerate(downloads, start=1): + if self._closing: + return + LOG.debug("Downloading file %s", gramps_id) + self.downloaded[gramps_id] = self._download_one(handle, path) + self._progress_from_worker("download", index / len(downloads)) + for index, (gramps_id, handle, path) in enumerate(uploads, start=1): + if self._closing: + return + LOG.debug("Uploading file %s", gramps_id) + self.uploaded[gramps_id] = self._upload_one(handle, path) + self._progress_from_worker("upload", index / len(uploads)) + + def _on_transferred(self, _result: Any) -> None: + """Finish the run.""" + if self._closing: + return + self._advance() + + def _download_one(self, handle: str, path: str) -> bool: + """Download one media file, reporting failure rather than raising.""" + assert self.backend is not None + try: + return self.backend.download_media_file(handle, path) + except Exception as exc: # noqa: BLE001 -- one bad file must not abort + LOG.warning("Failed to download media file %s: %s", handle, exc) + return False + + def _upload_one(self, handle: str, path: str) -> bool: + """Upload one media file. + + :param handle: Handle of the media object to upload. + :param path: Where its file lives locally. + :returns: Whether the file was uploaded. + """ + import os + + assert self.backend is not None + if not os.path.exists(path): + LOG.warning("Cannot upload media file %s: not on disk (%s)", handle, path) + return False + return self.backend.upload_media_file(handle, path) + + # -------------------------------------------------------- + # Step failure + # -------------------------------------------------------- + def _on_step_error(self, exc: BaseException, step: Step) -> None: + """Handle an exception escaping one of the steps.""" + if self._closing: + return + # Stale local data means the snapshots are worthless; a retry has to + # compare again rather than resume where it stopped. + resume = Step.DIFF if isinstance(exc, StaleLocalData) else step + self._fail(self._classify(exc), resume) diff --git a/GrampsWebSync/tests/__init__.py b/GrampsWebSync/tests/__init__.py new file mode 100644 index 000000000..ddadbf783 --- /dev/null +++ b/GrampsWebSync/tests/__init__.py @@ -0,0 +1,45 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Test package for the Gramps Web Sync addon. + +Importing this package adds :data:`ADDON_DIR` and :data:`ADDONS_ROOT` to +``sys.path`` and sets ``GRAMPS_RESOURCES`` if unset, so test modules can +import ``gramps`` and the addon's flat modules directly. GTK/Gdk version +pinning is handled repo-wide by the root ``tests/__init__.py`` (PR #950). +""" + +from __future__ import annotations + +import os +import sys + +#: The ``GrampsWebSync`` addon directory, i.e. the parent of this package. +ADDON_DIR: str = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + +#: The ``addons-source`` checkout root. +ADDONS_ROOT: str = os.path.dirname(ADDON_DIR) +if ADDONS_ROOT not in sys.path: + sys.path.insert(0, ADDONS_ROOT) + +if "GRAMPS_RESOURCES" not in os.environ: + import gramps + + os.environ["GRAMPS_RESOURCES"] = os.path.dirname(os.path.dirname(gramps.__file__)) diff --git a/GrampsWebSync/tests/fakes.py b/GrampsWebSync/tests/fakes.py new file mode 100644 index 000000000..759206aa4 --- /dev/null +++ b/GrampsWebSync/tests/fakes.py @@ -0,0 +1,411 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""In-process test doubles for the :mod:`session` ports. + +:class:`FakeGrampsWebServer` implements :class:`session.Backend` over a real +Gramps database, exporting Gramps XML and applying transaction payloads; +:meth:`~FakeGrampsWebServer.fail_next` and +:meth:`~FakeGrampsWebServer.fail_always` inject faults. + +The remaining classes stand in for the other ports: +:class:`InlineTaskRunner`, :class:`FrozenClock`, +:class:`MemoryCredentialStore`, :class:`DirectoryMediaStore` and +:class:`RecordingListener`. None depend on a test framework. +""" + +from __future__ import annotations + +import os +import tempfile +from collections.abc import Callable +from pathlib import Path +from typing import Any +from urllib.error import HTTPError + +from const import MIN_API_VERSION_TEXT +from gramps.cli.user import User +from gramps.gen.db import DbTxn +from gramps.gen.db.utils import make_database +from gramps.gen.lib.json_utils import data_to_object +from gramps.plugins.export.exportxml import export_data + +#: Permissions a Gramps Web user needs for a sync to be allowed to proceed. +DEFAULT_PERMISSIONS = frozenset({"ViewPrivate", "EditObject", "AddObject"}) + +#: A Gramps Web API version new enough to pass :data:`const.MIN_API_VERSION`. +DEFAULT_API_VERSION = f"{MIN_API_VERSION_TEXT}.0" + + +def http_error(code: int, url: str = "https://example.org/api/") -> HTTPError: + """Build an :class:`HTTPError` with ``code``, for fault injection. + + :param code: The HTTP status to simulate. + :param url: The URL to attribute the error to. + :returns: A ready-to-raise :class:`HTTPError`. + """ + return HTTPError(url, code, f"Simulated HTTP {code}", {}, None) # type: ignore[arg-type] + + +# ------------------------------------------------------------ +# +# FakeGrampsWebServer +# +# ------------------------------------------------------------ +class FakeGrampsWebServer: + """A Gramps Web server backed by a real in-memory Gramps database. + + Satisfies the :class:`session.Backend` protocol. + + :param db: The database to serve. A fresh empty one is created if omitted. + :param permissions: Permissions to report for the logged-in user. + :param lang: Value returned by :meth:`get_lang`. + :param api_version: Value returned by :meth:`get_api_version`. ``None`` + stands for a server too old to report one at all. + :param task_queue: Whether the server reports a background task queue. + :param tree_name: What the server calls the tree it serves. + """ + + def __init__( + self, + db=None, + permissions: set[str] | frozenset[str] = DEFAULT_PERMISSIONS, + lang: str | None = "en", + api_version: str | None = DEFAULT_API_VERSION, + task_queue: bool = True, + tree_name: str = "Family Tree", + ) -> None: + if db is None: + db = make_database("sqlite") + db.load(":memory:") + self.db = db + self.permissions = set(permissions) + self.lang = lang + self.api_version = api_version + self.task_queue = task_queue + self.tree_name = tree_name + self.user = User(auto_accept=True, quiet=True) + + #: Handles of media objects whose file the server actually holds. + self.media_files: dict[str, bytes] = {} + #: Every method call made against this server, in order. + self.calls: list[str] = [] + #: Each payload passed to :meth:`commit`. + self.committed: list[list[dict[str, Any]]] = [] + #: Method name -> exception, raised once then cleared. + self._fail_once: dict[str, BaseException] = {} + #: Method name -> exception, raised on every call. + self._fail_always: dict[str, BaseException] = {} + self._tempfiles: list[Path] = [] + + # -------------------------------------------------------- + # Fault injection + # -------------------------------------------------------- + def fail_next(self, method: str, exc: BaseException) -> None: + """Make the next call to ``method`` raise ``exc``. + + :param method: Name of the backend method, e.g. ``"download_xml"``. + :param exc: The exception to raise. + """ + self._fail_once[method] = exc + + def fail_always(self, method: str, exc: BaseException) -> None: + """Make every call to ``method`` raise ``exc``.""" + self._fail_always[method] = exc + + def _enter(self, method: str) -> None: + """Record a call and honour any fault configured for it.""" + self.calls.append(method) + exc = self._fail_once.pop(method, None) or self._fail_always.get(method) + if exc is not None: + raise exc + + # -------------------------------------------------------- + # Backend protocol + # -------------------------------------------------------- + def get_permissions(self) -> set[str]: + """Return the logged-in user's permissions.""" + self._enter("get_permissions") + return set(self.permissions) + + def get_api_version(self) -> str | None: + """Return the Gramps Web API version this server claims to run.""" + self._enter("get_api_version") + return self.api_version + + def has_task_queue(self) -> bool: + """Whether this server claims a background task queue.""" + self._enter("has_task_queue") + return self.task_queue + + def get_tree_name(self) -> str: + """Return the name this server gives its tree.""" + self._enter("get_tree_name") + return self.tree_name + + def get_lang(self) -> str | None: + """Return the server's configured language.""" + self._enter("get_lang") + return self.lang + + def download_xml(self) -> Path: + """Export the served database to a Gramps XML file. + + The caller owns the file and is expected to unlink it. + + :returns: Path to the exported ``.gramps`` file. + """ + self._enter("download_xml") + handle, name = tempfile.mkstemp(suffix=".gramps", prefix="fakeweb_") + os.close(handle) + path = Path(name) + self._tempfiles.append(path) + if not export_data(self.db, str(path), self.user): + raise ValueError("Fake server failed to export XML") + return path + + def commit( + self, + payload: list[dict[str, Any]], + force: bool = True, + progress_callback: Callable | None = None, + ) -> None: + """Apply a transaction payload to the served database. + + :param payload: Items as produced by + :func:`webapihandler.transaction_to_json`. + :param force: Accepted for protocol compatibility; ignored. + :param progress_callback: Called with a fraction in ``[0, 1]``. + """ + self._enter("commit") + self.committed.append(payload) + if not payload: + return + with DbTxn("Fake server transaction", self.db, batch=True) as trans: + for index, item in enumerate(payload): + self._apply_item(item, trans) + if progress_callback is not None: + progress_callback((index + 1) / len(payload)) + + def _apply_item(self, item: dict[str, Any], trans: DbTxn) -> None: + """Apply a single transaction item to the served database.""" + class_name = item["_class"] + if item["type"] == "delete": + method = self.db.method("remove_%s", class_name) + assert method is not None + method(item["handle"], trans) + return + obj = data_to_object(item["new"]) + # commit_* upserts, covering both "add" and "update". Passing the + # object's own change time keeps timestamps meaningful across a sync. + method = self.db.method("commit_%s", class_name) + assert method is not None + method(obj, trans, obj.change) + + def get_missing_files(self) -> list[dict[str, Any]]: + """Return media objects the server knows about but has no file for.""" + self._enter("get_missing_files") + return [ + {"gramps_id": media.gramps_id, "handle": media.handle} + for media in self.db.iter_media() + if media.handle not in self.media_files + ] + + def download_media_file(self, handle: str, path: str) -> bool: + """Write the server's copy of a media file to ``path``.""" + self._enter("download_media_file") + if handle not in self.media_files: + raise http_error(404) + Path(path).parent.mkdir(parents=True, exist_ok=True) + Path(path).write_bytes(self.media_files[handle]) + return True + + def upload_media_file(self, handle: str, path: str) -> bool: + """Store a media file uploaded by the client.""" + self._enter("upload_media_file") + self.media_files[handle] = Path(path).read_bytes() + return True + + # -------------------------------------------------------- + # Lifecycle + # -------------------------------------------------------- + def close(self) -> None: + """Close the served database and remove leftover export files.""" + for path in self._tempfiles: + path.unlink(missing_ok=True) + self._tempfiles.clear() + try: + self.db.close() + except Exception: # noqa: BLE001 -- teardown must not mask failures + pass + + +# ------------------------------------------------------------ +# +# Simple doubles +# +# ------------------------------------------------------------ +class InlineTaskRunner: + """Runs each task synchronously on the calling thread. + + By the time ``run`` returns, the step and its completion callback have + both finished. Standing in for both runners keeps scenarios single-threaded + and their assertions deterministic. + """ + + def run( + self, + func: Callable[[], Any], + on_success: Callable[[Any], None], + on_error: Callable[[BaseException], None], + ) -> None: + """Execute ``func`` and dispatch to the appropriate callback.""" + try: + result = func() + except BaseException as exc: # noqa: BLE001 -- mirrors the real runner + on_error(exc) + else: + on_success(result) + + def post(self, func: Callable[[], None]) -> None: + """Run ``func`` immediately; there is no other thread to marshal from.""" + func() + + +class FrozenClock: + """A clock that only moves when :meth:`advance` is called. + + :param start: The initial time, as a POSIX timestamp. + """ + + def __init__(self, start: float = 1_700_000_000.0) -> None: + self.time = start + + def now(self) -> float: + """Return the current fake time.""" + return self.time + + def advance(self, seconds: float) -> None: + """Move the clock forward by ``seconds``.""" + self.time += seconds + + +class MemoryCredentialStore: + """In-memory stand-in for the config file and keyring. + + Keyed by ``(url, username)`` like the real store, so each server keeps its + own sync baseline and switching between them does not discard one. + + :param url: Initially stored server URL. + :param username: Initially stored user name. + :param password: Initially stored password. + :param timestamp: Initially stored last-sync time. + """ + + def __init__( + self, + url: str = "https://example.org/api", + username: str = "owner", + password: str = "secret", + timestamp: float = 0.0, + ) -> None: + self.url = url + self.username = username + self.password = password + #: ``(url, username)`` -> last successful sync time. + self.timestamps: dict[tuple[str, str], float] = {(url, username): timestamp} + #: ``(url, username)`` -> whether its password may be stored. + self.remembered: dict[tuple[str, str], bool] = {} + #: Every ``(url, username, password)`` passed to + #: :meth:`save_credentials`. + self.saved: list[tuple[str, str, str]] = [] + + @property + def timestamp(self) -> float: + """The baseline of the last-used entry, for convenient assertions.""" + return self.timestamps.get((self.url, self.username), 0.0) + + @timestamp.setter + def timestamp(self, value: float) -> None: + self.timestamps[(self.url, self.username)] = value + + def get_url(self) -> str: + return self.url + + def get_username(self) -> str: + return self.username + + def get_password(self) -> str | None: + return self.password + + def get_timestamp(self, url: str, username: str) -> float: + return self.timestamps.get((url, username), 0.0) + + def set_timestamp(self, url: str, username: str, timestamp: float) -> None: + self.timestamps[(url, username)] = timestamp + self.url = url + self.username = username + + def save_credentials( + self, url: str, username: str, password: str, remember_password: bool = True + ) -> None: + self.url = url + self.username = username + self.password = password if remember_password else None + self.remembered[(url, username)] = remember_password + self.timestamps.setdefault((url, username), 0.0) + self.saved.append((url, username, password)) + + +class DirectoryMediaStore: + """Media store resolving paths under a directory owned by the test. + + :param base_dir: Directory that plays the role of the Gramps media path. + """ + + def __init__(self, base_dir: str) -> None: + self.base_dir = base_dir + + def full_path(self, media: Any) -> str: + """Return the absolute path of ``media``'s file.""" + return os.path.join(self.base_dir, media.get_path()) + + def exists(self, media: Any) -> bool: + """Whether ``media``'s file is present on disk.""" + return os.path.exists(self.full_path(media)) + + +class RecordingListener: + """Records state, status and progress updates for later assertions.""" + + def __init__(self) -> None: + #: States entered, in order. + self.states: list[Any] = [] + #: ``(kind, fraction)`` progress updates, in order. + self.progress: list[tuple[str, float]] = [] + #: Status stages reported, in order. + self.statuses: list[str] = [] + + def on_state_changed(self, state) -> None: + self.states.append(state) + + def on_progress(self, kind: str, fraction: float) -> None: + self.progress.append((kind, fraction)) + + def on_status(self, stage: str) -> None: + self.statuses.append(stage) diff --git a/GrampsWebSync/tests/scenario.py b/GrampsWebSync/tests/scenario.py new file mode 100644 index 000000000..fd2ea34b8 --- /dev/null +++ b/GrampsWebSync/tests/scenario.py @@ -0,0 +1,406 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""A small DSL for writing Gramps Web Sync scenarios. + +:class:`SyncScenario` holds a local tree and a remote one derived from it. +Seed the local tree, call :meth:`~SyncScenario.share` to create the remote +side, edit either through its :class:`TreeEditor`, then +:meth:`~SyncScenario.run` a full sync and inspect the :class:`RunResult`:: + + with SyncScenario() as sc: + sc.seed_person("I0001", surname="Doe", changed_at=T0) + sc.share() + sc.local.edit_person("I0001", surname="Müller", changed_at=T2) + sc.remote.edit_person("I0001", surname="Mueller", changed_at=T3) + result = sc.run() + +:data:`T0` to :data:`T3` are increasing timestamps for the ``changed_at`` +argument every mutator takes. +""" + +from __future__ import annotations + +import os +import shutil +import tempfile +from dataclasses import dataclass +from typing import Any + +from const import MODE_BIDIRECTIONAL +from gramps.cli.user import User +from gramps.gen.db import DbTxn +from gramps.gen.db.utils import import_as_dict, make_database +from gramps.gen.lib import Media, Name, Person, Surname, Tag +from session import State, SyncSession + +from .fakes import ( + DirectoryMediaStore, + FakeGrampsWebServer, + FrozenClock, + InlineTaskRunner, + MemoryCredentialStore, + RecordingListener, +) + +#: The server a scenario authenticates against unless told otherwise. Named so +#: that tests asserting on a per-server baseline can name the same entry. +DEFAULT_URL = "https://example.org/api" +DEFAULT_USERNAME = "owner" + +#: A convenient baseline "already synced" time for scenarios. +T0 = 1_600_000_000.0 +#: A time after :data:`T0`, for an edit on one side. +T1 = T0 + 1_000 +#: A time after :data:`T1`, for a later or competing edit. +T2 = T0 + 2_000 +#: A time after :data:`T2`. +T3 = T0 + 3_000 + + +class TreeEditor: + """Mutates one side of a scenario with explicit change timestamps. + + :param db: The database to edit. + :param media_dir: Directory holding this side's media files, if any. + """ + + def __init__(self, db, media_dir: str | None = None) -> None: + self.db = db + self.media_dir = media_dir + + # -------------------------------------------------------- + # Lookup + # -------------------------------------------------------- + def person(self, gramps_id: str) -> Person | None: + """Return the person with ``gramps_id``, or ``None`` if absent.""" + return self.db.get_person_from_gramps_id(gramps_id) + + def surname(self, gramps_id: str) -> str | None: + """Return the primary surname of ``gramps_id``, or ``None`` if absent.""" + person = self.person(gramps_id) + if person is None: + return None + return person.get_primary_name().get_surname() + + def tag(self, name: str) -> Tag | None: + """Return the tag called ``name``, or ``None`` if absent.""" + return self.db.get_tag_from_name(name) + + def person_ids(self) -> set[str]: + """Return every Gramps ID in this tree.""" + return { + self.db.get_person_from_handle(handle).gramps_id + for handle in self.db.get_person_handles() + } + + # -------------------------------------------------------- + # Mutation + # -------------------------------------------------------- + def add_person( + self, + gramps_id: str, + surname: str = "Doe", + first_name: str = "John", + changed_at: float = T0, + ) -> str: + """Add a person and return its handle. + + :param gramps_id: The Gramps ID to assign. + :param surname: Primary surname. + :param first_name: Given name. + :param changed_at: Value to record as the object's change time. + :returns: The new person's handle. + """ + person = Person() + person.set_gramps_id(gramps_id) + name = Name() + name.set_first_name(first_name) + surname_obj = Surname() + surname_obj.set_surname(surname) + name.add_surname(surname_obj) + person.set_primary_name(name) + with DbTxn(f"add {gramps_id}", self.db) as trans: + handle = self.db.add_person(person, trans) + self.db.commit_person(person, trans, changed_at) + return handle + + def edit_person( + self, + gramps_id: str, + surname: str | None = None, + first_name: str | None = None, + changed_at: float = T1, + ) -> None: + """Modify an existing person. + + :param gramps_id: Which person to edit. + :param surname: New primary surname, if given. + :param first_name: New given name, if given. + :param changed_at: Value to record as the object's change time. + :raises LookupError: If no such person exists. + """ + person = self.person(gramps_id) + if person is None: + raise LookupError(f"No person {gramps_id} in this tree") + name = person.get_primary_name() + if surname is not None: + surname_obj = Surname() + surname_obj.set_surname(surname) + name.set_surname_list([surname_obj]) + if first_name is not None: + name.set_first_name(first_name) + person.set_primary_name(name) + with DbTxn(f"edit {gramps_id}", self.db) as trans: + self.db.commit_person(person, trans, changed_at) + + def delete_person(self, gramps_id: str) -> None: + """Remove a person from this tree. + + :param gramps_id: Which person to remove. + :raises LookupError: If no such person exists. + """ + person = self.person(gramps_id) + if person is None: + raise LookupError(f"No person {gramps_id} in this tree") + with DbTxn(f"delete {gramps_id}", self.db) as trans: + self.db.remove_person(person.handle, trans) + + def add_media( + self, + gramps_id: str, + filename: str, + content: bytes = b"fake image bytes", + changed_at: float = T0, + on_disk: bool = True, + ) -> str: + """Add a media object, optionally writing its file. + + :param gramps_id: The Gramps ID to assign. + :param filename: Path relative to the media directory. + :param content: Bytes to write when ``on_disk`` is true. + :param changed_at: Value to record as the object's change time. + :param on_disk: Whether to create the file. ``False`` produces a media + object whose file is missing. + :returns: The new media object's handle. + """ + media = Media() + media.set_gramps_id(gramps_id) + media.set_path(filename) + media.set_description(gramps_id) + with DbTxn(f"add media {gramps_id}", self.db) as trans: + handle = self.db.add_media(media, trans) + self.db.commit_media(media, trans, changed_at) + if on_disk and self.media_dir is not None: + target = os.path.join(self.media_dir, filename) + os.makedirs(os.path.dirname(target), exist_ok=True) + with open(target, "wb") as fobj: + fobj.write(content) + return handle + + def add_tag(self, name: str, changed_at: float = T0) -> str: + """Add a tag and return its handle.""" + tag = Tag() + tag.set_name(name) + with DbTxn(f"add tag {name}", self.db) as trans: + handle = self.db.add_tag(tag, trans) + self.db.commit_tag(tag, trans, changed_at) + return handle + + +@dataclass +class RunResult: + """The outcome of a :meth:`SyncScenario.run`. + + :param states: Every state the session entered, in order. + :param progress: Progress updates as ``(kind, fraction)``. + :param statuses: Status stages reported, in order. + :param session: The session itself, for further assertions. + """ + + states: list[State] + progress: list[tuple[str, float]] + statuses: list[str] + session: SyncSession + + @property + def final_state(self) -> State: + """The state the session ended in.""" + return self.states[-1] if self.states else State.CONNECT + + @property + def error(self): + """The terminal error, if the run failed.""" + return self.session.error + + @property + def login_error(self): + """The recoverable login error, if login was rejected.""" + return self.session.login_error + + def change_ids(self, change_type: str) -> set[str]: + """Return the Gramps IDs reported under a given change type. + + :param change_type: One of the ``C_*`` constants from :mod:`const`. + :returns: The set of Gramps IDs (tag *names*, for tags). + """ + ids = set() + for kind, _handle, class_name, obj1, obj2 in self.session.changes: + if kind != change_type: + continue + obj = obj1 if obj1 is not None else obj2 + ids.add(obj.name if class_name == "Tag" else obj.gramps_id) + return ids + + +class SyncScenario: + """Builds two related trees, then runs a full sync between them. + + Use as a context manager so the databases and temporary directories are + cleaned up:: + + with SyncScenario() as sc: + ... + """ + + def __init__(self, permissions: set[str] | None = None) -> None: + self._tmpdir = tempfile.mkdtemp(prefix="gws_scenario_") + self.local_media_dir = os.path.join(self._tmpdir, "local_media") + os.makedirs(self.local_media_dir, exist_ok=True) + + self.user = User(auto_accept=True, quiet=True) + self.db1 = make_database("sqlite") + self.db1.load(":memory:") + + self.local = TreeEditor(self.db1, media_dir=self.local_media_dir) + #: Set by :meth:`share`; until then there is no remote tree. + self.remote: TreeEditor | None = None + self.server: FakeGrampsWebServer | None = None + self._permissions = permissions + + self.clock = FrozenClock() + self.credentials = MemoryCredentialStore() + self.listener = RecordingListener() + + # -------------------------------------------------------- + # Setup + # -------------------------------------------------------- + def seed_person(self, gramps_id: str, **kwargs: Any) -> str: + """Add a person to the local tree before it is shared.""" + return self.local.add_person(gramps_id, **kwargs) + + def share(self, last_synced: float | None = T0) -> None: + """Create the remote tree as a copy of the local one. + + :param last_synced: Value to record as the last successful sync time. + Pass ``None`` to simulate a first-ever sync. + """ + export_path = os.path.join(self._tmpdir, "seed.gramps") + from gramps.plugins.export.exportxml import export_data + + if not export_data(self.db1, export_path, self.user): + raise RuntimeError("Failed to export the seed tree") + remote_db = import_as_dict(export_path, self.user) + if remote_db is None: + raise RuntimeError("Failed to import the seed tree") + + kwargs: dict[str, Any] = {"db": remote_db} + if self._permissions is not None: + kwargs["permissions"] = self._permissions + self.server = FakeGrampsWebServer(**kwargs) + self.remote = TreeEditor(remote_db) + self.credentials.timestamp = last_synced or 0.0 + + def _require_shared(self) -> FakeGrampsWebServer: + """Return the server, raising a clear error if :meth:`share` was skipped.""" + if self.server is None: + raise RuntimeError("Call share() before running the scenario") + return self.server + + def make_session(self) -> SyncSession: + """Build a :class:`SyncSession` wired to this scenario's fakes.""" + server = self._require_shared() + return SyncSession( + db=self.db1, + user=self.user, + backend_factory=lambda url, username, password: server, + credentials=self.credentials, + media=DirectoryMediaStore(self.local_media_dir), + runner=InlineTaskRunner(), + clock=self.clock, + listener=self.listener, + ) + + # -------------------------------------------------------- + # Running + # -------------------------------------------------------- + def run( + self, + mode: int = MODE_BIDIRECTIONAL, + confirm: bool = True, + transfer_media: bool = True, + url: str = DEFAULT_URL, + username: str = DEFAULT_USERNAME, + password: str = "secret", + ) -> RunResult: + """Drive a complete sync, answering the confirmation. + + Stops early if the session fails or returns to the connect pane. + + :param mode: The sync mode to confirm with. + :param confirm: Whether to accept at all. ``False`` leaves the session + on :attr:`State.REVIEW`. + :param transfer_media: Whether to accept the media transfer along with + the object changes. + :param url: Server URL to submit. + :param username: User name to submit. + :param password: Password to submit. + :returns: A :class:`RunResult` describing the run. + """ + session = self.make_session() + session.submit_credentials(url, username, password) + + if session.state is State.REVIEW and confirm: + session.confirm(mode, transfer_media) + + return RunResult( + states=list(self.listener.states), + progress=list(self.listener.progress), + statuses=list(self.listener.statuses), + session=session, + ) + + # -------------------------------------------------------- + # Lifecycle + # -------------------------------------------------------- + def close(self) -> None: + """Release databases and temporary directories.""" + if self.server is not None: + self.server.close() + self.server = None + try: + self.db1.close() + except Exception: # noqa: BLE001 -- teardown must not mask failures + pass + shutil.rmtree(self._tmpdir, ignore_errors=True) + + def __enter__(self) -> SyncScenario: + return self + + def __exit__(self, *exc_info: Any) -> None: + self.close() diff --git a/GrampsWebSync/tests/test_adapters.py b/GrampsWebSync/tests/test_adapters.py new file mode 100644 index 000000000..2f3ae8365 --- /dev/null +++ b/GrampsWebSync/tests/test_adapters.py @@ -0,0 +1,136 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Tests for the production ports in :mod:`adapters`. + +Drives a real :class:`GLib.MainLoop`; no widgets are built, so no display is +needed. +""" + +from __future__ import annotations + +import threading +import unittest + +from adapters import GLibTaskRunner, IoRunner +from gi.repository import GLib + +#: Milliseconds before an unresponsive loop is torn down. +TIMEOUT_MS = 5000 + + +def run_task(func, runner=None): + """Run ``func`` through a runner and return the outcome. + + :param func: The task to schedule. + :param runner: The runner to use. Defaults to :class:`GLibTaskRunner`. + :returns: Dict with ``result`` or ``error``, ``thread`` and + ``callback_thread``. + """ + outcome: dict = {} + loop = GLib.MainLoop() + + def on_success(result): + outcome["result"] = result + outcome["callback_thread"] = threading.current_thread() + loop.quit() + + def on_error(exc): + outcome["error"] = exc + outcome["callback_thread"] = threading.current_thread() + loop.quit() + + def wrapped(): + outcome["thread"] = threading.current_thread() + return func() + + (runner or GLibTaskRunner()).run(wrapped, on_success, on_error) + GLib.timeout_add(TIMEOUT_MS, loop.quit) + loop.run() + return outcome + + +class GLibTaskRunnerTest(unittest.TestCase): + """The runner must keep work on the thread that owns GTK.""" + + def test_task_runs_on_the_calling_thread(self) -> None: + """Steps drive Gramps progress through the GUI ``User``, which touches + widgets. Running them on a worker thread segfaults inside ``diff_dbs``, + so the runner must not spawn one.""" + outcome = run_task(lambda: "done") + self.assertEqual(outcome.get("result"), "done") + self.assertIs(outcome["thread"], threading.current_thread()) + + def test_success_callback_receives_the_return_value(self) -> None: + self.assertEqual(run_task(lambda: 42).get("result"), 42) + + def test_failure_is_reported_to_the_error_callback(self) -> None: + def boom(): + raise ValueError("boom") + + outcome = run_task(boom) + self.assertNotIn("result", outcome) + self.assertIsInstance(outcome.get("error"), ValueError) + + def test_task_is_run_exactly_once(self) -> None: + """The idle source must remove itself, or it repeats forever.""" + calls = [] + run_task(lambda: calls.append(1)) + self.assertEqual(len(calls), 1) + + +class IoRunnerTest(unittest.TestCase): + """Network steps leave the main loop, but their callbacks come back to it.""" + + def test_task_runs_off_the_calling_thread(self) -> None: + """This is what keeps the window responsive while a request is in + flight, and what makes Cancel work at all.""" + outcome = run_task(lambda: "done", runner=IoRunner()) + self.assertEqual(outcome.get("result"), "done") + self.assertIsNot(outcome["thread"], threading.current_thread()) + + def test_callback_returns_to_the_main_loop(self) -> None: + """Listeners draw widgets, so they must not run on the worker.""" + outcome = run_task(lambda: "done", runner=IoRunner()) + self.assertIs(outcome["callback_thread"], threading.current_thread()) + + def test_failure_is_reported_to_the_error_callback(self) -> None: + def boom(): + raise ValueError("boom") + + outcome = run_task(boom, runner=IoRunner()) + self.assertNotIn("result", outcome) + self.assertIsInstance(outcome.get("error"), ValueError) + + def test_post_runs_on_the_main_loop(self) -> None: + """Progress raised inside a network step is marshalled through this.""" + seen: dict = {} + loop = GLib.MainLoop() + + def note(): + seen["thread"] = threading.current_thread() + loop.quit() + + threading.Thread(target=lambda: IoRunner().post(note)).start() + GLib.timeout_add(TIMEOUT_MS, loop.quit) + loop.run() + self.assertIs(seen.get("thread"), threading.current_thread()) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/tests/test_credentials.py b/GrampsWebSync/tests/test_credentials.py new file mode 100644 index 000000000..ba06b4216 --- /dev/null +++ b/GrampsWebSync/tests/test_credentials.py @@ -0,0 +1,785 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Tests for :class:`adapters.ConfigCredentialStore` and its keyring guard. + +Every store is built against a config manager in a temporary directory, so no +test can write to the user's own Gramps configuration. +""" + +from __future__ import annotations + +import itertools +import os +import shutil +import tempfile +import unittest +import unittest.mock +from typing import cast + +from adapters import ( + LEGACY_TIMESTAMP, + LEGACY_URL, + LEGACY_USERNAME, + ConfigCredentialStore, + Keyring, + normalize_url, + snap_connect_command, +) +from gramps.gen.config import config as configman + +URL = "https://example.org/api" +OTHER = "https://other.example/api" + +#: Config managers are cached by name, so each store needs its own. +_counter = itertools.count() + + +class FakeKeyring: + """A keyring that records calls and can be made to fail.""" + + def __init__(self, fail: Exception | None = None) -> None: + self.stored: dict[tuple[str, str], str] = {} + self.deleted: list[tuple[str, str]] = [] + self.unavailable = None + self._fail = fail + + def get(self, service, username): + return self.stored.get((service, username)) + + def set(self, service, username, password): + if self._fail is not None: + self.unavailable = self._fail + return False + self.stored[(service, username)] = password + return True + + def delete(self, service, username): + self.deleted.append((service, username)) + self.stored.pop((service, username), None) + + +class StoreTestCase(unittest.TestCase): + """Builds isolated stores.""" + + def setUp(self) -> None: + self.tmpdir = tempfile.mkdtemp(prefix="gws_config_") + self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True) + + def make_config(self, name: str | None = None): + """Return a config manager writing into this test's directory. + + The override has to name the ``.ini`` file. Given a bare directory, + Gramps splits it and keeps only the parent, so every test would share + one file in the system temporary directory and inherit whatever a + previous run left in it. + """ + name = name or f"webapisync_test_{next(_counter)}" + return configman.register_manager( + name, os.path.join(self.tmpdir, f"{name}.ini"), use_plugins_path=False + ) + + def make_store( + self, config=None, keyring=None, tree_id: str = "" + ) -> ConfigCredentialStore: + """Return a store over an isolated config manager.""" + return ConfigCredentialStore( + keyring=cast(Keyring, keyring or FakeKeyring()), + config=config or self.make_config(), + tree_id=tree_id, + ) + + +class NormalizeUrlTest(unittest.TestCase): + """The key has to survive the ways people type a URL.""" + + def test_trailing_slash_and_whitespace_are_ignored(self) -> None: + """These used to look like different servers and cost the baseline.""" + self.assertEqual(normalize_url(" https://x.org/ "), "https://x.org") + self.assertEqual(normalize_url("https://x.org///"), "https://x.org") + + +class PerServerBaselineTest(StoreTestCase): + """Each server keeps its own last-sync time.""" + + def test_two_servers_do_not_share_a_baseline(self) -> None: + store = self.make_store() + store.set_timestamp(URL, "owner", 111.0) + store.set_timestamp(OTHER, "owner", 222.0) + self.assertEqual(store.get_timestamp(URL, "owner"), 111.0) + self.assertEqual(store.get_timestamp(OTHER, "owner"), 222.0) + + def test_same_server_different_users_are_separate_trees(self) -> None: + """A Gramps Web account maps to one tree, so the user is part of the key.""" + store = self.make_store() + store.set_timestamp(URL, "alice", 111.0) + store.set_timestamp(URL, "bob", 222.0) + self.assertEqual(store.get_timestamp(URL, "alice"), 111.0) + + def test_a_trailing_slash_does_not_lose_the_baseline(self) -> None: + store = self.make_store() + store.set_timestamp(URL, "owner", 111.0) + self.assertEqual(store.get_timestamp(URL + "/", "owner"), 111.0) + + def test_an_unknown_server_has_no_baseline(self) -> None: + store = self.make_store() + self.assertEqual(store.get_timestamp("https://new.example", "owner"), 0.0) + + +class MigrationTest(StoreTestCase): + """Upgrading from the pre-multi-server layout must lose nothing.""" + + def test_legacy_keys_become_an_entry(self) -> None: + config = self.make_config() + config.register(LEGACY_URL, "") + config.register(LEGACY_USERNAME, "") + config.register(LEGACY_TIMESTAMP, 0) + config.set(LEGACY_URL, URL) + config.set(LEGACY_USERNAME, "owner") + config.set(LEGACY_TIMESTAMP, 999) + + store = self.make_store(config=config) + + self.assertEqual(store.get_url(), URL) + self.assertEqual(store.get_username(), "owner") + self.assertEqual(store.get_timestamp(URL, "owner"), 999.0) + + def test_migration_preserves_the_baseline(self) -> None: + """Losing it would make the first run after upgrading a cold sync.""" + config = self.make_config() + for key, value in ( + (LEGACY_URL, URL), + (LEGACY_USERNAME, "owner"), + (LEGACY_TIMESTAMP, 4242), + ): + config.register(key, "" if isinstance(value, str) else 0) + config.set(key, value) + store = self.make_store(config=config) + self.assertNotEqual(store.get_timestamp(URL, "owner"), 0.0) + + def test_nothing_stored_migrates_to_nothing(self) -> None: + store = self.make_store() + self.assertEqual(store.get_url(), "") + self.assertEqual(store.get_username(), "") + + +class LegacyMirrorTest(StoreTestCase): + """The old keys stay current so a downgrade still works.""" + + def test_saving_mirrors_into_the_legacy_keys(self) -> None: + config = self.make_config() + store = self.make_store(config=config) + store.save_credentials(URL, "owner", "secret") + store.set_timestamp(URL, "owner", 777.0) + + self.assertEqual(config.get(LEGACY_URL), URL) + self.assertEqual(config.get(LEGACY_USERNAME), "owner") + self.assertEqual(config.get(LEGACY_TIMESTAMP), 777) + + def test_a_newer_legacy_baseline_wins_on_re_upgrade(self) -> None: + """An older version may have synced while it was installed.""" + config = self.make_config() + store = self.make_store(config=config) + store.set_timestamp(URL, "owner", 100.0) + # Stand in for an older version syncing and writing only its own keys. + config.set(LEGACY_TIMESTAMP, 500) + config.save() + + reopened = self.make_store(config=config) + + self.assertEqual(reopened.get_timestamp(URL, "owner"), 500.0) + + def test_an_older_legacy_baseline_does_not_regress_the_entry(self) -> None: + config = self.make_config() + store = self.make_store(config=config) + store.set_timestamp(URL, "owner", 500.0) + config.set(LEGACY_TIMESTAMP, 100) + config.save() + + reopened = self.make_store(config=config) + + self.assertEqual(reopened.get_timestamp(URL, "owner"), 500.0) + + def test_an_unreadable_server_list_is_treated_as_empty(self) -> None: + """A value the config manager could not parse is stored as None. + + Registering a default does not help: defaults apply only when a key is + absent, not when it is present and None, so the store has to check the + type rather than assume it. Written to the file directly because + ``set`` type-checks and would reject it. + """ + config = self.make_config() + with open(config.filename, "w", encoding="utf-8") as fobj: + fobj.write("[credentials]\nservers=<< None: + keyring = FakeKeyring() + store = self.make_store(keyring=keyring) + store.save_credentials(URL, "owner", "secret", remember_password=True) + self.assertEqual(keyring.stored[(URL, "owner")], "secret") + + def test_declining_deletes_rather_than_merely_skipping(self) -> None: + """Otherwise the setting appears inert for anyone who had it on.""" + keyring = FakeKeyring() + store = self.make_store(keyring=keyring) + store.save_credentials(URL, "owner", "secret", remember_password=True) + + store.save_credentials(URL, "owner", "secret", remember_password=False) + + self.assertNotIn((URL, "owner"), keyring.stored) + self.assertIn((URL, "owner"), keyring.deleted) + + def test_the_entry_survives_even_when_the_password_does_not(self) -> None: + """The baseline is not a credential; dropping it would force cold syncs.""" + store = self.make_store() + store.set_timestamp(URL, "owner", 321.0) + store.save_credentials(URL, "owner", "secret", remember_password=False) + self.assertEqual(store.get_timestamp(URL, "owner"), 321.0) + + def test_an_unremembered_password_is_not_returned(self) -> None: + store = self.make_store() + store.save_credentials(URL, "owner", "secret", remember_password=False) + self.assertIsNone(store.get_password()) + + +class RememberPasswordChoiceTest(StoreTestCase): + """The stored choice comes back, so the checkbox can show it.""" + + def test_a_server_never_seen_before_defaults_to_remembering(self) -> None: + """Which is what the tool did unconditionally before there was a box.""" + self.assertTrue(self.make_store().get_remember_password()) + + def test_declining_is_remembered(self) -> None: + store = self.make_store() + store.save_credentials(URL, "owner", "secret", remember_password=False) + self.assertFalse(store.get_remember_password()) + + def test_accepting_is_remembered(self) -> None: + store = self.make_store() + store.save_credentials(URL, "owner", "secret", remember_password=True) + self.assertTrue(store.get_remember_password()) + + def test_it_survives_being_reopened(self) -> None: + config = self.make_config() + first = self.make_store(config=config) + first.save_credentials(URL, "owner", "secret", remember_password=False) + + self.assertFalse(self.make_store(config=config).get_remember_password()) + + def test_the_choice_is_per_server(self) -> None: + config = self.make_config() + store = self.make_store(config=config) + store.save_credentials(URL, "owner", "secret", remember_password=False) + store.save_credentials(OTHER, "owner", "secret", remember_password=True) + + self.assertTrue(self.make_store(config=config).get_remember_password()) + + +class ForgetTest(StoreTestCase): + """Forgetting is the wider case of the same delete.""" + + def test_forget_removes_entry_keyring_and_mirror(self) -> None: + config = self.make_config() + keyring = FakeKeyring() + store = self.make_store(config=config, keyring=keyring) + store.save_credentials(URL, "owner", "secret") + + store.forget(URL, "owner") + + self.assertEqual(store.get_url(), "") + self.assertEqual(store.get_timestamp(URL, "owner"), 0.0) + self.assertIn((URL, "owner"), keyring.deleted) + self.assertEqual(config.get(LEGACY_URL), "") + + def test_forget_leaves_other_servers_alone(self) -> None: + store = self.make_store() + store.set_timestamp(URL, "owner", 111.0) + store.set_timestamp(OTHER, "owner", 222.0) + + store.forget(URL, "owner") + + self.assertEqual(store.get_timestamp(OTHER, "owner"), 222.0) + + +class OpenTreeTest(StoreTestCase): + """Entries record the local tree they were synced from. + + Nothing else ties a server to a family tree, and syncing a tree against a + server holding a different one classifies every object as deleted on the + far side, so a bidirectional run proposes emptying both. + """ + + def synced(self, config, tree_id: str, url: str, username: str) -> None: + """Drive one complete sync of ``tree_id`` against ``url``.""" + store = self.make_store(config=config, tree_id=tree_id) + store.save_credentials(url, username, "pw") + store.set_timestamp(url, username, 100.0) + + def test_a_completed_sync_records_the_open_tree(self) -> None: + store = self.make_store(config=self.make_config(), tree_id="tree-a") + store.save_credentials(URL, "owner", "pw") + store.set_timestamp(URL, "owner", 100.0) + + self.assertTrue(store.is_for_open_tree()) + self.assertFalse(store.is_from_another_tree()) + + def test_merely_connecting_claims_nothing(self) -> None: + """Backing out at the review pane must leave no association behind. + + Authenticating against the wrong server is an easy mistake and its own + review pane is where it gets noticed. Claiming the tree at that point + would go on connecting there unprompted, with no warning, because the + entry would say it *is* this tree. + """ + config = self.make_config() + self.make_store(config=config, tree_id="my-tree").save_credentials( + OTHER, "someone", "pw" + ) + + reopened = self.make_store(config=config, tree_id="my-tree") + + self.assertFalse(reopened.is_for_open_tree()) + + def test_the_wrong_server_is_still_offered_for_correcting(self) -> None: + """Refusing to connect unprompted must not also empty the pane.""" + config = self.make_config() + self.make_store(config=config, tree_id="my-tree").save_credentials( + OTHER, "someone", "pw" + ) + + reopened = self.make_store(config=config, tree_id="my-tree") + + self.assertEqual(reopened.get_url(), OTHER) + + def test_moving_a_tree_to_another_server_drops_the_old_claim(self) -> None: + """Otherwise both entries claim the tree and which one wins depends on + the order they happen to sit in.""" + config = self.make_config() + first = self.make_store(config=config, tree_id="tree-a") + first.save_credentials(URL, "owner", "pw") + first.set_timestamp(URL, "owner", 100.0) + + moved = self.make_store(config=config, tree_id="tree-a") + moved.save_credentials(OTHER, "owner", "pw") + moved.set_timestamp(OTHER, "owner", 200.0) + + claims = [ + entry["url"] + for entry in config.get("credentials.servers") + if entry.get("tree_id") == "tree-a" + ] + self.assertEqual(claims, [OTHER]) + + def test_another_tree_may_not_be_connected_to_unprompted(self) -> None: + config = self.make_config() + self.synced(config, "tree-a", URL, "owner") + + reopened = self.make_store(config=config, tree_id="tree-b") + + self.assertFalse(reopened.is_for_open_tree()) + self.assertTrue(reopened.is_from_another_tree()) + + def test_another_tree_still_gets_its_fields_pre_filled(self) -> None: + """Withholding the automatic connect must not also empty the pane.""" + config = self.make_config() + self.synced(config, "tree-a", URL, "owner") + + reopened = self.make_store(config=config, tree_id="tree-b") + + self.assertEqual(reopened.get_url(), URL) + self.assertEqual(reopened.get_username(), "owner") + + def test_each_tree_is_offered_its_own_server(self) -> None: + """Once both are recorded, opening either finds the right one.""" + config = self.make_config() + self.synced(config, "tree-a", URL, "owner") + self.synced(config, "tree-b", OTHER, "other") + + back_to_a = self.make_store(config=config, tree_id="tree-a") + + self.assertEqual(back_to_a.get_url(), URL) + self.assertTrue(back_to_a.is_for_open_tree()) + + def test_the_tree_beats_the_last_used_entry(self) -> None: + """Otherwise reopening a tree would offer whichever server was touched + most recently, which is the hazard being fixed.""" + config = self.make_config() + self.synced(config, "tree-a", URL, "owner") + self.synced(config, "tree-b", OTHER, "other") + + back_to_a = self.make_store(config=config, tree_id="tree-a") + + self.assertEqual(back_to_a.get_url(), URL) + + +class MultiTreeServerTest(StoreTestCase): + """One server hosting several trees, with one account per tree. + + The shape of a hosted Gramps Web deployment: the URL is shared, and only + the account distinguishes one tree from another. Entries are keyed by both, + so nothing here may fall together. + """ + + def setUp(self) -> None: + super().setUp() + self.config = self.make_config() + self.keyring = FakeKeyring() + for tree, user, baseline in ( + ("tree-a", "alice", 1000.0), + ("tree-b", "bob", 2000.0), + ): + store = self.store_for(tree) + store.save_credentials(URL, user, f"pw-{user}") + store.set_timestamp(URL, user, baseline) + + def store_for(self, tree_id: str) -> ConfigCredentialStore: + """Return a store as it would be built with ``tree_id`` open.""" + return self.make_store( + config=self.config, keyring=self.keyring, tree_id=tree_id + ) + + def test_each_tree_is_offered_its_own_account(self) -> None: + self.assertEqual(self.store_for("tree-a").get_username(), "alice") + self.assertEqual(self.store_for("tree-b").get_username(), "bob") + + def test_both_may_connect_unprompted(self) -> None: + self.assertTrue(self.store_for("tree-a").is_for_open_tree()) + self.assertTrue(self.store_for("tree-b").is_for_open_tree()) + + def test_the_tree_wins_over_whichever_was_used_last(self) -> None: + """``tree-b`` synced most recently, so a last-used fallback would + hand ``tree-a`` the wrong account.""" + self.assertEqual(self.store_for("tree-a").get_username(), "alice") + + def test_each_account_keeps_its_own_baseline(self) -> None: + self.assertEqual(self.store_for("tree-a").get_timestamp(URL, "alice"), 1000.0) + self.assertEqual(self.store_for("tree-b").get_timestamp(URL, "bob"), 2000.0) + + def test_passwords_do_not_collide_in_the_keyring(self) -> None: + """The keyring is keyed by service and user, and the service is the + shared URL, so the account has to be what separates them.""" + self.assertEqual(self.store_for("tree-a").get_password(), "pw-alice") + self.assertEqual(self.store_for("tree-b").get_password(), "pw-bob") + + def test_a_third_tree_is_pre_filled_but_not_connected_to(self) -> None: + """Opening an unsynced tree must not sync it against someone else's.""" + store = self.store_for("tree-c") + self.assertFalse(store.is_for_open_tree()) + self.assertTrue(store.is_from_another_tree()) + self.assertEqual(store.get_url(), URL) + + +class ForgetClearsAssociationTest(StoreTestCase): + """Forgetting a server also gives up the tree it had claimed.""" + + def synced(self, config, tree_id: str, url: str, username: str, keyring=None): + """Drive one complete sync of ``tree_id`` against ``url``.""" + store = self.make_store(config=config, keyring=keyring, tree_id=tree_id) + store.save_credentials(url, username, "pw") + store.set_timestamp(url, username, 100.0) + return store + + def test_the_entry_and_its_claim_are_both_gone(self) -> None: + config = self.make_config() + store = self.synced(config, "tree-a", URL, "owner") + + store.forget(URL, "owner") + + self.assertEqual(config.get("credentials.servers"), []) + self.assertFalse(store.is_for_open_tree()) + + def test_the_password_goes_too(self) -> None: + """Leaving it behind would be the setting appearing to do nothing.""" + config = self.make_config() + keyring = FakeKeyring() + store = self.synced(config, "tree-a", URL, "owner", keyring=keyring) + + store.forget(URL, "owner") + + self.assertIn((URL, "owner"), keyring.deleted) + + def test_the_baseline_goes_with_it(self) -> None: + """Which is why forgetting is worth confirming: the next run compares + the two trees from scratch.""" + config = self.make_config() + store = self.synced(config, "tree-a", URL, "owner") + + store.forget(URL, "owner") + + self.assertEqual(store.get_timestamp(URL, "owner"), 0.0) + + def test_another_tree_keeps_its_own_server(self) -> None: + config = self.make_config() + self.synced(config, "tree-a", URL, "alice") + self.synced(config, "tree-b", OTHER, "bob") + + self.make_store(config=config, tree_id="tree-a").forget(URL, "alice") + + still_there = self.make_store(config=config, tree_id="tree-b") + self.assertTrue(still_there.is_for_open_tree()) + self.assertEqual(still_there.get_url(), OTHER) + + +class MovedServerTest(StoreTestCase): + """A tree that moves to another deployment leaves the old one behind.""" + + def move(self, config, url: str, username: str, timestamp: float): + """Sync ``tree-a`` against ``url``.""" + store = self.make_store(config=config, tree_id="tree-a") + store.save_credentials(url, username, "pw") + store.set_timestamp(url, username, timestamp) + return store + + def test_the_new_server_is_the_one_offered(self) -> None: + config = self.make_config() + self.move(config, URL, "owner", 100.0) + + moved = self.move(config, OTHER, "owner", 200.0) + + self.assertEqual(moved.get_url(), OTHER) + self.assertTrue(moved.is_for_open_tree()) + + def test_the_old_server_loses_its_baseline(self) -> None: + """It asserted the tree and that server were identical at a moment + which syncing elsewhere has since made untrue. The comparison takes the + later of the stored baseline and what it computes, so keeping a stale + one can only push the cutoff too far forward -- and too far forward is + where objects stop looking added and start looking deleted. + """ + config = self.make_config() + self.move(config, URL, "owner", 100.0) + + self.move(config, OTHER, "owner", 200.0) + + self.assertEqual( + self.make_store(config=config).get_timestamp(URL, "owner"), 0.0 + ) + + def test_the_old_entry_itself_survives(self) -> None: + """Only the claim and the baseline go; the address and user name are + still worth offering if the user goes back.""" + config = self.make_config() + self.move(config, URL, "owner", 100.0) + + self.move(config, OTHER, "owner", 200.0) + + urls = [entry["url"] for entry in config.get("credentials.servers")] + self.assertIn(URL, urls) + + +class UpgradedEntryTest(StoreTestCase): + """An entry stored before tree ids existed must keep working.""" + + def make_pre_upgrade_entry(self, config) -> None: + """Store an entry the way the previous version did, with no tree id.""" + store = self.make_store(config=config) + store.save_credentials(URL, "owner", "pw") + store.set_timestamp(URL, "owner", 1234.0) + + def test_it_is_still_offered(self) -> None: + config = self.make_config() + self.make_pre_upgrade_entry(config) + + store = self.make_store(config=config, tree_id="tree-a") + + self.assertEqual(store.get_url(), URL) + self.assertEqual(store.get_username(), "owner") + + def test_its_baseline_survives(self) -> None: + """Losing it would turn the next run into a full cold resync.""" + config = self.make_config() + self.make_pre_upgrade_entry(config) + + store = self.make_store(config=config, tree_id="tree-a") + + self.assertEqual(store.get_timestamp(URL, "owner"), 1234.0) + + def test_it_does_not_connect_unprompted_until_it_has_synced_once(self) -> None: + """No tree was recorded, so which one it belongs to is simply unknown.""" + config = self.make_config() + self.make_pre_upgrade_entry(config) + + store = self.make_store(config=config, tree_id="tree-a") + + self.assertFalse(store.is_for_open_tree()) + + def test_nor_is_it_reported_as_belonging_elsewhere(self) -> None: + """An unknown tree is not a different tree; warning would be a guess.""" + config = self.make_config() + self.make_pre_upgrade_entry(config) + + store = self.make_store(config=config, tree_id="tree-a") + + self.assertFalse(store.is_from_another_tree()) + + def test_the_first_completed_sync_adopts_the_tree(self) -> None: + config = self.make_config() + self.make_pre_upgrade_entry(config) + + store = self.make_store(config=config, tree_id="tree-a") + store.save_credentials(URL, "owner", "pw") + store.set_timestamp(URL, "owner", 5678.0) + + self.assertTrue(store.is_for_open_tree()) + self.assertEqual(store.get_timestamp(URL, "owner"), 5678.0) + + def test_a_tool_that_cannot_identify_the_tree_never_auto_connects(self) -> None: + """``get_dbid`` returning nothing must fail safe, not fail open.""" + config = self.make_config() + store_a = self.make_store(config=config, tree_id="tree-a") + store_a.save_credentials(URL, "owner", "pw") + store_a.set_timestamp(URL, "owner", 100.0) + + store = self.make_store(config=config, tree_id="") + + self.assertFalse(store.is_for_open_tree()) + self.assertFalse(store.is_from_another_tree()) + self.assertEqual(store.get_url(), URL) + + +class ExplodingBackend: + """A keyring backend that raises, as one does under snap confinement.""" + + def __init__(self, exc: Exception) -> None: + self.exc = exc + self.calls: list[str] = [] + + def get_password(self, *_args): + self.calls.append("get") + raise self.exc + + def set_password(self, *_args): + self.calls.append("set") + raise self.exc + + def delete_password(self, *_args): + self.calls.append("delete") + raise self.exc + + +class KeyringOverBackend(Keyring): + """The real guard logic over a backend the test controls.""" + + def __init__(self, backend: ExplodingBackend) -> None: + super().__init__() + self.backend = backend + + def _module(self): + return None if self.unavailable is not None else self.backend + + +class KeyringGuardTest(unittest.TestCase): + """A broken keyring must not take Gramps down with it.""" + + def test_a_backend_raising_is_reported_not_propagated(self) -> None: + """Under snap confinement this arrives as a jeepney DBusErrorResponse. + + That does not derive from ``keyring.errors``, because it comes from a + transitive dependency of the backend, so guarding on the keyring + package's own exception hierarchy would not catch it. + """ + + class DBusErrorResponse(Exception): + pass + + keyring = KeyringOverBackend( + ExplodingBackend(DBusErrorResponse("An AppArmor policy prevents...")) + ) + + self.assertIsNone(keyring.get("svc", "user")) + problem = keyring.unavailable + self.assertIsNotNone(problem) + assert problem is not None # for the type checker + self.assertIn("AppArmor", problem.detail) + + def test_a_write_failure_is_reported_as_not_stored(self) -> None: + keyring = KeyringOverBackend(ExplodingBackend(RuntimeError("denied"))) + self.assertFalse(keyring.set("svc", "user", "pw")) + self.assertIsNotNone(keyring.unavailable) + + def test_a_failure_stops_further_attempts(self) -> None: + """One denial is enough; retrying each call just repeats the stall.""" + backend = ExplodingBackend(RuntimeError("denied")) + keyring = KeyringOverBackend(backend) + + keyring.set("svc", "user", "pw") + keyring.set("svc", "user", "pw") + keyring.get("svc", "user") + + self.assertEqual(backend.calls, ["set"]) + + def test_deleting_a_missing_entry_is_not_a_failure(self) -> None: + """Backends raise when asked to delete something that is not there. + + A working keyring still reads cleanly, which is how that is told apart + from a keyring that cannot delete because it is broken. + """ + + class AbsentEntryBackend(ExplodingBackend): + def get_password(self, *_args): + return None + + keyring = KeyringOverBackend(AbsentEntryBackend(RuntimeError("no such item"))) + self.assertTrue(keyring.delete("svc", "nobody")) + self.assertIsNone(keyring.unavailable) + + def test_a_delete_that_leaves_the_password_behind_is_a_failure(self) -> None: + """Otherwise turning off "remember password" silently does nothing.""" + + class StubbornBackend(ExplodingBackend): + def get_password(self, *_args): + return "still here" + + keyring = KeyringOverBackend(StubbornBackend(RuntimeError("denied"))) + self.assertFalse(keyring.delete("svc", "user")) + self.assertIsNotNone(keyring.unavailable) + + +class SnapHintTest(unittest.TestCase): + """Under snap the failure is a setting the user can change.""" + + def test_no_command_outside_snap(self) -> None: + with unittest.mock.patch.dict("os.environ", {}, clear=True): + self.assertIsNone(snap_connect_command()) + + def test_the_instance_name_is_used_when_present(self) -> None: + """A parallel install is named gramps_foo, and the command must match.""" + env = {"SNAP": "/snap/gramps/11", "SNAP_INSTANCE_NAME": "gramps_beta"} + with unittest.mock.patch.dict("os.environ", env, clear=True): + command = snap_connect_command() or "" + self.assertIn("gramps_beta:password-manager-service", command) + + def test_it_falls_back_to_the_snap_name(self) -> None: + env = {"SNAP": "/snap/gramps/11", "SNAP_NAME": "gramps"} + with unittest.mock.patch.dict("os.environ", env, clear=True): + self.assertEqual( + snap_connect_command(), "snap connect gramps:password-manager-service" + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/tests/test_errors.py b/GrampsWebSync/tests/test_errors.py new file mode 100644 index 000000000..6e91e9df8 --- /dev/null +++ b/GrampsWebSync/tests/test_errors.py @@ -0,0 +1,465 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Failure handling, exercised by injecting faults into the fake server.""" + +from __future__ import annotations + +import unittest +from urllib.error import URLError + +from const import API_MAJOR, MIN_API_VERSION, MIN_API_VERSION_TEXT +from session import ErrorKind, State, api_version_problem + +from .fakes import http_error +from .scenario import ( + DEFAULT_URL as URL, + DEFAULT_USERNAME as USERNAME, + T0, + T2, + SyncScenario, +) + + +class ApiVersionTest(unittest.TestCase): + """A server too old to sync with says so, at connect time.""" + + def make_scenario(self, **kwargs) -> SyncScenario: + scenario = SyncScenario(**kwargs) + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.share() + return scenario + + def test_a_version_below_the_minimum_is_refused(self) -> None: + scenario = self.make_scenario() + scenario.server.api_version = f"{MIN_API_VERSION[0] - 1}.9" + result = scenario.run() + self.assertIs(result.final_state, State.CONNECT) + self.assertIs(result.login_error.kind, ErrorKind.SERVER_TOO_OLD) + + def test_the_reported_version_is_carried_into_the_message(self) -> None: + scenario = self.make_scenario() + scenario.server.api_version = "1.2.3" + result = scenario.run() + self.assertEqual(result.login_error.detail, "1.2.3") + + def test_a_server_reporting_no_version_is_refused(self) -> None: + """The field postdates the endpoints this addon needs.""" + scenario = self.make_scenario() + scenario.server.api_version = None + result = scenario.run() + self.assertIs(result.login_error.kind, ErrorKind.SERVER_TOO_OLD) + + def test_the_version_is_checked_before_the_permissions(self) -> None: + """An API too old to report a permission claim yields an empty set, + which would otherwise be blamed on the user's account settings.""" + scenario = self.make_scenario(permissions=set()) + scenario.server.api_version = "1.0.0" + result = scenario.run() + self.assertIs(result.login_error.kind, ErrorKind.SERVER_TOO_OLD) + + def test_an_outdated_server_is_never_downloaded_from(self) -> None: + scenario = self.make_scenario() + scenario.server.api_version = "1.0.0" + scenario.run() + self.assertNotIn("download_xml", scenario.server.calls) + + def test_a_server_from_the_future_is_refused_too(self) -> None: + """And pointed at Gramps, since a later API pairs with a later Gramps + rather than with a later build of this addon.""" + scenario = self.make_scenario() + scenario.server.api_version = f"{API_MAJOR + 1}.1.0" + + result = scenario.run() + + self.assertIs(result.final_state, State.CONNECT) + self.assertIs(result.login_error.kind, ErrorKind.SERVER_TOO_NEW) + self.assertNotIn("download_xml", scenario.server.calls) + + def test_a_rejected_server_leaves_its_version_on_screen(self) -> None: + """The message names that version, so the footer showing it agrees.""" + scenario = self.make_scenario() + scenario.server.api_version = "1.0.0" + + result = scenario.run() + + self.assertEqual(result.session.api_version, "1.0.0") + + def test_but_a_second_attempt_does_not_inherit_it(self) -> None: + """The context strip titles itself with the tree name and the footer + with the version, so carrying either forward labels the new attempt + with the previous server's identity.""" + scenario = self.make_scenario() + scenario.server.api_version = "1.0.0" + session = scenario.make_session() + session.submit_credentials(URL, USERNAME, "secret") + self.assertEqual(session.api_version, "1.0.0") + + scenario.server.fail_always("get_api_version", URLError("no route")) + session.submit_credentials("https://elsewhere.example/api", "other", "pw") + + self.assertIs(session.state, State.CONNECT) + self.assertIsNone(session.api_version) + self.assertEqual(session.tree_name, "") + + def test_a_current_server_is_accepted(self) -> None: + scenario = self.make_scenario() + result = scenario.run() + self.assertIsNone(result.login_error) + + +class TaskQueueTest(unittest.TestCase): + """A server with no background task queue cannot be synced with.""" + + def make_scenario(self, **kwargs) -> SyncScenario: + scenario = SyncScenario(**kwargs) + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.share() + return scenario + + def test_a_server_without_one_is_refused(self) -> None: + """Transactions run synchronously there, and time out on any real + tree, which surfaces as an arbitrary failure part-way through.""" + scenario = self.make_scenario() + scenario.server.task_queue = False + + result = scenario.run() + + self.assertIs(result.final_state, State.CONNECT) + self.assertIs(result.login_error.kind, ErrorKind.SERVER_NO_TASK_QUEUE) + + def test_it_is_refused_before_anything_is_downloaded(self) -> None: + scenario = self.make_scenario() + scenario.server.task_queue = False + + scenario.run() + + self.assertNotIn("download_xml", scenario.server.calls) + + def test_the_version_is_reported_first_when_both_are_wrong(self) -> None: + """An outdated server is the more actionable of the two, and may not + report the queue flag at all.""" + scenario = self.make_scenario() + scenario.server.api_version = "1.0.0" + scenario.server.task_queue = False + + result = scenario.run() + + self.assertIs(result.login_error.kind, ErrorKind.SERVER_TOO_OLD) + + def test_the_queue_is_checked_before_the_permissions(self) -> None: + """What the server is configured to do is not the user's account's + fault, and telling them to fix their account would misdirect.""" + scenario = self.make_scenario(permissions=set()) + scenario.server.task_queue = False + + result = scenario.run() + + self.assertIs(result.login_error.kind, ErrorKind.SERVER_NO_TASK_QUEUE) + + def test_a_configured_server_is_accepted(self) -> None: + scenario = self.make_scenario() + result = scenario.run() + self.assertIsNone(result.login_error) + + +class ApiVersionRangeTest(unittest.TestCase): + """The version comparison itself, at both ends.""" + + def test_the_minimum_itself_is_accepted(self) -> None: + self.assertIsNone(api_version_problem(f"{MIN_API_VERSION_TEXT}.0")) + + def test_a_later_minor_of_the_same_major_is_accepted(self) -> None: + self.assertIsNone(api_version_problem(f"{MIN_API_VERSION[0]}.99")) + + def test_a_prerelease_suffix_is_ignored(self) -> None: + self.assertIsNone(api_version_problem(f"{MIN_API_VERSION_TEXT}.0-beta1")) + + def test_an_earlier_version_is_too_old(self) -> None: + self.assertIs( + api_version_problem(f"{MIN_API_VERSION[0] - 1}.9"), + ErrorKind.SERVER_TOO_OLD, + ) + + def test_the_next_major_is_too_new(self) -> None: + """This branch of the addon speaks one API major, the one its Gramps + release line pairs with.""" + self.assertIs( + api_version_problem(f"{API_MAJOR + 1}.0"), ErrorKind.SERVER_TOO_NEW + ) + + def test_a_much_later_major_is_too_new(self) -> None: + self.assertIs( + api_version_problem(f"{API_MAJOR + 5}.2"), ErrorKind.SERVER_TOO_NEW + ) + + def test_the_supported_major_is_not_derived_from_the_minimum(self) -> None: + """The two are declared separately on purpose, so raising the minimum + within a major cannot silently move the upper bound with it.""" + self.assertEqual(MIN_API_VERSION[0], API_MAJOR) + + def test_nothing_at_all_counts_as_too_old(self) -> None: + """The field predates neither bound, so a server that cannot report + one is far likelier to be ancient than to be from the future.""" + self.assertIs(api_version_problem(None), ErrorKind.SERVER_TOO_OLD) + self.assertIs(api_version_problem(""), ErrorKind.SERVER_TOO_OLD) + + def test_an_unreadable_version_is_rejected_rather_than_raising(self) -> None: + """Whatever a misbehaving server sends must not reach the user as a + traceback.""" + self.assertIs( + api_version_problem("not a version"), ErrorKind.SERVER_TOO_OLD + ) + + +class LoginFailureTest(unittest.TestCase): + """Authentication and reachability problems at connect time.""" + + def make_scenario(self, **kwargs) -> SyncScenario: + scenario = SyncScenario(**kwargs) + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.share() + return scenario + + def test_http_statuses_map_to_distinct_error_kinds(self) -> None: + """Each status the server can return is reported as its own kind.""" + cases = [ + (401, ErrorKind.AUTH_FAILED), + (403, ErrorKind.FORBIDDEN), + (404, ErrorKind.NOT_FOUND), + (429, ErrorKind.RATE_LIMITED), + (503, ErrorKind.TREE_DISABLED), + (500, ErrorKind.SERVER_ERROR), + ] + for code, expected in cases: + with self.subTest(code=code): + scenario = self.make_scenario() + scenario.server.fail_always("get_permissions", http_error(code)) + result = scenario.run() + self.assertIs(result.final_state, State.CONNECT) + self.assertIsNotNone(result.login_error) + self.assertIs(result.login_error.kind, expected) + + def test_login_failure_is_recoverable_not_terminal(self) -> None: + """A rejected login must not set the terminal error.""" + scenario = self.make_scenario() + scenario.server.fail_always("get_permissions", http_error(401)) + result = scenario.run() + self.assertIsNone(result.error) + self.assertIs(result.final_state, State.CONNECT) + + def test_unreachable_server_reports_a_connection_failure(self) -> None: + scenario = self.make_scenario() + scenario.server.fail_always("get_permissions", URLError("no route to host")) + result = scenario.run() + self.assertIs(result.login_error.kind, ErrorKind.CONNECTION_FAILED) + + def test_non_api_response_reports_an_invalid_response(self) -> None: + """Something answered, but it was not the Gramps Web API.""" + scenario = self.make_scenario() + scenario.server.fail_always("get_permissions", ValueError("not JSON")) + result = scenario.run() + self.assertIs(result.login_error.kind, ErrorKind.INVALID_RESPONSE) + + def test_user_without_required_permission_is_refused(self) -> None: + """Without ViewPrivate the export is partial, so sync must not start.""" + scenario = self.make_scenario(permissions={"ViewObject"}) + result = scenario.run() + self.assertIs(result.final_state, State.CONNECT) + self.assertIs(result.login_error.kind, ErrorKind.INSUFFICIENT_PERMISSIONS) + + def test_failed_login_does_not_touch_either_tree(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.server.fail_always("get_permissions", http_error(401)) + + scenario.run() + + self.assertEqual(scenario.remote.surname("I0001"), "Doe") + self.assertEqual(scenario.server.committed, []) + + +class MidSyncFailureTest(unittest.TestCase): + """Failures after the connection is established are terminal.""" + + def make_scenario(self) -> SyncScenario: + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.seed_person("I0002", surname="Roe", changed_at=T0) + scenario.share() + return scenario + + def test_export_download_failure_fails_the_run(self) -> None: + scenario = self.make_scenario() + scenario.server.fail_always("download_xml", http_error(500)) + result = scenario.run() + self.assertIs(result.final_state, State.FAILED) + self.assertIs(result.error.kind, ErrorKind.SERVER_ERROR) + + def test_transaction_conflict_is_reported_as_a_conflict(self) -> None: + """HTTP 409 means the server rejected the transaction as stale.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.server.fail_always("commit", http_error(409)) + + result = scenario.run() + + self.assertIs(result.final_state, State.FAILED) + self.assertIs(result.error.kind, ErrorKind.CONFLICT) + + def test_expired_token_mid_sync_is_reported_as_auth_failure(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.server.fail_always("commit", http_error(401)) + result = scenario.run() + self.assertIs(result.error.kind, ErrorKind.AUTH_FAILED) + + def test_failed_run_does_not_record_a_sync_timestamp(self) -> None: + """Recording it would move the diff cutoff past the unsynced changes.""" + scenario = self.make_scenario() + scenario.credentials.timestamp = T0 + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.server.fail_always("commit", http_error(409)) + + scenario.run() + + self.assertEqual(scenario.credentials.get_timestamp(URL, USERNAME), T0) + + def test_local_changes_survive_a_failed_remote_commit(self) -> None: + """The local half commits before the remote half is even attempted.""" + scenario = self.make_scenario() + scenario.remote.edit_person("I0002", surname="Neu", changed_at=T2) + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + scenario.server.fail_always("commit", http_error(500)) + + result = scenario.run() + + self.assertIs(result.final_state, State.FAILED) + self.assertEqual(scenario.local.surname("I0002"), "Neu") + + +class CancellationTest(unittest.TestCase): + """Cancelling must stop work that has been scheduled but not yet run.""" + + def make_scenario(self) -> SyncScenario: + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.share() + return scenario + + def test_cancel_before_compare_skips_the_download(self) -> None: + scenario = self.make_scenario() + session = scenario.make_session() + session.cancel() + + session._fetch_xml() + + self.assertNotIn("download_xml", scenario.server.calls) + + def test_cancel_before_apply_sends_nothing(self) -> None: + """``cancel`` releases the diff handler, so applying must not proceed.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + session = scenario.make_session() + session.submit_credentials("https://example.org/api", "owner", "secret") + session.cancel() + + session._apply_local() + + self.assertEqual(scenario.server.committed, []) + + def test_cancel_before_transfer_moves_no_files(self) -> None: + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.local.add_media("O0001", "photo.jpg", changed_at=T0) + scenario.share() + session = scenario.make_session() + session.submit_credentials("https://example.org/api", "owner", "secret") + session.cancel() + + session._transfer(*session._resolve_transfers()) + + self.assertEqual(scenario.server.media_files, {}) + + +class MediaFailureTest(unittest.TestCase): + """One bad media file must not abort the whole transfer.""" + + def make_scenario(self) -> SyncScenario: + scenario = SyncScenario() + self.addCleanup(scenario.close) + return scenario + + def test_failed_download_is_recorded_without_failing_the_run(self) -> None: + """A single unreadable file is recorded and the run continues.""" + scenario = self.make_scenario() + first = scenario.local.add_media("O0001", "first.jpg", on_disk=False) + second = scenario.local.add_media("O0002", "second.jpg", on_disk=False) + scenario.share() + # The server holds both files, so nothing needs uploading and this + # test isolates the download path. + scenario.server.media_files[first] = b"first bytes" + scenario.server.media_files[second] = b"second bytes" + scenario.server.fail_next("download_media_file", http_error(500)) + + result = scenario.run() + + self.assertIs(result.final_state, State.DONE) + self.assertEqual(result.session.downloaded, {"O0001": False, "O0002": True}) + + def test_file_missing_on_both_sides_is_reported_once(self) -> None: + """Neither side can supply such a file, so neither transfer is tried. + + It used to appear in both missing lists, so the download 404d and the + upload found nothing to send, and the user was told of two errors for + one file that simply does not exist anywhere. + """ + scenario = self.make_scenario() + scenario.local.add_media("O0001", "nowhere.jpg", on_disk=False) + scenario.share() + + result = scenario.run() + + self.assertIs(result.final_state, State.DONE) + self.assertIsNone(result.error) + self.assertEqual([gid for gid, _h in result.session.missing_both], ["O0001"]) + self.assertEqual(result.session.missing_local, []) + self.assertEqual(result.session.missing_remote, []) + self.assertEqual(result.session.downloaded, {}) + self.assertEqual(result.session.uploaded, {}) + + def test_upload_still_fails_the_run_on_a_server_error(self) -> None: + """The new guard must not swallow genuine transport failures.""" + scenario = self.make_scenario() + scenario.local.add_media("O0002", "present.jpg", changed_at=T0) + scenario.share() + scenario.server.fail_always("upload_media_file", http_error(500)) + + result = scenario.run() + + self.assertIs(result.final_state, State.FAILED) + self.assertIs(result.error.kind, ErrorKind.SERVER_ERROR) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/tests/test_presentation.py b/GrampsWebSync/tests/test_presentation.py new file mode 100644 index 000000000..f510e37a1 --- /dev/null +++ b/GrampsWebSync/tests/test_presentation.py @@ -0,0 +1,419 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Unit tests for :mod:`presentation`, the interface's non-GTK half. + +Constructs no widgets, so no display is needed. +""" + +from __future__ import annotations + +import unittest +from types import SimpleNamespace + +import presentation +from adapters import KeyringUnavailable +from const import ( + A_ADD_LOC, + A_ADD_REM, + A_DEL_LOC, + A_DEL_REM, + A_MRG_REM, + A_UPD_REM, + MIN_API_VERSION_TEXT, +) +from gramps.gen.lib import Name, Person, Surname, Tag +from presentation import ( + LOCAL, + REMOTE, + VERB_ADD, + VERB_MERGE, + VERB_UPDATE, + build_review, + context_lines, + deletion_warning, + describe_object, + error_message, + format_last_synced, + is_insecure, + keyring_message, + object_id, + outcome_summary, + sanitize_url, +) +from session import ErrorKind, State + + +def person(gramps_id: str, first: str = "John", last: str = "Smith") -> Person: + """Build a person carrying a primary name, for description tests.""" + obj = Person() + obj.set_gramps_id(gramps_id) + name = Name() + name.set_first_name(first) + surname = Surname() + surname.set_surname(last) + name.add_surname(surname) + obj.set_primary_name(name) + return obj + + +def action(kind: str, obj=None, obj2=None, obj_type: str = "Person"): + """Build one action tuple in the shape :func:`build_review` consumes.""" + return (kind, "handle", obj_type, obj, obj2) + + +class ErrorMessageTest(unittest.TestCase): + """Every error the session can record must render as something readable.""" + + def test_every_error_kind_has_a_message(self) -> None: + """An unmapped kind would surface to the user as an empty dialog.""" + for kind in ErrorKind: + with self.subTest(kind=kind.name): + message = error_message(kind, "42") + self.assertTrue(message.strip(), f"{kind.name} rendered empty") + + def test_detail_is_included_where_it_carries_information(self) -> None: + """Status codes must reach the user for the otherwise-opaque kinds.""" + self.assertIn("42", error_message(ErrorKind.SERVER_ERROR, "42")) + self.assertIn("boom", error_message(ErrorKind.UNEXPECTED, "boom")) + + def test_a_failed_server_task_reports_what_the_server_said(self) -> None: + """This used to render a stringified status dict plus advice to check + the connection, which was neither true nor actionable.""" + message = error_message(ErrorKind.SERVER_TASK_FAILED, "disk full") + self.assertIn("disk full", message) + self.assertNotIn("connection", message.lower()) + + def test_an_outdated_server_is_told_both_versions(self) -> None: + message = error_message(ErrorKind.SERVER_TOO_OLD, "2.4.1") + self.assertIn("2.4.1", message) + self.assertIn(MIN_API_VERSION_TEXT, message) + + def test_the_two_version_bounds_advise_opposite_remedies(self) -> None: + """Too old means update the server. Too new means move to a newer + Gramps: an API major pairs with a Gramps release line, so no build of + this addon will ever speak the next one. Sending someone after the + wrong one has them hunting for something that does not exist.""" + too_old = error_message(ErrorKind.SERVER_TOO_OLD, "1.0.0") + too_new = error_message(ErrorKind.SERVER_TOO_NEW, "9.0.0") + self.assertIn("update the server", too_old.lower()) + self.assertIn("newer version of gramps", too_new.lower()) + self.assertNotIn("update the server", too_new.lower()) + self.assertNotIn("update the addon", too_new.lower()) + + def test_the_version_found_is_named_at_both_ends(self) -> None: + self.assertIn("1.0.0", error_message(ErrorKind.SERVER_TOO_OLD, "1.0.0")) + self.assertIn("9.0.0", error_message(ErrorKind.SERVER_TOO_NEW, "9.0.0")) + + def test_a_server_reporting_no_version_still_gets_a_message(self) -> None: + """The detail is empty in that case, so the general branch must cope.""" + message = error_message(ErrorKind.SERVER_TOO_OLD) + self.assertTrue(message.strip()) + self.assertIn(MIN_API_VERSION_TEXT, message) + + +class KeyringMessageTest(unittest.TestCase): + """An unusable keyring is reported, and under snap it is fixable.""" + + def test_the_snap_command_is_included_when_there_is_one(self) -> None: + problem = KeyringUnavailable( + "denied", snap_command="snap connect gramps:password-manager-service" + ) + self.assertIn("snap connect gramps", keyring_message(problem)) + + def test_elsewhere_the_message_says_what_to_expect_instead(self) -> None: + message = keyring_message(KeyringUnavailable("no backend")) + self.assertNotIn("snap", message.lower()) + self.assertTrue(message.strip()) + + +class UrlTest(unittest.TestCase): + """Completing and judging what the user typed into the URL entry.""" + + def test_a_bare_host_gets_https(self) -> None: + self.assertEqual( + sanitize_url("example.org/api"), "https://example.org/api" + ) + + def test_surrounding_whitespace_is_dropped(self) -> None: + self.assertEqual(sanitize_url(" example.org "), "https://example.org") + + def test_an_explicit_scheme_is_left_alone(self) -> None: + """Including http: the user is warned, not overruled.""" + self.assertEqual(sanitize_url("http://localhost:5000"), "http://localhost:5000") + self.assertEqual(sanitize_url("https://example.org"), "https://example.org") + + def test_an_empty_entry_stays_empty(self) -> None: + """Otherwise the entry would fill itself in with a bare scheme.""" + self.assertEqual(sanitize_url(" "), "") + + def test_only_http_counts_as_insecure(self) -> None: + self.assertTrue(is_insecure("http://example.org")) + self.assertFalse(is_insecure("https://example.org")) + self.assertFalse(is_insecure("example.org")) + + +class DescribeObjectTest(unittest.TestCase): + """Rows say what an object is, not only that one exists.""" + + def test_a_person_is_described_by_name(self) -> None: + name = describe_object(person("I0001"), "Person") + self.assertIn("Smith", name) + self.assertIn("John", name) + + def test_a_tag_is_described_by_its_name_and_has_no_id(self) -> None: + """Tags carry no Gramps ID, so the ID column must stay empty for them.""" + tag = Tag() + tag.set_name("Needs sources") + self.assertEqual(describe_object(tag, "Tag"), "Needs sources") + self.assertEqual(object_id(tag, "Tag"), "") + + def test_a_class_needing_a_database_degrades_rather_than_raising(self) -> None: + """Objects only the remote tree has may arrive without one.""" + self.assertEqual(describe_object(Person(), "Family"), "") + + def test_a_missing_object_is_not_an_error(self) -> None: + self.assertEqual(describe_object(None, "Person"), "") + + +class ReviewModelTest(unittest.TestCase): + """Actions are grouped by the database they change, not by what differs.""" + + def test_actions_are_filed_under_their_destination(self) -> None: + model = build_review( + [ + action(A_ADD_LOC, obj2=person("I0001")), + action(A_UPD_REM, obj=person("I0002")), + ] + ) + self.assertEqual([d.where for d in model.destinations], [LOCAL, REMOTE]) + self.assertEqual(model.destinations[0].groups[0].verb, VERB_ADD) + self.assertEqual(model.destinations[1].groups[0].verb, VERB_UPDATE) + + def test_a_merge_appears_under_both_destinations(self) -> None: + """It writes the combined object to each, so reporting it once would + understate what happens to one of the trees.""" + model = build_review([action(A_MRG_REM, obj=person("I0001"))]) + self.assertEqual([d.where for d in model.destinations], [LOCAL, REMOTE]) + for destination in model.destinations: + self.assertEqual(destination.groups[0].verb, VERB_MERGE) + self.assertEqual(destination.count, 1) + + def test_a_destination_nothing_happens_to_is_left_out(self) -> None: + model = build_review([action(A_ADD_REM, obj=person("I0001"))]) + self.assertEqual([d.where for d in model.destinations], [REMOTE]) + + def test_deletions_are_counted_per_side(self) -> None: + model = build_review( + [ + action(A_DEL_LOC, obj2=person("I0001")), + action(A_DEL_REM, obj=person("I0002")), + action(A_DEL_REM, obj=person("I0003")), + ] + ) + self.assertEqual(model.local_deletions, 1) + self.assertEqual(model.remote_deletions, 2) + self.assertTrue(model.deletes) + + def test_rows_are_sorted_so_the_list_does_not_reshuffle(self) -> None: + """The actions arrive in dictionary order, which is not stable enough + to show the same user the same list twice.""" + model = build_review( + [ + action(A_ADD_REM, obj=person("I0003", last="Zeta")), + action(A_ADD_REM, obj=person("I0001", last="Alpha")), + action(A_ADD_REM, obj=person("I0002", last="Mu")), + ] + ) + names = [row.name for row in model.destinations[0].groups[0].rows] + self.assertEqual(names, sorted(names)) + + def test_an_unknown_action_is_skipped_rather_than_crashing_the_pane(self) -> None: + model = build_review([action("no_such_action", obj=person("I0001"))]) + self.assertEqual(model.destinations, ()) + + def test_no_actions_produce_an_empty_model(self) -> None: + model = build_review([]) + self.assertEqual(model.destinations, ()) + self.assertFalse(model.deletes) + + +class DeletionWarningTest(unittest.TestCase): + """What will be removed is stated before it happens.""" + + def test_nothing_deleted_means_no_warning(self) -> None: + model = build_review([action(A_ADD_REM, obj=person("I0001"))]) + self.assertEqual(deletion_warning(model), "") + + def test_both_sides_are_named_when_both_lose_objects(self) -> None: + model = build_review( + [ + action(A_DEL_LOC, obj2=person("I0001")), + action(A_DEL_REM, obj=person("I0002")), + ] + ) + warning = deletion_warning(model) + self.assertIn("computer", warning) + self.assertIn("server", warning) + + def test_a_bidirectional_run_that_deletes_is_warned_about_too(self) -> None: + """The warning is derived from the actions, not from the mode, because + propagating a deletion destroys data just as surely as a reset does.""" + model = build_review([action(A_DEL_REM, obj=person("I0001"))]) + self.assertTrue(deletion_warning(model)) + + +class OutcomeSummaryTest(unittest.TestCase): + """The final report covers both databases and the media files.""" + + @staticmethod + def session(**kwargs): + """Build a stand-in exposing what :func:`outcome_summary` reads.""" + defaults = { + "actions": [], + "downloaded": {}, + "uploaded": {}, + "missing_both": [], + "error": None, + } + defaults.update(kwargs) + return SimpleNamespace(**defaults) + + def test_object_changes_are_reported_not_only_media(self) -> None: + """A run applying hundreds of changes and no media used to report + "Media files are in sync." and nothing else.""" + summary = outcome_summary(self.session(actions=[1, 2, 3])) + self.assertIn("3", summary) + + def test_an_untouched_pair_of_trees_says_so(self) -> None: + """The commonest outcome of all, and the one that used to be described + purely in terms of media files.""" + summary = outcome_summary(self.session()) + self.assertIn("trees", summary.lower()) + + def test_the_trees_are_reported_before_the_media(self) -> None: + """Whatever else happened, the databases are what the user came for.""" + summary = outcome_summary(self.session()) + self.assertLess( + summary.lower().index("trees"), summary.lower().index("media") + ) + + def test_a_media_only_run_still_reports_the_trees(self) -> None: + summary = outcome_summary(self.session(uploaded={"O1": True})) + self.assertIn("trees", summary.lower()) + self.assertIn("1", summary) + + def test_partial_progress_survives_a_failure(self) -> None: + """A connection lost after two of three uploads must still report the + two, rather than showing the error alone.""" + summary = outcome_summary( + self.session( + error=object(), + uploaded={"O1": True, "O2": True, "O3": False}, + ) + ) + self.assertIn("2", summary) + self.assertIn("1", summary) + + def test_files_missing_on_both_sides_are_called_out(self) -> None: + summary = outcome_summary(self.session(missing_both=[("O1", "h1")])) + self.assertIn("both", summary.lower()) + + def test_a_failure_does_not_claim_media_are_in_sync(self) -> None: + """Nothing was checked, so asserting it would be a guess.""" + summary = outcome_summary(self.session(error=object())) + self.assertNotIn("in sync", summary) + + +class ContextLinesTest(unittest.TestCase): + """The strip has to answer "which tree am I about to write to?".""" + + def test_the_tree_name_becomes_the_heading_once_known(self) -> None: + """It is the only thing that distinguishes two trees on a hosted + deployment, where the address is shared and only the account differs.""" + title, subtitle = context_lines( + "https://hub.example/api", "alice", "Smith Family", "Last synced today" + ) + self.assertEqual(title, "Smith Family") + self.assertIn("alice", subtitle) + self.assertIn("hub.example", subtitle) + self.assertIn("Last synced today", subtitle) + + def test_before_connecting_the_account_leads(self) -> None: + """The server has not said what it calls its tree yet.""" + title, subtitle = context_lines( + "https://hub.example/api", "alice", "", "Never synced" + ) + self.assertIn("alice", title) + self.assertIn("hub.example", title) + self.assertEqual(subtitle, "Never synced") + + def test_a_named_tree_never_synced_still_says_where_it_is(self) -> None: + title, subtitle = context_lines( + "https://hub.example/api", "alice", "Smith Family", "" + ) + self.assertEqual(title, "Smith Family") + self.assertIn("alice", subtitle) + + def test_nothing_configured_says_so(self) -> None: + title, subtitle = context_lines("", "", "", "Never synced") + self.assertTrue(title.strip()) + self.assertEqual(subtitle, "") + + def test_a_half_configured_server_is_not_presented_as_real(self) -> None: + self.assertEqual( + context_lines("https://hub.example/api", ""), context_lines("", "") + ) + + +class LastSyncedTest(unittest.TestCase): + """The context strip says how current the baseline is.""" + + def test_never_synced_is_said_plainly(self) -> None: + self.assertTrue(format_last_synced(0).strip()) + + def test_recent_and_distant_syncs_read_differently(self) -> None: + now = 1_700_000_000.0 + recent = format_last_synced(now - 120, now) + older = format_last_synced(now - 3 * 3600, now) + self.assertNotEqual(recent, older) + self.assertIn("2", recent) + self.assertIn("3", older) + + def test_a_baseline_in_the_future_does_not_produce_nonsense(self) -> None: + """Clock skew between the two machines is entirely possible.""" + now = 1_700_000_000.0 + self.assertTrue(format_last_synced(now + 5000, now).strip()) + + +class PhaseLabelTest(unittest.TestCase): + """Every phase the working pane lists must have a name.""" + + def test_each_working_state_is_named(self) -> None: + from session import WORKING_STATES + + for state in WORKING_STATES: + with self.subTest(state=state.name): + self.assertTrue(presentation.state_label(state).strip()) + + def test_a_state_that_is_not_a_phase_has_no_label(self) -> None: + self.assertEqual(presentation.state_label(State.REVIEW), "") + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/tests/test_recovery.py b/GrampsWebSync/tests/test_recovery.py new file mode 100644 index 000000000..02d3ee48d --- /dev/null +++ b/GrampsWebSync/tests/test_recovery.py @@ -0,0 +1,340 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Recovering from a failed run, and refusing to commit a stale comparison.""" + +from __future__ import annotations + +import unittest + +from const import MODE_BIDIRECTIONAL +from gramps.gen.db import DbTxn +from session import ErrorKind, State, Step, SyncSession + +from .fakes import http_error +from .scenario import ( + DEFAULT_URL as URL, + DEFAULT_USERNAME as USERNAME, + T0, + T2, + SyncScenario, +) + + +class RecoveryTestCase(unittest.TestCase): + """Drives a session by hand, so a run can be interrupted mid-flow.""" + + def make_scenario(self) -> SyncScenario: + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.seed_person("I0002", surname="Roe", changed_at=T0) + scenario.share() + return scenario + + def connect(self, scenario: SyncScenario) -> SyncSession: + """Return a session that has connected and compared.""" + session = scenario.make_session() + session.submit_credentials(URL, USERNAME, "secret") + return session + + +class RetryTest(RecoveryTestCase): + """A failed run resumes where it stopped rather than starting over.""" + + def test_no_retry_is_offered_without_a_failure(self) -> None: + scenario = self.make_scenario() + session = self.connect(scenario) + self.assertFalse(session.can_retry) + + def test_a_failed_push_can_be_retried(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + scenario.server.fail_next("commit", http_error(500)) + session = self.connect(scenario) + + session.confirm(MODE_BIDIRECTIONAL) + self.assertIs(session.state, State.FAILED) + self.assertIs(session.failed_in, Step.PUSH_REMOTE) + + session.retry() + + self.assertIsNone(session.error) + self.assertEqual(scenario.remote.surname("I0001"), "Mueller") + + def test_retrying_a_push_does_not_re_apply_the_local_half(self) -> None: + """Resuming at the failed step is the whole point of tracking it.""" + scenario = self.make_scenario() + scenario.remote.edit_person("I0002", surname="Neu", changed_at=T2) + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + scenario.server.fail_next("commit", http_error(500)) + session = self.connect(scenario) + + session.confirm(MODE_BIDIRECTIONAL) + session.retry() + + # One payload reached the server: the retry re-sent, it did not stack a + # second local transaction on top of the first. + self.assertEqual(len(scenario.server.committed), 1) + self.assertEqual(scenario.local.surname("I0002"), "Neu") + + def test_retrying_a_push_does_not_download_the_tree_again(self) -> None: + """The remote database is kept open precisely so this is cheap.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + scenario.server.fail_next("commit", http_error(500)) + session = self.connect(scenario) + + session.confirm(MODE_BIDIRECTIONAL) + session.retry() + + self.assertEqual(scenario.server.calls.count("download_xml"), 1) + + def test_a_failed_download_is_retried_from_the_start(self) -> None: + scenario = self.make_scenario() + scenario.server.fail_next("download_xml", http_error(500)) + session = scenario.make_session() + session.submit_credentials(URL, USERNAME, "secret") + + self.assertIs(session.state, State.FAILED) + self.assertIs(session.failed_in, Step.FETCH) + + session.retry() + + self.assertIsNone(session.error) + self.assertEqual(scenario.server.calls.count("download_xml"), 2) + + def test_a_failed_transfer_resumes_with_the_remaining_files(self) -> None: + """Files already moved are not sent twice.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.local.add_media("O0001", "one.jpg", changed_at=T0) + scenario.local.add_media("O0002", "two.jpg", changed_at=T0) + scenario.share() + session = self.connect(scenario) + self.assertIs(session.state, State.REVIEW) + + scenario.server.fail_next("upload_media_file", http_error(500)) + session.confirm(MODE_BIDIRECTIONAL) + self.assertIs(session.state, State.FAILED) + + session.retry() + + self.assertIs(session.state, State.DONE) + self.assertEqual(session.uploaded, {"O0001": True, "O0002": True}) + self.assertEqual(len(scenario.server.media_files), 2) + + def test_retry_without_a_recorded_step_does_nothing(self) -> None: + scenario = self.make_scenario() + session = self.connect(scenario) + session.failed_in = None + session.retry() # must not raise + self.assertIsNone(session.error) + + +class MediaScanTest(RecoveryTestCase): + """Asking the server which files it lacks is a network step of its own.""" + + def test_a_failed_scan_is_retryable_as_its_own_step(self) -> None: + """It used to run inline on the main loop, freezing the UI while it ran.""" + scenario = self.make_scenario() + scenario.server.fail_next("get_missing_files", http_error(500)) + session = self.connect(scenario) + + self.assertIs(session.state, State.FAILED) + self.assertIs(session.failed_in, Step.SCAN_MEDIA) + + session.retry() + + self.assertIs(session.state, State.DONE) + self.assertIsNone(session.error) + + +class StaleComparisonTest(RecoveryTestCase): + """Edits made while the review page is open must not be overwritten. + + The comparison captures object snapshots and the tool does not block the + main window, so the user can keep editing. Committing those snapshots would + silently discard whatever they did in the meantime. + """ + + def test_an_edit_during_review_stops_the_commit(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + self.assertIs(session.state, State.REVIEW) + + scenario.local.edit_person("I0001", surname="Later", changed_at=T2 + 10) + session.confirm(MODE_BIDIRECTIONAL) + + self.assertIs(session.state, State.FAILED) + self.assertIs(session.error.kind, ErrorKind.STALE_LOCAL_DATA) + + def test_nothing_is_sent_when_the_comparison_is_stale(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + + scenario.local.edit_person("I0001", surname="Later", changed_at=T2 + 10) + session.confirm(MODE_BIDIRECTIONAL) + + self.assertEqual(scenario.server.committed, []) + self.assertEqual(scenario.local.surname("I0001"), "Later") + + def test_a_deletion_during_review_is_caught(self) -> None: + """A delete is as destructive as an edit and must be caught too.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + + scenario.local.delete_person("I0001") + session.confirm(MODE_BIDIRECTIONAL) + + self.assertIs(session.error.kind, ErrorKind.STALE_LOCAL_DATA) + + def test_an_object_appearing_locally_is_caught(self) -> None: + """The action was 'add here', which now would collide with real data.""" + scenario = self.make_scenario() + scenario.remote.add_person("I0003", surname="Nieuw", changed_at=T2) + session = self.connect(scenario) + self.assertIs(session.state, State.REVIEW) + + person = scenario.remote.person("I0003") + with DbTxn("local add", scenario.db1) as trans: + scenario.db1.add_person(person, trans) + + session.confirm(MODE_BIDIRECTIONAL) + + self.assertIs(session.error.kind, ErrorKind.STALE_LOCAL_DATA) + + def test_retry_after_a_stale_comparison_compares_again(self) -> None: + """The snapshots are worthless, so resuming the commit is not an option.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + + scenario.local.edit_person("I0001", surname="Later", changed_at=T2 + 10) + session.confirm(MODE_BIDIRECTIONAL) + self.assertIs(session.failed_in, Step.DIFF) + + session.retry() + + self.assertEqual(scenario.server.calls.count("download_xml"), 2) + self.assertIsNone(session.error) + + def test_an_untouched_tree_commits_normally(self) -> None: + """The guard must not fire on a run where nothing changed underneath.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + + session.confirm(MODE_BIDIRECTIONAL) + + self.assertIsNone(session.error) + self.assertEqual(scenario.remote.surname("I0001"), "Mueller") + + +if __name__ == "__main__": + unittest.main() + + +class AbandonTest(RecoveryTestCase): + """Switching server mid-run, without closing the tool. + + Auto-connect makes this necessary: a tree whose server has moved would + otherwise sit through a whole download of the old one before the user + could reach the connect pane. + """ + + def test_a_comparison_can_be_walked_away_from(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + self.assertIs(session.state, State.REVIEW) + + session.abandon() + + self.assertIs(session.state, State.CONNECT) + self.assertEqual(session.changes, []) + + def test_the_remote_tree_is_released(self) -> None: + """It is an in-memory database; keeping it would leak one per switch.""" + session = self.connect(self.make_scenario()) + + session.abandon() + + self.assertIsNone(session.db2) + + def test_a_previous_failure_is_cleared(self) -> None: + scenario = self.make_scenario() + scenario.server.fail_next("download_xml", http_error(500)) + session = scenario.make_session() + session.submit_credentials(URL, USERNAME, "secret") + self.assertIs(session.state, State.FAILED) + + session.abandon() + + self.assertIs(session.state, State.CONNECT) + self.assertIsNone(session.error) + self.assertFalse(session.can_retry) + + def test_the_session_still_works_afterwards(self) -> None: + scenario = self.make_scenario() + session = self.connect(scenario) + session.abandon() + + session.submit_credentials(URL, USERNAME, "secret") + + self.assertIn(session.state, (State.REVIEW, State.DONE)) + self.assertIsNone(session.error) + + def test_the_connection_is_forgotten_with_the_run(self) -> None: + """The context strip and the version footer are rendered from these, + so leaving them set describes the server just walked away from.""" + session = self.connect(self.make_scenario()) + + session.abandon() + + self.assertEqual(session.url, "") + self.assertEqual(session.username, "") + self.assertEqual(session.password, "") + self.assertEqual(session.tree_name, "") + self.assertIsNone(session.api_version) + self.assertIsNone(session.backend) + + def test_a_run_that_has_started_writing_is_not_abandoned(self) -> None: + """Walking away mid-apply would leave no record of what got through.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Mueller", changed_at=T2) + session = self.connect(scenario) + session.state = State.APPLYING + + session.abandon() + + self.assertIs(session.state, State.APPLYING) + + def test_a_callback_from_the_abandoned_run_is_dropped(self) -> None: + """The step it belongs to is already in flight and cannot be recalled, + so its result must not drive the session that replaced it.""" + session = self.connect(self.make_scenario()) + stale = session._guarded(lambda _result: session._goto(State.DONE)) + + session.abandon() + stale(None) + + self.assertIs(session.state, State.CONNECT) diff --git a/GrampsWebSync/tests/test_sync_flow.py b/GrampsWebSync/tests/test_sync_flow.py new file mode 100644 index 000000000..2fb64f6a3 --- /dev/null +++ b/GrampsWebSync/tests/test_sync_flow.py @@ -0,0 +1,423 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""End-to-end sync runs against an in-process fake Gramps Web server.""" + +from __future__ import annotations + +import os +import unittest + +from const import ( + C_ADD_LOC, + C_ADD_REM, + C_DEL_LOC, + C_DEL_REM, + C_UPD_BOTH, + C_UPD_LOC, + C_UPD_REM, + MODE_BIDIRECTIONAL, + MODE_RESET_TO_LOCAL, + MODE_RESET_TO_REMOTE, +) +from session import ( + STATUS_COMPARING, + STATUS_FETCHING, + STATUS_LOCAL_APPLIED, + State, +) + +from .scenario import ( + DEFAULT_URL as URL, + DEFAULT_USERNAME as USERNAME, + T0, + T2, + T3, + SyncScenario, +) + + +class SyncFlowTestCase(unittest.TestCase): + """Base class providing a seeded, shared two-tree scenario.""" + + def make_scenario(self, **kwargs) -> SyncScenario: + """Return a shared scenario with two people, registered for teardown.""" + scenario = SyncScenario(**kwargs) + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.seed_person("I0002", surname="Roe", changed_at=T0) + scenario.share() + return scenario + + +class InSyncTest(SyncFlowTestCase): + """Two identical trees.""" + + def test_identical_trees_report_no_changes(self) -> None: + """A tree exported and reimported must diff as unchanged.""" + scenario = self.make_scenario() + result = scenario.run() + self.assertEqual(result.session.changes, []) + self.assertIs(result.final_state, State.DONE) + + def test_confirmation_stage_is_skipped(self) -> None: + """With nothing to confirm, the flow bypasses the review pane.""" + scenario = self.make_scenario() + result = scenario.run() + self.assertNotIn(State.REVIEW, result.states) + self.assertNotIn(State.APPLYING, result.states) + + def test_nothing_is_sent_to_the_server(self) -> None: + """An in-sync run must not post a transaction.""" + scenario = self.make_scenario() + scenario.run() + self.assertEqual(scenario.server.committed, []) + + def test_a_run_with_nothing_to_do_goes_straight_to_the_result(self) -> None: + """No pane may be shown with nothing on it but an Apply button.""" + scenario = self.make_scenario() + result = scenario.run() + self.assertNotIn(State.REVIEW, result.states) + self.assertEqual( + result.states, [State.CONNECTING, State.COMPARING, State.DONE] + ) + + +class BidirectionalSyncTest(SyncFlowTestCase): + """Changes made on one side only, propagating to the other.""" + + def test_local_edit_reaches_the_server(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + result = scenario.run() + self.assertEqual(result.change_ids(C_UPD_LOC), {"I0001"}) + self.assertEqual(scenario.remote.surname("I0001"), "Müller") + + def test_remote_edit_reaches_the_local_tree(self) -> None: + scenario = self.make_scenario() + scenario.remote.edit_person("I0001", surname="Mueller", changed_at=T2) + result = scenario.run() + self.assertEqual(result.change_ids(C_UPD_REM), {"I0001"}) + self.assertEqual(scenario.local.surname("I0001"), "Mueller") + + def test_local_addition_reaches_the_server(self) -> None: + scenario = self.make_scenario() + scenario.local.add_person("I9001", surname="Neu", changed_at=T2) + result = scenario.run() + self.assertEqual(result.change_ids(C_ADD_LOC), {"I9001"}) + self.assertIn("I9001", scenario.remote.person_ids()) + + def test_remote_addition_reaches_the_local_tree(self) -> None: + scenario = self.make_scenario() + scenario.remote.add_person("I9002", surname="Neuer", changed_at=T2) + result = scenario.run() + self.assertEqual(result.change_ids(C_ADD_REM), {"I9002"}) + self.assertIn("I9002", scenario.local.person_ids()) + + def test_local_deletion_reaches_the_server(self) -> None: + scenario = self.make_scenario() + scenario.local.delete_person("I0002") + result = scenario.run() + self.assertEqual(result.change_ids(C_DEL_LOC), {"I0002"}) + self.assertNotIn("I0002", scenario.remote.person_ids()) + + def test_remote_deletion_reaches_the_local_tree(self) -> None: + scenario = self.make_scenario() + scenario.remote.delete_person("I0002") + result = scenario.run() + self.assertEqual(result.change_ids(C_DEL_REM), {"I0002"}) + self.assertNotIn("I0002", scenario.local.person_ids()) + + def test_edits_on_both_sides_are_flagged_as_simultaneous(self) -> None: + """Competing edits to one object are reported as simultaneous.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.remote.edit_person("I0001", surname="Mueller", changed_at=T3) + result = scenario.run() + self.assertEqual(result.change_ids(C_UPD_BOTH), {"I0001"}) + + def test_independent_changes_on_both_sides_both_propagate(self) -> None: + """Each side's change lands on the other in a single run.""" + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.remote.add_person("I9003", surname="Neu", changed_at=T2) + scenario.run() + self.assertEqual(scenario.remote.surname("I0001"), "Müller") + self.assertIn("I9003", scenario.local.person_ids()) + + +class SyncModeTest(SyncFlowTestCase): + """The four sync modes resolve the same divergence differently.""" + + def diverged(self) -> SyncScenario: + """Return a scenario where each side added a distinct person.""" + scenario = self.make_scenario() + scenario.local.add_person("I9100", surname="LocalOnly", changed_at=T2) + scenario.remote.add_person("I9200", surname="RemoteOnly", changed_at=T2) + return scenario + + def test_bidirectional_keeps_both_additions(self) -> None: + scenario = self.diverged() + scenario.run(mode=MODE_BIDIRECTIONAL) + for side in (scenario.local, scenario.remote): + self.assertIn("I9100", side.person_ids()) + self.assertIn("I9200", side.person_ids()) + + def test_reset_to_local_makes_the_server_match_the_local_tree(self) -> None: + scenario = self.diverged() + scenario.run(mode=MODE_RESET_TO_LOCAL) + self.assertIn("I9100", scenario.remote.person_ids()) + self.assertNotIn("I9200", scenario.remote.person_ids()) + self.assertNotIn("I9200", scenario.local.person_ids()) + + def test_reset_to_remote_makes_the_local_tree_match_the_server(self) -> None: + scenario = self.diverged() + scenario.run(mode=MODE_RESET_TO_REMOTE) + self.assertIn("I9200", scenario.local.person_ids()) + self.assertNotIn("I9100", scenario.local.person_ids()) + self.assertNotIn("I9100", scenario.remote.person_ids()) + + def test_an_unknown_mode_is_rejected(self) -> None: + """Modes are a closed set; an unknown one must not sync silently.""" + scenario = self.make_scenario() + scenario.local.delete_person("I0002") + result = scenario.run(mode=99) + self.assertIs(result.final_state, State.FAILED) + + +class MediaFileTest(SyncFlowTestCase): + """Media files, which sync separately from object data.""" + + def test_file_missing_locally_is_downloaded(self) -> None: + scenario = SyncScenario() + self.addCleanup(scenario.close) + handle = scenario.local.add_media( + "O0001", "photo.jpg", changed_at=T0, on_disk=False + ) + scenario.share() + scenario.server.media_files[handle] = b"server image bytes" + + result = scenario.run() + + self.assertIs(result.final_state, State.DONE) + self.assertEqual(result.session.downloaded, {"O0001": True}) + media = scenario.db1.get_media_from_handle(handle) + local_path = os.path.join(scenario.local_media_dir, media.get_path()) + self.assertTrue(os.path.exists(local_path)) + with open(local_path, "rb") as fobj: + self.assertEqual(fobj.read(), b"server image bytes") + + def test_file_missing_remotely_is_uploaded(self) -> None: + scenario = SyncScenario() + self.addCleanup(scenario.close) + handle = scenario.local.add_media( + "O0002", "portrait.jpg", content=b"local bytes", changed_at=T0 + ) + scenario.share() + self.assertNotIn(handle, scenario.server.media_files) + + result = scenario.run() + + self.assertEqual(result.session.uploaded, {"O0002": True}) + self.assertEqual(scenario.server.media_files[handle], b"local bytes") + + def test_the_review_knows_what_will_be_transferred(self) -> None: + """The media scan runs before the review, not after the apply, so the + checkbox can state the counts rather than promising the unknown.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.local.add_media("O0010", "up.jpg", changed_at=T0) + scenario.share() + + session = scenario.make_session() + session.submit_credentials(URL, USERNAME, "secret") + + self.assertIs(session.state, State.REVIEW) + self.assertEqual([gid for gid, _h in session.missing_remote], ["O0010"]) + + def test_media_arriving_with_the_sync_is_still_transferred(self) -> None: + """A media object added by the sync has no file on the receiving side, + which the scan taken before the review cannot have seen.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.share() + handle = scenario.remote.add_media("O0011", "new.jpg", changed_at=T2) + scenario.server.media_files[handle] = b"server image bytes" + + result = scenario.run() + + self.assertIs(result.final_state, State.DONE) + self.assertEqual(result.session.downloaded, {"O0011": True}) + media = scenario.db1.get_media_from_handle(handle) + local_path = os.path.join(scenario.local_media_dir, media.get_path()) + with open(local_path, "rb") as fobj: + self.assertEqual(fobj.read(), b"server image bytes") + + def test_a_file_neither_side_holds_is_never_offered_as_an_upload(self) -> None: + """The server has the object but no file, and this tree has never had + the object at all, so there is nothing here to send. Counting it as an + upload promised a transfer that cannot happen, and the run then + contradicted itself at the end by reporting it missing on both sides. + """ + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.share() + scenario.remote.add_media("O0020", "ghost.jpg", changed_at=T2) + + session = scenario.make_session() + session.submit_credentials(URL, USERNAME, "secret") + + self.assertEqual([gid for gid, _h in session.missing_remote], []) + self.assertEqual([gid for gid, _h in session.missing_both], ["O0020"]) + + def test_it_is_reported_once_at_the_end_too(self) -> None: + """Same run, carried through: the count the review showed has to be + the count the summary reports.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.share() + scenario.remote.add_media("O0021", "ghost.jpg", changed_at=T2) + + result = scenario.run() + + self.assertIs(result.final_state, State.DONE) + self.assertEqual(len(result.session.missing_both), 1) + self.assertEqual(result.session.uploaded, {}) + + def test_a_run_touching_no_media_object_scans_only_once(self) -> None: + """The second scan exists for the case above; making every run pay for + it would be a round trip for nothing.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.seed_person("I0001", surname="Doe", changed_at=T0) + scenario.share() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + + scenario.run() + + self.assertEqual(scenario.server.calls.count("get_missing_files"), 1) + + def test_transfer_stage_is_skipped_when_all_files_are_present(self) -> None: + scenario = SyncScenario() + self.addCleanup(scenario.close) + handle = scenario.local.add_media("O0003", "ok.jpg", changed_at=T0) + scenario.share() + scenario.server.media_files[handle] = b"fake image bytes" + + result = scenario.run() + + self.assertNotIn(State.TRANSFERRING, result.states) + self.assertIs(result.final_state, State.DONE) + + def test_leaving_the_review_unanswered_transfers_nothing(self) -> None: + """Until the user confirms, the run must neither advance nor fail.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.local.add_media("O0004", "skipped.jpg", changed_at=T0) + scenario.share() + + result = scenario.run(confirm=False) + + self.assertIs(result.final_state, State.REVIEW) + self.assertEqual(scenario.server.media_files, {}) + + def test_unchecking_the_media_box_skips_the_transfer(self) -> None: + """The box is the only control over media now that the second + confirmation page is gone, so it has to actually govern.""" + scenario = SyncScenario() + self.addCleanup(scenario.close) + scenario.local.add_media("O0005", "declined.jpg", changed_at=T0) + scenario.share() + + result = scenario.run(transfer_media=False) + + self.assertIs(result.final_state, State.DONE) + self.assertNotIn(State.TRANSFERRING, result.states) + self.assertEqual(scenario.server.media_files, {}) + + +class TimestampTest(SyncFlowTestCase): + """The last-sync timestamp, which drives every later diff.""" + + def test_successful_run_records_the_sync_time(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + scenario.clock.time = 1_700_000_500.0 + + scenario.run() + + self.assertEqual(scenario.credentials.get_timestamp(URL, USERNAME), 1_700_000_500.0) + + def test_in_sync_run_also_records_the_sync_time(self) -> None: + """Finding no differences still counts as a successful sync.""" + scenario = self.make_scenario() + scenario.clock.time = 1_700_000_900.0 + + scenario.run() + + self.assertEqual( + scenario.credentials.get_timestamp(URL, USERNAME), 1_700_000_900.0 + ) + + def test_each_server_keeps_its_own_baseline(self) -> None: + """Syncing elsewhere must not disturb this server's baseline. + + The baseline used to be a single value that a changed URL reset, so + alternating between two servers discarded it every time and made every + run after a switch a full "modified in both" comparison. + """ + scenario = self.make_scenario() + scenario.credentials.timestamps[(URL, USERNAME)] = T3 + scenario.run(url="https://elsewhere.example/api") + self.assertEqual(scenario.credentials.get_timestamp(URL, USERNAME), T3) + + +class ProgressTest(SyncFlowTestCase): + """Progress and status reporting reach the listener.""" + + def test_applying_changes_reports_api_progress(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + result = scenario.run() + api_progress = [f for kind, f in result.progress if kind == "api"] + self.assertTrue(api_progress) + self.assertEqual(api_progress[-1], 1.0) + + def test_comparison_reports_fetching_then_comparing(self) -> None: + scenario = self.make_scenario() + result = scenario.run() + self.assertEqual( + [s for s in result.statuses if s in (STATUS_FETCHING, STATUS_COMPARING)], + [STATUS_FETCHING, STATUS_COMPARING], + ) + + def test_local_commit_is_reported(self) -> None: + scenario = self.make_scenario() + scenario.remote.edit_person("I0001", surname="Mueller", changed_at=T2) + result = scenario.run() + self.assertIn(STATUS_LOCAL_APPLIED, result.statuses) + + def test_remote_only_changes_do_not_report_a_local_commit(self) -> None: + scenario = self.make_scenario() + scenario.local.edit_person("I0001", surname="Müller", changed_at=T2) + result = scenario.run() + self.assertNotIn(STATUS_LOCAL_APPLIED, result.statuses) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/tests/test_transitions.py b/GrampsWebSync/tests/test_transitions.py new file mode 100644 index 000000000..0601914ec --- /dev/null +++ b/GrampsWebSync/tests/test_transitions.py @@ -0,0 +1,136 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Unit tests for :func:`session.next_state`.""" + +from __future__ import annotations + +import unittest +from types import SimpleNamespace + +from session import ErrorKind, State, SyncError, next_state + + +def fake_session( + error: SyncError | None = None, + changes: list | None = None, + has_missing_files: bool = False, + transfer_media: bool = True, +) -> SimpleNamespace: + """Build a stand-in exposing the attributes :func:`next_state` reads. + + :param error: A terminal error, if any. + :param changes: The pending change list. + :param has_missing_files: Whether media files are missing on either side. + :param transfer_media: Whether the confirmed run should move media files. + :returns: The stand-in session. + """ + changes = changes if changes is not None else [] + return SimpleNamespace( + error=error, + changes=changes, + has_missing_files=has_missing_files, + has_review_content=bool(changes or has_missing_files), + will_transfer=transfer_media and has_missing_files, + ) + + +class NextStateTest(unittest.TestCase): + """The happy-path chain and its two conditional skips.""" + + def test_linear_path_with_changes_and_files(self) -> None: + """With both changes and missing files, every state is visited.""" + session = fake_session(changes=["a change"], has_missing_files=True) + expected = [ + (State.CONNECT, State.CONNECTING), + (State.CONNECTING, State.COMPARING), + (State.COMPARING, State.REVIEW), + (State.REVIEW, State.APPLYING), + (State.APPLYING, State.TRANSFERRING), + (State.TRANSFERRING, State.DONE), + ] + for state, following in expected: + with self.subTest(state=state.name): + self.assertIs(next_state(state, session), following) + + def test_media_alone_is_worth_reviewing(self) -> None: + """Transferring files is a decision even when no object changed.""" + session = fake_session(changes=[], has_missing_files=True) + self.assertIs(next_state(State.COMPARING, session), State.REVIEW) + + def test_nothing_to_do_at_all_ends_the_run(self) -> None: + """Fully in sync: no confirmation of any kind is shown.""" + session = fake_session(changes=[], has_missing_files=False) + self.assertIs(next_state(State.COMPARING, session), State.DONE) + + def test_applying_with_no_missing_files_ends_the_run(self) -> None: + """The transfer stage is skipped rather than entered with nothing to do.""" + session = fake_session(changes=["a change"], has_missing_files=False) + self.assertIs(next_state(State.APPLYING, session), State.DONE) + + def test_changes_present_requires_confirmation(self) -> None: + """Any pending change must be confirmed before it is applied.""" + session = fake_session(changes=["a change"]) + self.assertIs(next_state(State.COMPARING, session), State.REVIEW) + + def test_declining_the_media_transfer_skips_it(self) -> None: + """Unchecking the box on the review pane must actually prevent it.""" + session = fake_session( + changes=["a change"], has_missing_files=True, transfer_media=False + ) + self.assertIs(next_state(State.APPLYING, session), State.DONE) + + def test_missing_files_requires_transfer(self) -> None: + """Missing media on either side means a transfer stage.""" + session = fake_session(has_missing_files=True) + self.assertIs(next_state(State.APPLYING, session), State.TRANSFERRING) + + +class ErrorShortCircuitTest(unittest.TestCase): + """A recorded error overrides the flow from wherever it happened.""" + + def test_error_from_any_state_goes_to_failed(self) -> None: + """Every non-terminal state jumps to FAILED once an error is set.""" + session = fake_session( + error=SyncError(ErrorKind.CONFLICT), changes=["a change"] + ) + for state in State: + if state in (State.DONE, State.FAILED): + continue + with self.subTest(state=state.name): + self.assertIs(next_state(state, session), State.FAILED) + + def test_error_outranks_the_skip_conditions(self) -> None: + """The error check runs before any branch that might route elsewhere.""" + session = fake_session(error=SyncError(ErrorKind.AUTH_FAILED), changes=[]) + self.assertIs(next_state(State.COMPARING, session), State.FAILED) + + +class TerminalStateTest(unittest.TestCase): + """Terminal states do not advance on their own.""" + + def test_done_is_terminal(self) -> None: + self.assertIs(next_state(State.DONE, fake_session()), State.DONE) + + def test_failed_is_terminal(self) -> None: + session = fake_session(error=SyncError(ErrorKind.UNEXPECTED)) + self.assertIs(next_state(State.FAILED, session), State.FAILED) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/tests/test_view_mapping.py b/GrampsWebSync/tests/test_view_mapping.py new file mode 100644 index 000000000..01a2de7b5 --- /dev/null +++ b/GrampsWebSync/tests/test_view_mapping.py @@ -0,0 +1,69 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +"""Checks that the GTK view's lookup tables cover the session's enums. + +Constructs no widgets, so no display is needed. +""" + +from __future__ import annotations + +import unittest + +from grampswebsync import ( + PANE_CONNECT, + PANE_FOR_STATE, + PANE_RESULT, + PANE_REVIEW, + PANE_WORKING, +) +from session import State, WORKING_STATES + + +class PaneMappingTest(unittest.TestCase): + """Every flow state must correspond to a pane the stack owns.""" + + def test_every_state_maps_to_a_pane(self) -> None: + self.assertEqual(set(PANE_FOR_STATE), set(State)) + + def test_only_the_four_panes_are_named(self) -> None: + """A typo in a pane name would leave the stack showing the wrong child.""" + self.assertEqual( + set(PANE_FOR_STATE.values()), + {PANE_CONNECT, PANE_WORKING, PANE_REVIEW, PANE_RESULT}, + ) + + def test_terminal_states_share_the_result_pane(self) -> None: + """Success and failure are both reported in the same place.""" + self.assertEqual(PANE_FOR_STATE[State.DONE], PANE_RESULT) + self.assertEqual(PANE_FOR_STATE[State.FAILED], PANE_RESULT) + + def test_every_working_state_shows_the_working_pane(self) -> None: + """The phase list is indexed by this tuple, so the two must agree.""" + for state in WORKING_STATES: + with self.subTest(state=state.name): + self.assertEqual(PANE_FOR_STATE[state], PANE_WORKING) + + def test_working_states_are_the_ones_that_run_unattended(self) -> None: + """Anything else waits for the user, and must not join the phase list.""" + waiting = {State.CONNECT, State.REVIEW, State.DONE, State.FAILED} + self.assertEqual(set(WORKING_STATES), set(State) - waiting) + + +if __name__ == "__main__": + unittest.main() diff --git a/GrampsWebSync/webapihandler.py b/GrampsWebSync/webapihandler.py index 85959a64e..745e032d9 100644 --- a/GrampsWebSync/webapihandler.py +++ b/GrampsWebSync/webapihandler.py @@ -43,6 +43,41 @@ LOG = logging.getLogger("grampswebsync") +#: Seconds before a request that has produced nothing is abandoned. Without +#: this, ``urlopen`` waits forever and an unreachable-but-listening server +#: hangs the tool with no way out. +TIMEOUT = 60 + + +class ServerTaskFailed(Exception): + """A background task on the server reported failure. + + Carries the server's own description rather than a stringified status dict, + so the message shown to the user says what went wrong. + """ + + +def describe_task_failure(task_status: dict[str, Any]) -> str: + """Extract a readable reason from a failed task status. + + The status dict carries the reason in one of a few shapes depending on how + the task died. Stringifying the whole dict, as this once did, produced a + message no user could act on. + + :param task_status: The server's task status document. + :returns: The most specific description available. + """ + info = task_status.get("info") + if isinstance(info, dict): + for key in ("message", "error", "detail"): + value = info.get(key) + if value: + return str(value) + elif info: + return str(info) + state = task_status.get("state", "FAILURE") + return f"The server reported task state {state}." + def parse_version(version) -> tuple[int, int]: """Simple dependency-free version to parse a SemVer into a list of ints.""" @@ -116,6 +151,10 @@ def __init__( self.fetch_token() self._metadata: dict | None = None + def _open(self, req: Request): + """Open ``req`` with this handler's SSL context and timeout.""" + return urlopen(req, context=self._ctx, timeout=TIMEOUT) + @property def access_token(self) -> str: """Get the access token. Cached after first call unless refresh needed. Auto-refreshing""" @@ -153,7 +192,7 @@ def fetch_metadata(self) -> None: f"{self.url}/metadata/", headers={"Authorization": f"Bearer {self.access_token}", "User-Agent": "GrampsWebSync"}, ) - with urlopen(req, context=self._ctx) as res: + with self._open(req) as res: self._metadata = json.load(res) def fetch_token(self) -> None: @@ -166,7 +205,7 @@ def fetch_token(self) -> None: headers={"Content-Type": "application/json", "User-Agent": "GrampsWebSync"}, ) try: - with urlopen(req, context=self._ctx) as res: + with self._open(req) as res: res_json = json.load(res) except (UnicodeDecodeError, json.JSONDecodeError, HTTPError): if "/api" not in self.url: @@ -184,9 +223,17 @@ def get_lang(self) -> str | None: return (self.metadata.get("locale") or {}).get("lang") def get_api_version(self) -> str | None: - """Fet API version info.""" + """Fetch API version info.""" return (self.metadata.get("gramps_webapi") or {}).get("version") + def get_tree_name(self) -> str: + """Return the name the server gives the tree it is serving.""" + return ((self.metadata.get("database") or {}).get("name") or "") + + def has_task_queue(self) -> bool: + """Whether the server runs transactions on a background task queue.""" + return bool((self.metadata.get("server") or {}).get("task_queue")) + def download_xml(self) -> Path: """Download an XML export and return the path of the temp file.""" url = f"{self.url}/exporters/gramps/file" @@ -204,22 +251,18 @@ def download_xml(self) -> Path: def commit( self, - payload: dict[str, Any], + payload: list[dict[str, Any]], force: bool = True, progress_callback: Callable | None = None, ) -> None: """Commit the changes to the remote database.""" if payload: - api_version = self.get_api_version() - background = api_version and parse_version(api_version) >= (2, 7) data = json.dumps(payload).encode() - endpoint = f"{self.url}/transactions/" - if force: - endpoint = f"{endpoint}?force=1" - if background: - endpoint = f"{endpoint}&background=1" - elif background: - endpoint = f"{endpoint}?background=1" + # Always in the background. The version this addon requires always + # supports it, and a server whose task queue is switched off is + # refused at connect time rather than left to time out here. + query = "force=1&background=1" if force else "background=1" + endpoint = f"{self.url}/transactions/?{query}" req = Request( endpoint, data=data, @@ -230,7 +273,7 @@ def commit( }, ) json_response: dict | None = None - with urlopen(req, context=self._ctx) as res: + with self._open(req) as res: status_code = res.getcode() if status_code == 202: json_response = json.load(res) @@ -264,30 +307,23 @@ def update_task_status( endpoint, headers={"Authorization": f"Bearer {self.access_token}", "User-Agent": "GrampsWebSync"}, ) - try: - with urlopen(req, context=self._ctx) as res: - task_status = json.load(res) - if task_status["state"] == "SUCCESS": - return True - if task_status["state"] in {"FAILURE", "REVOKED"}: - LOG.warning(f"Server task failed: {task_status}") - raise ValueError(str(task_status.get("info", "Server task failed"))) - if progress_callback: - try: - progress = task_status["result_object"]["progress"] - except (KeyError, TypeError): - progress = -1 - progress_callback(progress) - return False - except HTTPError as e: - LOG.warning(f"HTTPError while fetching task status: {e.code} - {e.reason}") - raise ValueError(f"HTTP Error: {e.code} - {e.reason}") - except URLError as e: - LOG.warning(f"URLError while fetching task status: {e.reason}") - raise ValueError(f"URL Error: {e.reason}") - except socket.timeout as e: - LOG.warning(f"Timeout while fetching task status: {e}") - raise ValueError("Connection timed out while fetching task status.") + # HTTPError and URLError are deliberately not wrapped: the caller + # classifies them into specific, actionable messages, which converting + # them to a ValueError would flatten into a generic server error. + with self._open(req) as res: + task_status = json.load(res) + if task_status["state"] == "SUCCESS": + return True + if task_status["state"] in {"FAILURE", "REVOKED"}: + LOG.warning("Server task failed: %s", task_status) + raise ServerTaskFailed(describe_task_failure(task_status)) + if progress_callback: + try: + progress = task_status["result_object"]["progress"] + except (KeyError, TypeError): + progress = -1 + progress_callback(progress) + return False def get_missing_files(self, retry: bool = True) -> list: """Get a list of remote media objects with missing files.""" @@ -296,7 +332,7 @@ def get_missing_files(self, retry: bool = True) -> list: headers={"Authorization": f"Bearer {self.access_token}", "User-Agent": "GrampsWebSync"}, ) try: - with urlopen(req, context=self._ctx) as res: + with self._open(req) as res: res_json = json.load(res) except HTTPError as exc: if exc.code == 401 and retry: @@ -319,8 +355,8 @@ def _download_file( headers={"Authorization": f"Bearer {self.access_token}", "User-Agent": "GrampsWebSync"}, ) try: - with urlopen(req, context=self._ctx) as res: - chunk_size = 1024 + with self._open(req) as res: + chunk_size = 64 * 1024 chunk = res.read(chunk_size) fobj.write(chunk) while chunk: @@ -367,7 +403,7 @@ def _upload_file(self, url: str, fobj, retry: bool = True): method="PUT", ) try: - with urlopen(req, context=self._ctx) as res: + with self._open(req) as res: pass except HTTPError as exc: if exc.code == 401 and retry: diff --git a/JSON/JSON.gpr.py b/JSON/JSON.gpr.py index d735cf11d..98b88306d 100644 --- a/JSON/JSON.gpr.py +++ b/JSON/JSON.gpr.py @@ -3,14 +3,14 @@ id="JSON Export", name=_("JSON Export"), description=_("This is a JSON export"), - version = '1.0.27', + version = '1.0.28', gramps_target_version="6.0", status=STABLE, fname="JSONExport.py", export_function="exportData", export_options="WriterOptionBox", export_options_title=_("JSON options"), - extension="json", + extension="jsonl", help_url="Addon:JSON_Export_Import#Export_JSON", ) @@ -19,11 +19,11 @@ id="JSON Import", name=_("JSON Import"), description=_("This is a JSON import"), - version = '1.0.27', + version = '1.0.28', gramps_target_version="6.0", status=STABLE, fname="JSONImport.py", import_function="importData", - extension="json", + extension="jsonl", help_url="Addon:JSON_Export_Import#Import_JSON", ) diff --git a/PersonRelationshipFilter/PersonRelationshipFilter.gpr.py b/PersonRelationshipFilter/PersonRelationshipFilter.gpr.py new file mode 100644 index 000000000..817e9b6b8 --- /dev/null +++ b/PersonRelationshipFilter/PersonRelationshipFilter.gpr.py @@ -0,0 +1,38 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2015 Nick Hall +# Copyright (C) 2024 Paul Womack (BugBear) +# Copyright (C) 2026 Doug Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +register( + GRAMPLET, + id="Person Relationship Filter", + name=_("Person Relationship Filter"), + description=_("Gramplet providing a person filter on relationships"), + version = '1.0.1', + gramps_target_version="6.0", + status=STABLE, + fname="PersonRelationshipFilter.py", + height=200, + gramplet="PersonRelationshipFilter", + gramplet_title=_("Relationship Filter"), + navtypes=["Person"], + authors=["Paul Womack", "Doug Blank"], + authors_email=["doug.blank@gmail.com"], + help_url="Addon:PersonRelationshipFilter", +) diff --git a/PersonRelationshipFilter/PersonRelationshipFilter.py b/PersonRelationshipFilter/PersonRelationshipFilter.py new file mode 100644 index 000000000..a35169405 --- /dev/null +++ b/PersonRelationshipFilter/PersonRelationshipFilter.py @@ -0,0 +1,353 @@ +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2010 Doug Blank +# Copyright (C) 2011 Nick Hall +# Copyright (C) 2011 Tim G L Lyons +# Copyright (C) 2024 Paul Womack (BugBear) +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +# ------------------------------------------------------------------------- +# +# Gramps modules +# +# ------------------------------------------------------------------------- + +from gi.repository import Gtk + +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.plug import Gramplet + +from gramps.gen.filters.rules import Rule +from gramps.gen.filters.rules.person import ProbablyAlive +from gramps.gen.lib.person import Person +from gramps.gen.lib import Date +from gramps.gen.datehandler import displayer +from gramps.gen.filters import GenericFilter +from gramps.gui import widgets +from gramps.gui.filters.sidebar import SidebarFilter + +_ = glocale.translation.gettext + + +class _RegExpNameList(Rule): + """Rule that checks for full or partial name matches""" + + labels = [_("Text:")] + name = _("People with a name matching ") + description = _( + "Matches people's names containing a substring or " + "matching a regular expression" + ) + category = _("General filters") + allow_regex = True + + def field_list(self, name): + raise NotImplementedError + + def apply_to_one(self, db, person): + for name in [person.primary_name] + person.alternate_names: + for field in self.field_list(name): + if self.match_substring(0, field): + return True + else: + return False + + +class RegExpPersonal(_RegExpNameList): + def field_list(self, name): + return [name.first_name, name.title, name.call, name.nick] + + +class RegExpFamily(_RegExpNameList): + def field_list(self, name): + return [name.get_surname(), name.suffix, name.famnick] + + +class _HasNamedRelation(Rule): + labels = [_("Filter name:")] + name = _("Children of name match") + category = _("Family filters") + description = _("Matches children of anybody with a given name") + + def __init__(self, arg, name_matcher, use_regex=False): + super().__init__(arg, use_regex) + self.name_matcher = name_matcher(arg, use_regex) + + def prepare(self, db, user): + self.name_matcher.requestprepare(db, user) + + def reset(self): + self.name_matcher.requestreset() + + def get_rel_list(self, db, person): + raise NotImplementedError + + def get_spouse_list(self, db, person): + handles = [] + for fam_id in person.family_list: + fam = db.get_raw_family_data(fam_id) + if fam: + for spouse_id in [fam.father_handle, fam.mother_handle]: + if not spouse_id: + continue + if spouse_id == person.handle: + continue + handles.append(spouse_id) + return handles + + def apply_to_one(self, db, person): + for rel_id in self.get_rel_list(db, person): + if rel_id: + rel = db.get_raw_person_data(rel_id) + if self.name_matcher.apply_to_one(db, rel): + return True + return False + + +class _HasNamedParent(_HasNamedRelation): + def get_parent_families(self, db, person): + families = [] + for fam_id in person.parent_family_list: + fam = db.get_family_from_handle(fam_id) + if fam: + families.append(fam) + return families + + +class HasNamedFather(_HasNamedParent): + def get_rel_list(self, db, person): + return map( + lambda fam: fam.get_father_handle(), self.get_parent_families(db, person) + ) + + +class HasNamedMother(_HasNamedParent): + def get_rel_list(self, db, person): + return map( + lambda fam: fam.get_mother_handle(), self.get_parent_families(db, person) + ) + + +class IsSiblingofNamedSibling(_HasNamedRelation): + def get_rel_list(self, db, person): + handles = [] + fam_id = person.get_main_parents_family_handle() # or all families, per above? + fam = db.get_family_from_handle(fam_id) if fam_id else None + if fam: + for child_ref in fam.get_child_ref_list(): + if child_ref and child_ref.ref != person.handle: + handles.append(child_ref.ref) + return handles + + +class HasNamedChild(_HasNamedRelation): + def get_rel_list(self, db, person): + handles = [] + for fam_id in person.family_list: + fam = db.get_family_from_handle(fam_id) + if fam: + for child_ref in fam.get_child_ref_list(): + if child_ref: + handles.append(child_ref.ref) + return handles + + +class HasNamedSpouse(_HasNamedRelation): + def get_rel_list(self, db, person): + return self.get_spouse_list(db, person) + + +class HasName(_HasNamedRelation): + def get_rel_list(self, db, person): + if person.gender == Person.FEMALE and isinstance( + self.name_matcher, RegExpFamily + ): + # for female surnames, we want to trawl the spouses surnames + handles = self.get_spouse_list(db, person) + handles.append(person.handle) + return handles + else: + return [person.handle] + + +def extract_text(entry_widget): + """ + Extract the text from the entry widget, strips off any extra spaces. + """ + return str(entry_widget.get_text().strip()) + + +# leverage to split the name into fore and aft +class SearchableNamePair: + def __init__(self, label, rule_class): + self.widget_personal = widgets.BasicEntry() + self.widget_personal.set_placeholder_text(_("given")) + self.widget_family = widgets.BasicEntry() + self.widget_family.set_placeholder_text(_("surname")) + self.label = label + self.rule_class = rule_class + + def place(self, sidebar): + # container.add_text_entry(self.label, self.widget_personal) + # self.add_text_entry(container, self.label, self.widget_personal) + # unrolled + + sidebar.grid.attach(widgets.BasicLabel(self.label), 1, sidebar.position, 1, 1) + + self.widget_personal.set_hexpand(True) + sidebar.grid.attach(self.widget_personal, 2, sidebar.position, 1, 1) + self.widget_personal.connect("key-press-event", sidebar.key_press) + + self.widget_family.set_hexpand(True) + sidebar.grid.attach(self.widget_family, 3, sidebar.position, 1, 1) + self.widget_family.connect("key-press-event", sidebar.key_press) + sidebar.position += 1 + + def clear(self): + self.widget_personal.set_text("") + self.widget_family.set_text("") + + def _add_to_filter(self, generic_filter, regex, widget, search_class): + v = extract_text(widget) + if v: + rule = self.rule_class([v], search_class, use_regex=regex) + generic_filter.add_rule(rule) + + def add_to_filter(self, generic_filter, regex): + self._add_to_filter(generic_filter, regex, self.widget_personal, RegExpPersonal) + self._add_to_filter(generic_filter, regex, self.widget_family, RegExpFamily) + + +# ------------------------------------------------------------------------- +# +# PersonSidebarFilter class +# +# ------------------------------------------------------------------------- +class PersonSidebarFilter(SidebarFilter): + + def __init__(self, dbstate, uistate, clicked): + self.clicked_func = clicked + self.sensitive_regex = False + + self.names = [ + SearchableNamePair(_("Person"), HasName), + SearchableNamePair(_("Father"), HasNamedFather), + SearchableNamePair(_("Mother"), HasNamedMother), + SearchableNamePair(_("Spouse"), HasNamedSpouse), + SearchableNamePair(_("Sibling 1"), IsSiblingofNamedSibling), + SearchableNamePair(_("Sibling 2"), IsSiblingofNamedSibling), + SearchableNamePair(_("Child 1"), HasNamedChild), + SearchableNamePair(_("Child 2"), HasNamedChild), + ] + self.filter_alive = widgets.DateEntry(uistate, []) + + self.filter_regex = Gtk.CheckButton(label=_("Use regular expressions")) + + SidebarFilter.__init__(self, dbstate, uistate, "Person") + + def create_widget(self): + exdate1 = Date() + exdate2 = Date() + exdate1.set( + Date.QUAL_NONE, + Date.MOD_RANGE, + Date.CAL_GREGORIAN, + (0, 0, 1800, False, 0, 0, 1900, False), + ) + exdate2.set( + Date.QUAL_NONE, Date.MOD_BEFORE, Date.CAL_GREGORIAN, (0, 0, 1850, False) + ) + + msg1 = displayer.display(exdate1) + msg2 = displayer.display(exdate2) + + for w in self.names: + w.place(self) + + self.add_text_entry( + _("Probably Alive"), + self.filter_alive, + _("example: '%(msg1)s' or '%(msg2)s'") % {"msg1": msg1, "msg2": msg2}, + ) + self.add_regex_entry(self.filter_regex) + + def clear(self, obj): + for w in self.names: + w.clear() + self.filter_alive.set_text("") + + def get_filter(self): + """ + Extracts the text strings from the sidebar, and uses them to build up + a new filter. + """ + + regex = self.filter_regex.get_active() + + # build a GenericFilter + generic_filter = GenericFilter() + for w in self.names: + w.add_to_filter(generic_filter, regex) + + alive = extract_text(self.filter_alive) + if alive: + rule = ProbablyAlive([alive]) + generic_filter.add_rule(rule) + + return generic_filter + + +# ------------------------------------------------------------------------- +# +# Filter class +# +# ------------------------------------------------------------------------- +class Filter(Gramplet): + """ + The base class for all filter gramplets. + """ + + FILTER_CLASS: type[SidebarFilter] | None = None + + def init(self): + self.filter = self.FILTER_CLASS( + self.dbstate, self.uistate, self.__filter_clicked + ) + self.widget = self.filter.get_widget() + self.gui.get_container_widget().remove(self.gui.textview) + self.gui.get_container_widget().add(self.widget) + self.widget.show_all() + + def __filter_clicked(self): + """ + Called when the filter apply button is clicked. + """ + self.gui.view.generic_filter = self.filter.get_filter() + self.gui.view.build_tree() + + +# ------------------------------------------------------------------------- +# +# PersonFilter class +# +# ------------------------------------------------------------------------- +class PersonRelationshipFilter(Filter): + """ + A gramplet providing a Person Filter. + """ + + FILTER_CLASS = PersonSidebarFilter diff --git a/PersonRelationshipFilter/po/template.pot b/PersonRelationshipFilter/po/template.pot new file mode 100644 index 000000000..0ae174cf9 --- /dev/null +++ b/PersonRelationshipFilter/po/template.pot @@ -0,0 +1,117 @@ +# SOME DESCRIPTIVE TITLE. +# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER +# This file is distributed under the same license as the PACKAGE package. +# FIRST AUTHOR , YEAR. +# +#, fuzzy +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: \n" +"POT-Creation-Date: 2026-07-26 11:12-0700\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" +"Last-Translator: FULL NAME \n" +"Language-Team: LANGUAGE \n" +"Language: \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:49 +msgid "Text:" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:50 +msgid "People with a name matching " +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:52 +msgid "" +"Matches people's names containing a substring or matching a regular " +"expression" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:55 +msgid "General filters" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:81 +msgid "Filter name:" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:82 +msgid "Children of name match" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:83 +msgid "Family filters" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:84 +msgid "Matches children of anybody with a given name" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:198 +msgid "given" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:200 +msgid "surname" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:247 +msgid "Person" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:248 +msgid "Father" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:249 +msgid "Mother" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:250 +msgid "Spouse" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:251 +msgid "Sibling 1" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:252 +msgid "Sibling 2" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:253 +msgid "Child 1" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:254 +msgid "Child 2" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:258 +msgid "Use regular expressions" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:282 +msgid "Probably Alive" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.py:284 +#, python-format +msgid "example: '%(msg1)s' or '%(msg2)s'" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.gpr.py:25 +msgid "Person Relationship Filter" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.gpr.py:26 +msgid "Gramplet providing a person filter on relationships" +msgstr "" + +#: PersonRelationshipFilter/PersonRelationshipFilter.gpr.py:33 +msgid "Relationship Filter" +msgstr "" diff --git a/PersonRelationshipFilter/tests/__init__.py b/PersonRelationshipFilter/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/PersonRelationshipFilter/tests/test_person_relationship_filter_rules.py b/PersonRelationshipFilter/tests/test_person_relationship_filter_rules.py new file mode 100644 index 000000000..0b4ccb67f --- /dev/null +++ b/PersonRelationshipFilter/tests/test_person_relationship_filter_rules.py @@ -0,0 +1,321 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Doug Blank +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Unit tests for the relationship-matching filter rules defined in +``PersonRelationshipFilter.py``. + +The tests build a small family tree directly in an in-memory database +(no dependency on the shared ``example.gramps`` fixture) so each +rule's relationship traversal and name-field matching can be checked +precisely. Two regressions are covered explicitly: + +* ``IsSiblingofNamedSibling`` used to raise ``HandleError`` when + applied to a person with no recorded parents, because + ``get_family_from_handle(None)`` raises rather than returning + ``None``. +* ``RegExpPersonal``/``RegExpFamily`` searched mismatched ``Name`` + fields (``title`` was listed twice in the personal field list, and + ``call`` name was searched by the family/surname rule instead), so + personal search never matched a person's call name and surname + search could false-positive on an unrelated title or call name. +""" + +# ------------------------ +# Python modules +# ------------------------ +import os +import sys +import unittest + +# ------------------------ +# Gramps modules +# ------------------------ +# Addon root goes on sys.path so ``from PersonRelationshipFilter. +# PersonRelationshipFilter import ...`` resolves the class/functions inside +# the addon module. The fully-qualified form matters: when unittest loads +# this file as ``PersonRelationshipFilter.tests.test_...``, the outer +# ``PersonRelationshipFilter`` is already a namespace package in +# ``sys.modules``, so a bare ``from PersonRelationshipFilter import X`` +# would look for ``X`` as an attribute of that namespace package instead of +# importing the ``PersonRelationshipFilter.py`` submodule that defines it. +ADDON_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + +try: + import gramps +except ImportError as err: + raise unittest.SkipTest("gramps package not available: %s" % err) + +if "GRAMPS_RESOURCES" not in os.environ: + os.environ["GRAMPS_RESOURCES"] = os.path.dirname(os.path.dirname(gramps.__file__)) + +try: + from gramps.gen.db import DbTxn + from gramps.gen.db.utils import make_database + from gramps.gen.lib import ChildRef, Family, Name, Person, Surname + + from PersonRelationshipFilter.PersonRelationshipFilter import ( + HasName, + HasNamedChild, + HasNamedFather, + HasNamedMother, + HasNamedSpouse, + IsSiblingofNamedSibling, + RegExpFamily, + RegExpPersonal, + ) +except Exception as err: # noqa: BLE001 — environment guard + raise unittest.SkipTest("PersonRelationshipFilter module unavailable: %s" % err) + + +def _make_name(first, surname, call="", nick="", title="", famnick=""): + """Build a Name with the given given/surname plus the secondary fields + under test (call name, nickname, title, family nickname).""" + name = Name() + name.set_first_name(first) + name.set_call_name(call) + name.set_nick_name(nick) + name.set_title(title) + name.famnick = famnick + surname_obj = Surname() + surname_obj.set_surname(surname) + name.set_surname_list([surname_obj]) + return name + + +def _make_database(): + db = make_database("sqlite") + db.load(":memory:") + return db + + +class PersonRelationshipFilterRulesTest(unittest.TestCase): + """ + Exercises the rules against a small, precisely-built family tree:: + + Frank Farnsworth (father) + Martha Miller (mother) + -> Carol Farnsworth (call "Caz", nick "Care", title "Dr.", + family nickname "Farns") + -> Dan Farnsworth + Carol Farnsworth + Sam Smith (spouse) + Owen Orphanage -- no recorded parents + """ + + @classmethod + def setUpClass(cls): + cls.db = _make_database() + with DbTxn("build test tree", cls.db) as trans: + father = Person() + father.set_gender(Person.MALE) + father.set_primary_name(_make_name("Frank", "Farnsworth")) + father_handle = cls.db.add_person(father, trans) + + mother = Person() + mother.set_gender(Person.FEMALE) + mother.set_primary_name(_make_name("Martha", "Miller")) + mother_handle = cls.db.add_person(mother, trans) + + carol = Person() + carol.set_gender(Person.FEMALE) + carol.set_primary_name( + _make_name( + "Carol", + "Farnsworth", + call="Caz", + nick="Care", + title="Dr.", + famnick="Farns", + ) + ) + carol_handle = cls.db.add_person(carol, trans) + + dan = Person() + dan.set_gender(Person.MALE) + dan.set_primary_name(_make_name("Dan", "Farnsworth")) + dan_handle = cls.db.add_person(dan, trans) + + orphan = Person() + orphan.set_gender(Person.MALE) + orphan.set_primary_name(_make_name("Owen", "Orphanage")) + orphan_handle = cls.db.add_person(orphan, trans) + + spouse = Person() + spouse.set_gender(Person.MALE) + spouse.set_primary_name(_make_name("Sam", "Smith")) + spouse_handle = cls.db.add_person(spouse, trans) + + parent_family = Family() + parent_family.set_father_handle(father_handle) + parent_family.set_mother_handle(mother_handle) + for child_handle in (carol_handle, dan_handle): + child_ref = ChildRef() + child_ref.set_reference_handle(child_handle) + parent_family.add_child_ref(child_ref) + parent_family_handle = cls.db.add_family(parent_family, trans) + + father.add_family_handle(parent_family_handle) + mother.add_family_handle(parent_family_handle) + carol.add_parent_family_handle(parent_family_handle) + dan.add_parent_family_handle(parent_family_handle) + + marriage = Family() + marriage.set_father_handle(spouse_handle) + marriage.set_mother_handle(carol_handle) + marriage_handle = cls.db.add_family(marriage, trans) + spouse.add_family_handle(marriage_handle) + carol.add_family_handle(marriage_handle) + + for person in (father, mother, carol, dan, orphan, spouse): + cls.db.commit_person(person, trans) + + cls.father = cls.db.get_person_from_handle(father_handle) + cls.mother = cls.db.get_person_from_handle(mother_handle) + cls.carol = cls.db.get_person_from_handle(carol_handle) + cls.dan = cls.db.get_person_from_handle(dan_handle) + cls.orphan = cls.db.get_person_from_handle(orphan_handle) + cls.spouse = cls.db.get_person_from_handle(spouse_handle) + + @classmethod + def tearDownClass(cls): + cls.db.close() + cls.db = None + + def _match_name(self, matcher_class, value, person, use_regex=False): + """Apply a bare RegExpPersonal/RegExpFamily rule to one person.""" + rule = matcher_class([value], use_regex=use_regex) + rule.requestprepare(self.db, None) + try: + return rule.apply_to_one(self.db, person) + finally: + rule.requestreset() + + def _match_relation(self, rule_class, value, person, matcher_class=RegExpPersonal): + """Apply a _HasNamedRelation rule (Father/Mother/Sibling/Child/Spouse) + to one person.""" + rule = rule_class([value], matcher_class, use_regex=False) + rule.requestprepare(self.db, None) + try: + return rule.apply_to_one(self.db, person) + finally: + rule.requestreset() + + # -- name field matching -------------------------------------------- + + def test_personal_matches_first_name(self): + self.assertTrue(self._match_name(RegExpPersonal, "Carol", self.carol)) + + def test_personal_matches_call_name(self): + """Regression: the personal field list used to list 'title' twice + instead of including the call name.""" + self.assertTrue(self._match_name(RegExpPersonal, "Caz", self.carol)) + + def test_personal_matches_nick_name(self): + self.assertTrue(self._match_name(RegExpPersonal, "Care", self.carol)) + + def test_personal_matches_title(self): + self.assertTrue(self._match_name(RegExpPersonal, "Dr", self.carol)) + + def test_personal_does_not_match_surname(self): + self.assertFalse(self._match_name(RegExpPersonal, "Farnsworth", self.carol)) + + def test_family_matches_surname(self): + self.assertTrue(self._match_name(RegExpFamily, "Farnsworth", self.carol)) + + def test_family_matches_famnick(self): + self.assertTrue(self._match_name(RegExpFamily, "Farns", self.carol)) + + def test_family_does_not_match_title(self): + """Regression: the family/surname field list used to include the + personal title field, causing false-positive surname matches.""" + self.assertFalse(self._match_name(RegExpFamily, "Dr", self.carol)) + + def test_family_does_not_match_call_name(self): + """Regression: the family/surname field list used to include the + personal call name field, causing false-positive surname matches.""" + self.assertFalse(self._match_name(RegExpFamily, "Caz", self.carol)) + + def test_regex_mode_matches_pattern(self): + self.assertTrue( + self._match_name(RegExpPersonal, "^Car", self.carol, use_regex=True) + ) + self.assertFalse( + self._match_name(RegExpPersonal, "^ar", self.carol, use_regex=True) + ) + + # -- relationship traversal ------------------------------------------ + + def test_has_named_father(self): + self.assertTrue(self._match_relation(HasNamedFather, "Frank", self.carol)) + self.assertFalse(self._match_relation(HasNamedFather, "Nobody", self.carol)) + + def test_has_named_mother(self): + self.assertTrue(self._match_relation(HasNamedMother, "Martha", self.carol)) + + def test_has_named_child(self): + self.assertTrue(self._match_relation(HasNamedChild, "Carol", self.father)) + self.assertTrue(self._match_relation(HasNamedChild, "Dan", self.mother)) + + def test_has_named_spouse(self): + self.assertTrue(self._match_relation(HasNamedSpouse, "Sam", self.carol)) + self.assertFalse(self._match_relation(HasNamedSpouse, "Frank", self.carol)) + + def test_is_sibling_of_named_sibling(self): + self.assertTrue( + self._match_relation(IsSiblingofNamedSibling, "Dan", self.carol) + ) + self.assertTrue( + self._match_relation(IsSiblingofNamedSibling, "Carol", self.dan) + ) + + def test_is_sibling_of_named_sibling_excludes_self(self): + self.assertFalse( + self._match_relation(IsSiblingofNamedSibling, "Carol", self.carol) + ) + + def test_is_sibling_of_named_sibling_with_no_parents_does_not_crash(self): + """Regression: applying this rule to a person with no recorded + parents used to raise HandleError from + get_family_from_handle(None).""" + self.assertFalse( + self._match_relation(IsSiblingofNamedSibling, "Anyone", self.orphan) + ) + + def test_has_name_matches_own_name(self): + self.assertTrue(self._match_relation(HasName, "Carol", self.carol)) + + def test_has_name_female_family_search_trawls_spouse_surname(self): + """A female's own family/surname search also matches her spouse's + surname (so 'Person' search finds her under her married name).""" + self.assertTrue( + self._match_relation( + HasName, "Smith", self.carol, matcher_class=RegExpFamily + ) + ) + self.assertFalse( + self._match_relation( + HasName, "Smith", self.father, matcher_class=RegExpFamily + ) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/README.md b/README.md index 1f1fae31d..cd7ac8e66 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,11 @@ -addons-source [![Build Status](https://travis-ci.org/gramps-project/addons-source.svg?branch=master)](https://travis-ci.org/gramps-project/addons-source) +addons-source Translation status ============= Source code of contributed third-party addons for the [Gramps genealogy program](https://github.com/gramps-project/gramps). -You can develop your own addon following the [Addons Development](https://gramps-project.org/wiki/index.php?title=Addons_development) wiki. +You can develop your own addon following the in-repo [Addon Development manual](docs/addon-development/README.md); the contributor workflow (forks, branches, pull requests) is in [CONTRIBUTING.md](CONTRIBUTING.md). See also the [Addons Development](https://gramps-project.org/wiki/index.php?title=Addons_development) wiki page. Note: The default git branch is `master`. The master branch should only be used to develop addons that require features or changes found in the Gramps master branch. Most of the time addons should be developed to work with the current released version of Gramps (`maintenance/gramps60` for the Gramps 6.0.x versions for example). diff --git a/SharedPostgreSQL/shareddbapi.py b/SharedPostgreSQL/shareddbapi.py index 2c2cf7123..8431f1d0e 100644 --- a/SharedPostgreSQL/shareddbapi.py +++ b/SharedPostgreSQL/shareddbapi.py @@ -289,7 +289,9 @@ def _create_schema(self, json_data): self.dbapi.execute( "CREATE INDEX citation_gramps_id " "ON citation(treeid,gramps_id)" ) - self.dbapi.execute("CREATE INDEX media_desc " "ON media(treeid,desc)") + self.dbapi.execute( + f"CREATE INDEX media_desc ON media(treeid,{self._quote_column('desc')})" + ) self.dbapi.execute("CREATE INDEX media_gramps_id " "ON media(treeid,gramps_id)") self.dbapi.execute("CREATE INDEX place_title " "ON place(treeid,title)") self.dbapi.execute( @@ -699,7 +701,7 @@ def get_media_handles(self, sort_handles=False, locale=glocale): self.dbapi.execute( "SELECT handle FROM media " "WHERE treeid = ? " - "ORDER BY desc " + f"ORDER BY {self._quote_column('desc')} " 'COLLATE "%s"' % locale.get_collation(), [self.dbapi.treeid], ) @@ -882,7 +884,7 @@ def _commit_raw(self, data, obj_key): else: # Insert the object: sql = ( - f"INSERT INTO %s (treeid, handle, {self.serializer.data_field}) VALUES (?, ?)" + f"INSERT INTO %s (treeid, handle, {self.serializer.data_field}) VALUES (?, ?, ?)" ) % table self.dbapi.execute( sql, [self.dbapi.treeid, handle, self.serializer.data_to_string(data)] @@ -1306,6 +1308,17 @@ def get_surname_list(self): surname_list.append(row[0]) return surname_list + def _quote_column(self, col): + """ + Return a safe column name for the current dialect. + + Override in dialect subclasses to handle reserved keywords, e.g. by + quoting or renaming them. + """ + # Mirrors the hook added by gramps PR #2178; drop once that is merged + # and shareddbapi is resynced with core dbapi. + return col + def _sql_type(self, schema_type, max_length): """ Given a schema type, return the SQL type for @@ -1346,7 +1359,7 @@ def _create_secondary_columns(self): sql_type = self._sql_type(schema_type, max_length) self.dbapi.execute( "ALTER TABLE %s ADD COLUMN %s %s" - % (table_name, field, sql_type) + % (table_name, self._quote_column(field), sql_type) ) def _update_secondary_values(self, obj): @@ -1360,7 +1373,7 @@ def _update_secondary_values(self, obj): sets = [] values = [] for field in fields: - sets.append("%s = ?" % field) + sets.append("%s = ?" % self._quote_column(field)) values.append(getattr(obj, field)) # Derived fields diff --git a/SharedPostgreSQL/sharedpostgresql.gpr.py b/SharedPostgreSQL/sharedpostgresql.gpr.py index 9ab3c60f3..2021caa63 100644 --- a/SharedPostgreSQL/sharedpostgresql.gpr.py +++ b/SharedPostgreSQL/sharedpostgresql.gpr.py @@ -24,7 +24,7 @@ name=_("SharedPostgreSQL"), name_accell=_("Shared _PostgreSQL Database"), description=_("Shared PostgreSQL Database"), - version = '0.1.14', + version = '0.1.16', gramps_target_version="6.0", status=STABLE, fname="sharedpostgresql.py", diff --git a/SharedPostgreSQL/sharedpostgresql.py b/SharedPostgreSQL/sharedpostgresql.py index 5fbe6ed19..95a0a96ba 100644 --- a/SharedPostgreSQL/sharedpostgresql.py +++ b/SharedPostgreSQL/sharedpostgresql.py @@ -24,6 +24,7 @@ Backend for PostgreSQL database. """ +import hashlib import os import re from uuid import uuid4 @@ -53,6 +54,21 @@ # # ------------------------------------------------------------------------- class SharedPostgreSQL(SharedDBAPI): + dialect = "postgresql" + + # Column names as they physically exist in shared PostgreSQL databases. + # "desc" is reserved in PostgreSQL and was renamed by the old blanket + # substring rewrite, which also caught "description" as a side effect. + # Both names are kept so existing databases stay readable. + _COLUMN_NAMES = {"desc": "desc_", "description": "desc_ription"} + + def _quote_column(self, col): + return self._COLUMN_NAMES.get(col, col) + + def _sql_type(self, schema_type, max_length): + result = super()._sql_type(schema_type, max_length) + return "bytea" if result == "BLOB" else result + def get_summary(self): """ Return a diction of information about this database @@ -79,11 +95,7 @@ def _initialize(self, directory, username, password): config_mgr.register("tree.uuid", "") if not os.path.exists(config_file): - config_mgr.set("database.dbname", "gramps") - config_mgr.set("database.host", config.get("database.host")) - config_mgr.set("database.port", config.get("database.port")) - config_mgr.set("tree.uuid", uuid4().hex) - config_mgr.save() + self._create_settings(config_file, config_mgr, directory, username, password) config_mgr.load() @@ -106,6 +118,37 @@ def _initialize(self, directory, username, password): except psycopg2.OperationalError as msg: raise DbConnectionError(str(msg), config_file) + def _create_settings(self, config_file, config_mgr, directory, username, password): + """Create settings.ini for a tree that has not been initialized yet.""" + host = config.get("database.host") + port = config.get("database.port") + dbkwargs = {"dbname": "gramps", "host": host, "port": port} + if username: + dbkwargs["user"] = username + if password: + dbkwargs["password"] = password + # Two processes opening a brand-new tree at the same time would each + # generate a UUID and overwrite each other's settings.ini. Serialize + # on the database, since the filesystem may not do so itself. + digest = hashlib.sha256(os.path.abspath(directory).encode()).digest() + lock_key = int.from_bytes(digest[:8], "big", signed=True) + try: + conn = psycopg2.connect(**dbkwargs) + except psycopg2.OperationalError as msg: + raise DbConnectionError(str(msg), config_file) + try: + conn.autocommit = True + # The lock is released when the connection closes. + conn.cursor().execute("SELECT pg_advisory_lock(%s)", [lock_key]) + if not os.path.exists(config_file): + config_mgr.set("database.dbname", "gramps") + config_mgr.set("database.host", host) + config_mgr.set("database.port", port) + config_mgr.set("tree.uuid", uuid4().hex) + config_mgr.save() + finally: + conn.close() + # ------------------------------------------------------------------------- # @@ -179,18 +222,20 @@ def check_collation(self, locale): Checks that a collation exists and if not creates it. :param locale: Locale to be checked. - :param type: A GrampsLocale object. + :type locale: A GrampsLocale object. """ - # Duplicating system collations works, but to delete them the schema - # must be specified, so get the current schema collation = locale.get_collation() - self.execute( - 'CREATE COLLATION IF NOT EXISTS "%s"' - "(LOCALE = '%s')" % (collation, locale.collation) - ) + # Use pg_collation to check existence rather than IF NOT EXISTS, which + # requires PostgreSQL 12+. + self.execute("SELECT 1 FROM pg_collation WHERE collname = %s", [collation]) + if not self.fetchone(): + self.execute( + "CREATE COLLATION \"%s\" (LOCALE = '%s')" + % (collation, locale.collation) + ) def execute(self, *args, **kwargs): - sql = _hack_query(args[0]) + sql = _translate_sql(args[0]) if len(args) > 1: args = args[1] else: @@ -279,7 +324,7 @@ def execute(self, *args, **kwargs): :param kwargs: arguments to be passed to the sqlite3 execute statement :type kwargs: list """ - sql = _hack_query(args[0]) + sql = _translate_sql(args[0]) if len(args) > 1: args = args[1] else: @@ -297,25 +342,30 @@ def fetchmany(self): return None -def _hack_query(query): - query = query.replace("?", "%s") - query = query.replace("REGEXP", "~") - query = query.replace("desc", "desc_") - query = query.replace("BLOB", "bytea") - query = query.replace("INTEGER PRIMARY KEY", "SERIAL PRIMARY KEY") - ## LIMIT offset, count - ## count can be -1, for all - ## LIMIT -1 - ## LIMIT offset, -1 - query = query.replace("LIMIT -1", "LIMIT all") ## - match = re.match(".* LIMIT (.*), (.*) ", query) - if match and match.groups(): - offset, count = match.groups() - if count == "-1": - count = "all" - query = re.sub( - "(.*) LIMIT (.*), (.*) ", - "\\1 LIMIT %s OFFSET %s " % (count, offset), - query, - ) - return query +def _translate_sql(query): + """ + Translate an SQLite-flavoured SQL statement to PostgreSQL. + + :param query: the statement to translate. + :type query: str + :returns: the translated statement. + :rtype: str + """ + sql = query.replace("?", "%s") # qmark -> format paramstyle + sql = sql.replace(" REGEXP ", " ~ ") # SQLite REGEXP -> PostgreSQL ~ + # SQLite LIKE is case-insensitive (ASCII), PostgreSQL's is not; ILIKE is + # the case-insensitive equivalent. Its folding follows the connection + # locale, so non-ASCII patterns may fold differently than under SQLite. + sql = re.sub(r"\bLIKE\b", "ILIKE", sql, flags=re.IGNORECASE) + sql = sql.replace("INTEGER PRIMARY KEY", "SERIAL PRIMARY KEY") + sql = re.sub(r"\bBLOB\b", "BYTEA", sql) # SQLite BLOB -> PostgreSQL BYTEA + # LIMIT offset, count -> LIMIT count OFFSET offset; a count of -1 means all + sql = re.sub( + r"\bLIMIT\s+(-?\d+)\s*,\s*(-?\d+)", + lambda m: f'LIMIT {"ALL" if m.group(2) == "-1" else m.group(2)}' + f" OFFSET {m.group(1)}", + sql, + flags=re.IGNORECASE, + ) + sql = re.sub(r"\bLIMIT\s+-1\b", "LIMIT ALL", sql, flags=re.IGNORECASE) + return sql diff --git a/SharedPostgreSQL/tests/__init__.py b/SharedPostgreSQL/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/SharedPostgreSQL/tests/test_initialize.py b/SharedPostgreSQL/tests/test_initialize.py new file mode 100644 index 000000000..083cb136c --- /dev/null +++ b/SharedPostgreSQL/tests/test_initialize.py @@ -0,0 +1,294 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Unit tests for SharedPostgreSQL._create_settings(). + +Several processes can open a never-before-opened tree at the same time, and +each would generate its own tree UUID. Initialization is therefore serialized +on a PostgreSQL advisory lock rather than on the filesystem, which may not +order concurrent writes to settings.ini. These tests cover: + + - the advisory lock key derived from the tree directory + - the losing process adopting the winner's settings.ini + - the lock being taken before settings.ini is inspected + - the connection being closed (releasing the lock) even on failure + +The concurrency itself is not exercised here; that needs a real server and +several processes. psycopg2 is stubbed so no database is required. + +Run with:: + + python3 -m unittest SharedPostgreSQL.tests.test_initialize -v +""" + +# ------------------------------------------------------------------------- +# +# Standard python modules +# +# ------------------------------------------------------------------------- +import os +import shutil +import sys +import tempfile +import unittest +from unittest import mock + +# ------------------------------------------------------------------------- +# +# Stub psycopg2 before the addon is imported so no real DB driver is needed +# +# ------------------------------------------------------------------------- +ADDON_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + +_mock_psycopg2 = mock.MagicMock() +_mock_psycopg2.paramstyle = "format" +_mock_psycopg2.OperationalError = Exception +sys.modules.setdefault("psycopg2", _mock_psycopg2) + +# ------------------------------------------------------------------------- +# +# Gramps modules (required by the addon's import chain) +# +# ------------------------------------------------------------------------- +try: + import gramps +except ImportError as _err: + raise unittest.SkipTest("gramps package not available: %s" % _err) + +if "GRAMPS_RESOURCES" not in os.environ: + os.environ["GRAMPS_RESOURCES"] = os.path.dirname(os.path.dirname(gramps.__file__)) + +try: + from gramps.gen.utils.configmanager import ConfigManager + + from SharedPostgreSQL import sharedpostgresql + from SharedPostgreSQL.sharedpostgresql import SharedPostgreSQL +except Exception as _err: + raise unittest.SkipTest("SharedPostgreSQL module unavailable: %s" % _err) + + +# ------------------------------------------------------------------------- +# +# Base class +# +# ------------------------------------------------------------------------- +class CreateSettingsTestCase(unittest.TestCase): + """Shared fixture: an empty tree directory and a stubbed connection.""" + + def setUp(self): + self.pg = SharedPostgreSQL.__new__(SharedPostgreSQL) + self.tmpdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True) + self.config_file = os.path.join(self.tmpdir, "settings.ini") + + def make_config_mgr(self, config_file=None): + """A ConfigManager registered the way _initialize() registers it.""" + config_mgr = ConfigManager(config_file or self.config_file) + config_mgr.register("database.dbname", "") + config_mgr.register("database.host", "") + config_mgr.register("database.port", "") + config_mgr.register("tree.uuid", "") + return config_mgr + + def create_settings(self, directory=None, config_mgr=None, conn=None): + """Run _create_settings() against a stubbed psycopg2 connection.""" + directory = directory or self.tmpdir + config_file = os.path.join(directory, "settings.ini") + if config_mgr is None: + config_mgr = self.make_config_mgr(config_file) + conn = conn if conn is not None else mock.MagicMock() + with mock.patch.object( + sharedpostgresql.psycopg2, "connect", return_value=conn + ) as connect: + self.pg._create_settings(config_file, config_mgr, directory, None, None) + return conn, connect + + @staticmethod + def lock_key(conn): + """The advisory lock key passed to pg_advisory_lock().""" + sql, params = conn.cursor.return_value.execute.call_args[0] + assert "pg_advisory_lock" in sql, sql + return params[0] + + def stored_uuid(self, config_file=None): + """Read tree.uuid back from the settings file on disk.""" + config_mgr = self.make_config_mgr(config_file) + config_mgr.load() + return config_mgr.get("tree.uuid") + + +# ------------------------------------------------------------------------- +# +# TestAdvisoryLockKey +# +# ------------------------------------------------------------------------- +class TestAdvisoryLockKey(CreateSettingsTestCase): + """The lock key is derived deterministically from the tree directory.""" + + def test_same_directory_gives_same_key(self): + """Racing processes must agree on the key or the lock cannot bind.""" + first, _ = self.create_settings() + second, _ = self.create_settings() + self.assertEqual(self.lock_key(first), self.lock_key(second)) + + def test_trailing_separator_gives_same_key(self): + """abspath() normalizes the path, so a trailing slash is harmless.""" + plain, _ = self.create_settings(directory=self.tmpdir) + slashed, _ = self.create_settings(directory=self.tmpdir + os.sep) + self.assertEqual(self.lock_key(plain), self.lock_key(slashed)) + + def test_different_directories_give_different_keys(self): + """Unrelated trees must not serialize against each other.""" + other = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, other, ignore_errors=True) + mine, _ = self.create_settings() + theirs, _ = self.create_settings(directory=other) + self.assertNotEqual(self.lock_key(mine), self.lock_key(theirs)) + + def test_key_fits_signed_64_bit(self): + """pg_advisory_lock takes a bigint; a wider value is a runtime error.""" + conn, _ = self.create_settings() + self.assertGreaterEqual(self.lock_key(conn), -(2**63)) + self.assertLess(self.lock_key(conn), 2**63) + + def test_high_bit_digest_yields_negative_key(self): + """A digest with the top bit set must wrap to a negative bigint. + + Reading the digest unsigned would produce a value above 2**63 that + PostgreSQL rejects, and only for the fraction of paths whose hash + happens to have that bit set -- so pin it down explicitly. + """ + digest = mock.MagicMock() + digest.digest.return_value = b"\xff" * 32 + with mock.patch.object( + sharedpostgresql.hashlib, "sha256", return_value=digest + ): + conn, _ = self.create_settings() + self.assertEqual(self.lock_key(conn), -1) + + +# ------------------------------------------------------------------------- +# +# TestCreateSettingsRace +# +# ------------------------------------------------------------------------- +class TestCreateSettingsRace(CreateSettingsTestCase): + """Only the process holding the lock may generate the tree UUID.""" + + def test_winner_writes_uuid(self): + """Baseline: an uncontended call does create the settings file.""" + self.create_settings() + self.assertTrue(os.path.exists(self.config_file)) + self.assertTrue(self.stored_uuid()) + + def test_loser_adopts_winners_uuid(self): + """A settings file appearing while we block on the lock is kept. + + This is the failure that wedged trees before the lock existed: both + processes generated a UUID and the second overwrote the first, so the + data written under the first UUID became unreachable. + """ + conn = mock.MagicMock() + conn.cursor.return_value.execute.side_effect = self._win_race + self.create_settings(conn=conn) + self.assertEqual(self.stored_uuid(), "winner") + + def test_loser_leaves_file_byte_identical(self): + """The loser must not rewrite the file at all, not even equivalently.""" + conn = mock.MagicMock() + conn.cursor.return_value.execute.side_effect = self._win_race + self.create_settings(conn=conn) + with open(self.config_file, encoding="utf-8") as fh: + self.assertEqual(fh.read(), self._WINNER_INI) + + _WINNER_INI = "[database]\ndbname='gramps'\n\n[tree]\nuuid='winner'\n\n" + + def _win_race(self, *args, **kwargs): + """Simulate another process winning while this one waits for the lock.""" + with open(self.config_file, "w", encoding="utf-8") as fh: + fh.write(self._WINNER_INI) + + +# ------------------------------------------------------------------------- +# +# TestCreateSettingsLockOrdering +# +# ------------------------------------------------------------------------- +class TestCreateSettingsLockOrdering(CreateSettingsTestCase): + """The lock must be held before settings.ini is inspected.""" + + def test_lock_precedes_existence_check(self): + """Checking first and locking second would reopen the race window.""" + events = [] + real_exists = os.path.exists + + def recording_exists(path): + if path == self.config_file: + events.append("exists") + return real_exists(path) + + conn = mock.MagicMock() + conn.cursor.return_value.execute.side_effect = lambda *a, **k: events.append( + "lock" + ) + with mock.patch("os.path.exists", recording_exists): + self.create_settings(conn=conn) + + self.assertEqual(events, ["lock", "exists"]) + + +# ------------------------------------------------------------------------- +# +# TestCreateSettingsLockRelease +# +# ------------------------------------------------------------------------- +class TestCreateSettingsLockRelease(CreateSettingsTestCase): + """The lock is released by closing the session, so close() must always run. + + There is no explicit pg_advisory_unlock; a session level lock is dropped + when the connection ends. A leaked connection would therefore block every + other process opening the same tree. + """ + + def test_connection_closed_on_success(self): + conn, _ = self.create_settings() + conn.close.assert_called_once_with() + + def test_connection_closed_when_save_fails(self): + config_mgr = mock.MagicMock() + config_mgr.save.side_effect = RuntimeError("read-only filesystem") + conn = mock.MagicMock() + with self.assertRaises(RuntimeError): + self.create_settings(config_mgr=config_mgr, conn=conn) + conn.close.assert_called_once_with() + + def test_connection_closed_when_lock_fails(self): + conn = mock.MagicMock() + conn.cursor.return_value.execute.side_effect = RuntimeError("lock timeout") + with self.assertRaises(RuntimeError): + self.create_settings(conn=conn) + conn.close.assert_called_once_with() + + +if __name__ == "__main__": + unittest.main() diff --git a/SharedPostgreSQL/tests/test_sql_translations.py b/SharedPostgreSQL/tests/test_sql_translations.py new file mode 100644 index 000000000..4f6444fa2 --- /dev/null +++ b/SharedPostgreSQL/tests/test_sql_translations.py @@ -0,0 +1,447 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2015-2016 Douglas S. Blank +# Copyright (C) 2026 David Straub +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Unit tests for the SharedPostgreSQL SQL dialect translations. + +These tests cover every rewrite rule applied before a query reaches psycopg2: + - qmark -> format paramstyle (? -> %s) + - REGEXP operator (REGEXP -> ~) + - LIKE operator (LIKE -> ILIKE) + - autoincrement primary key (INTEGER PRIMARY KEY -> SERIAL PRIMARY KEY) + - BLOB column type (BLOB -> BYTEA) + - two-arg LIMIT (LIMIT offset, count -> LIMIT count OFFSET offset) + - unlimited LIMIT (LIMIT -1 -> LIMIT ALL) + +and the column naming applied by _quote_column(). + +psycopg2 is stubbed so no real database is required. gramps core is +required for the import chain; the whole module is skipped cleanly if +it is not present. + +Run with:: + + python3 -m unittest SharedPostgreSQL.tests.test_sql_translations -v +""" + +# ------------------------------------------------------------------------- +# +# Standard python modules +# +# ------------------------------------------------------------------------- +import os +import sys +import unittest +from unittest import mock + +# ------------------------------------------------------------------------- +# +# Stub psycopg2 before the addon is imported so no real DB driver is needed +# +# ------------------------------------------------------------------------- +ADDON_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if ADDON_DIR not in sys.path: + sys.path.insert(0, ADDON_DIR) + +_mock_psycopg2 = mock.MagicMock() +_mock_psycopg2.paramstyle = "format" +_mock_psycopg2.OperationalError = Exception +sys.modules.setdefault("psycopg2", _mock_psycopg2) + +# ------------------------------------------------------------------------- +# +# Gramps modules (required by the addon's import chain) +# +# ------------------------------------------------------------------------- +try: + import gramps +except ImportError as _err: + raise unittest.SkipTest("gramps package not available: %s" % _err) + +if "GRAMPS_RESOURCES" not in os.environ: + os.environ["GRAMPS_RESOURCES"] = os.path.dirname(os.path.dirname(gramps.__file__)) + +try: + from SharedPostgreSQL.sharedpostgresql import Connection, Cursor, SharedPostgreSQL +except Exception as _err: + raise unittest.SkipTest("SharedPostgreSQL module unavailable: %s" % _err) + +# The addon imports shareddbapi by bare name, the way Gramps loads addons, so +# reach the base class through the MRO rather than importing it a second time. +SharedDBAPI = SharedPostgreSQL.__bases__[0] + + +# ------------------------------------------------------------------------- +# +# Helpers +# +# ------------------------------------------------------------------------- + + +def _make_connection(): + """Return a (Connection, mock_cursor) pair without touching psycopg2.""" + conn = Connection.__new__(Connection) + cursor = mock.MagicMock() + conn._Connection__cursor = cursor + return conn, cursor + + +def _translated(sql): + """Return the SQL string that Connection.execute() would pass to psycopg2.""" + conn, cursor = _make_connection() + conn.execute(sql) + return cursor.execute.call_args[0][0] + + +# ------------------------------------------------------------------------- +# +# TestExecuteQmarkParamstyle +# +# ------------------------------------------------------------------------- +class TestExecuteQmarkParamstyle(unittest.TestCase): + """? -> %s substitution.""" + + def test_single_placeholder(self): + self.assertEqual( + _translated("SELECT * FROM person WHERE gramps_id = ?"), + "SELECT * FROM person WHERE gramps_id = %s", + ) + + def test_multiple_placeholders(self): + result = _translated("INSERT INTO t (treeid, a) VALUES (?, ?)") + self.assertEqual(result.count("%s"), 2) + self.assertNotIn("?", result) + + def test_no_placeholders_unchanged(self): + sql = "SELECT * FROM person" + self.assertEqual(_translated(sql), sql) + + +# ------------------------------------------------------------------------- +# +# TestExecuteRegexpOperator +# +# ------------------------------------------------------------------------- +class TestExecuteRegexpOperator(unittest.TestCase): + """REGEXP -> ~ substitution.""" + + def test_regexp_replaced(self): + result = _translated("SELECT * FROM person WHERE name REGEXP 'foo'") + self.assertIn(" ~ ", result) + self.assertNotIn("REGEXP", result) + + def test_no_regexp_unchanged(self): + sql = "SELECT * FROM person WHERE name = 'foo'" + self.assertEqual(_translated(sql), sql) + + +# ------------------------------------------------------------------------- +# +# TestExecuteLikeOperator +# +# ------------------------------------------------------------------------- +class TestExecuteLikeOperator(unittest.TestCase): + """LIKE -> ILIKE substitution.""" + + def test_like_replaced(self): + result = _translated("SELECT * FROM person WHERE surname LIKE ?") + self.assertIn("ILIKE", result) + + def test_lowercase_like_replaced(self): + result = _translated("SELECT * FROM person WHERE surname like ?") + self.assertIn("ILIKE", result) + + def test_not_like_replaced(self): + result = _translated("SELECT * FROM person WHERE surname NOT LIKE ?") + self.assertIn("NOT ILIKE", result) + + def test_ilike_not_double_rewritten(self): + sql = "SELECT * FROM person WHERE surname ILIKE %s" + self.assertEqual(_translated(sql), sql) + + def test_like_word_boundary_not_in_identifier(self): + """LIKE as part of a longer identifier is not replaced.""" + sql = "SELECT likelihood FROM person" + self.assertEqual(_translated(sql), sql) + + def test_no_like_unchanged(self): + sql = "SELECT * FROM person WHERE surname = %s" + self.assertEqual(_translated(sql), sql) + + +# ------------------------------------------------------------------------- +# +# TestExecuteSerialPrimaryKey +# +# ------------------------------------------------------------------------- +class TestExecuteSerialPrimaryKey(unittest.TestCase): + """INTEGER PRIMARY KEY -> SERIAL PRIMARY KEY. + + The trees table relies on the treeid being assigned automatically when a + new tree is inserted, which in PostgreSQL requires SERIAL. + """ + + def test_trees_table_uses_serial(self): + result = _translated( + "CREATE TABLE trees (treeid INTEGER PRIMARY KEY, uuid VARCHAR(32))" + ) + self.assertIn("treeid SERIAL PRIMARY KEY", result) + self.assertNotIn("INTEGER PRIMARY KEY", result) + + def test_plain_integer_column_unchanged(self): + sql = "ALTER TABLE person ADD COLUMN priority INTEGER" + self.assertEqual(_translated(sql), sql) + + +# ------------------------------------------------------------------------- +# +# TestExecuteBlobType +# +# ------------------------------------------------------------------------- +class TestExecuteBlobType(unittest.TestCase): + """BLOB -> BYTEA substitution.""" + + def test_metadata_table_blob_replaced(self): + result = _translated( + "CREATE TABLE metadata " + "(treeid INTEGER, setting VARCHAR(50), json_data TEXT, value BLOB)" + ) + self.assertIn("BYTEA", result) + self.assertNotIn("BLOB", result) + + def test_blob_data_column_replaced(self): + result = _translated( + "CREATE TABLE person " + "(treeid INTEGER, handle VARCHAR(50), blob_data BLOB)" + ) + self.assertIn("blob_data BYTEA", result) + + def test_blob_word_boundary_not_in_identifier(self): + """BLOB as part of a longer identifier is not replaced.""" + result = _translated("SELECT blobfield FROM person") + self.assertEqual(result, "SELECT blobfield FROM person") + + def test_multiple_blob_columns_all_replaced(self): + result = _translated("CREATE TABLE t (a BLOB, b TEXT, c BLOB)") + self.assertEqual(result.count("BYTEA"), 2) + self.assertNotIn("BLOB", result) + + +# ------------------------------------------------------------------------- +# +# TestExecuteLimitTranslations +# +# ------------------------------------------------------------------------- +class TestExecuteLimitTranslations(unittest.TestCase): + """LIMIT dialect translations.""" + + def test_limit_minus_one_becomes_all(self): + result = _translated("SELECT * FROM person LIMIT -1") + self.assertIn("LIMIT ALL", result) + self.assertNotIn("-1", result) + + def test_limit_offset_comma_count(self): + result = _translated("SELECT * FROM person LIMIT 5, 10") + self.assertIn("LIMIT 10 OFFSET 5", result) + + def test_limit_offset_comma_minus_one(self): + result = _translated("SELECT * FROM person LIMIT 5, -1") + self.assertIn("LIMIT ALL OFFSET 5", result) + + def test_plain_limit_unchanged(self): + result = _translated("SELECT * FROM person LIMIT 10") + self.assertEqual(result, "SELECT * FROM person LIMIT 10") + + def test_limit_with_offset_clause_unchanged(self): + result = _translated("SELECT * FROM person LIMIT 10 OFFSET 5") + self.assertEqual(result, "SELECT * FROM person LIMIT 10 OFFSET 5") + + +# ------------------------------------------------------------------------- +# +# TestExecuteLeavesIdentifiersAlone +# +# ------------------------------------------------------------------------- +class TestExecuteLeavesIdentifiersAlone(unittest.TestCase): + """Identifiers are no longer rewritten by blind substring replacement. + + The previous implementation replaced every occurrence of "desc", which + also corrupted unrelated identifiers. Column naming is now the job of + _quote_column(), so execute() must leave identifiers untouched. + """ + + def test_desc_column_not_rewritten(self): + sql = "SELECT handle FROM media ORDER BY desc_" + self.assertEqual(_translated(sql), sql) + + def test_description_not_corrupted(self): + sql = "SELECT description FROM event" + self.assertEqual(_translated(sql), sql) + + def test_descending_order_not_corrupted(self): + sql = "SELECT handle FROM person ORDER BY surname desc" + self.assertEqual(_translated(sql), sql) + + +# ------------------------------------------------------------------------- +# +# TestCursorTranslatesToo +# +# ------------------------------------------------------------------------- +class TestCursorTranslatesToo(unittest.TestCase): + """Cursor.execute applies the same translations as Connection.execute. + + Unlike core dbapi, shareddbapi passes bound parameters to cursor queries + in order to filter by treeid, so the cursor needs translation as well. + """ + + def test_cursor_translates_placeholders(self): + cursor_obj = Cursor.__new__(Cursor) + inner = mock.MagicMock() + cursor_obj._Cursor__cursor = inner + cursor_obj.execute("SELECT handle FROM person WHERE treeid = ?", [1]) + self.assertEqual( + inner.execute.call_args[0][0], + "SELECT handle FROM person WHERE treeid = %s", + ) + + +# ------------------------------------------------------------------------- +# +# TestSharedPostgreSQLSqlType +# +# ------------------------------------------------------------------------- +class TestSharedPostgreSQLSqlType(unittest.TestCase): + """SharedPostgreSQL._sql_type maps BLOB -> bytea; other types pass through.""" + + def setUp(self): + self.pg = SharedPostgreSQL.__new__(SharedPostgreSQL) + + def test_blob_becomes_bytea(self): + with mock.patch.object(SharedDBAPI, "_sql_type", return_value="BLOB"): + self.assertEqual(self.pg._sql_type("blob_field", 0), "bytea") + + def test_text_unchanged(self): + with mock.patch.object(SharedDBAPI, "_sql_type", return_value="TEXT"): + self.assertEqual(self.pg._sql_type("text_field", 255), "TEXT") + + def test_integer_unchanged(self): + with mock.patch.object(SharedDBAPI, "_sql_type", return_value="INTEGER"): + self.assertEqual(self.pg._sql_type("int_field", 0), "INTEGER") + + +# ------------------------------------------------------------------------- +# +# TestSharedPostgreSQLQuoteColumn +# +# ------------------------------------------------------------------------- +class TestSharedPostgreSQLQuoteColumn(unittest.TestCase): + """SharedPostgreSQL._quote_column returns the physical column names.""" + + def setUp(self): + self.pg = SharedPostgreSQL.__new__(SharedPostgreSQL) + + def test_desc_reserved(self): + self.assertEqual(self.pg._quote_column("desc"), "desc_") + + def test_description_keeps_legacy_name(self): + """Existing databases have desc_ription, created by the old rewrite.""" + self.assertEqual(self.pg._quote_column("description"), "desc_ription") + + def test_normal_column_unchanged(self): + self.assertEqual(self.pg._quote_column("gramps_id"), "gramps_id") + + def test_handle_unchanged(self): + self.assertEqual(self.pg._quote_column("handle"), "handle") + + def test_change_unchanged(self): + self.assertEqual(self.pg._quote_column("change"), "change") + + def test_base_class_is_identity(self): + base = SharedDBAPI.__new__(SharedDBAPI) + self.assertEqual(base._quote_column("desc"), "desc") + + +# ------------------------------------------------------------------------- +# +# TestSecondaryColumnNaming +# +# ------------------------------------------------------------------------- +class TestSecondaryColumnNaming(unittest.TestCase): + """Every Gramps secondary field maps to the column an existing database has. + + Guards against a rename of the two fields whose physical column names were + fixed by the previous substring rewrite. + """ + + def setUp(self): + self.pg = SharedPostgreSQL.__new__(SharedPostgreSQL) + + def test_media_desc_field(self): + from gramps.gen.lib import Media + + fields = [field[0] for field in Media.get_secondary_fields()] + self.assertIn("desc", fields) + self.assertEqual(self.pg._quote_column("desc"), "desc_") + + def test_event_description_field(self): + from gramps.gen.lib import Event + + fields = [field[0] for field in Event.get_secondary_fields()] + self.assertIn("description", fields) + self.assertEqual(self.pg._quote_column("description"), "desc_ription") + + def test_no_other_field_needs_renaming(self): + """Only desc and description were affected by the old rewrite.""" + from gramps.gen.lib import ( + Citation, + Event, + Family, + Media, + Note, + Person, + Place, + Repository, + Source, + Tag, + ) + + affected = set() + for cls in ( + Person, + Family, + Event, + Place, + Repository, + Source, + Citation, + Media, + Note, + Tag, + ): + for field, _type, _length in cls.get_secondary_fields(): + if "desc" in field: + affected.add(field) + self.assertEqual(affected, {"desc", "description"}) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/addon-development/01-overview.md b/docs/addon-development/01-overview.md new file mode 100644 index 000000000..85a906284 --- /dev/null +++ b/docs/addon-development/01-overview.md @@ -0,0 +1,180 @@ +# Addon Development + +[Index](01-overview.md) · [Next →](02-tutorials.md) + +## Overview + +A Gramps **addon** extends the application without modifying core. You add a feature, ship it on your own schedule, and users install it from the in-app Plugin Manager — no fork of Gramps, no waiting on a core release to put new functionality in front of people. An addon is just a folder of Python on the plugin path, so the barrier to entry is low; the trade-off is that you build against Gramps' API and track it across versions. This is how most of Gramps' reports, tools, and gramplets are delivered, and the same door is open to you. + +Addons are discovered from the plugin directory; see [the addon list](https://gramps-project.org/wiki/index.php/6.0_Addons) for what ships today. + +This page is the **start point** for the section: first a map to every other page, then everything a first-time author needs to go from "Gramps is installed" to "my addon shows up in the menu" — anatomy, prerequisites, and a minimal working Gramplet. The normative MUST / SHOULD rules every addon is held to live in [Rules](16-guidelines.md). + +## The section at a glance + +**New to addon development?** Work through this page, then read in order — from your first loaded addon to a tested, rules-compliant one: + +*this page* → [Addon Kinds](03-addon-kinds.md) → [Fundamentals](04-fundamentals.md) → [Data access](05-data-access.md) → [Testing](07-testing.md) → [Rules](16-guidelines.md) + +**Looking for something specific?** Jump straight to it: + +| If you want to… | Go to | +|-----------------|-------| +| Install the tooling and see your first addon load | *this page, below* | +| Follow an end-to-end walkthrough for your addon kind | [Tutorials](02-tutorials.md) | +| Choose which kind of addon to build | [Addon Kinds](03-addon-kinds.md) | +| Learn the cross-cutting basics — `.gpr.py`, discovery, `_()`, logging, lifecycle | [Fundamentals](04-fundamentals.md) | +| Read from or write to the database | [Data access](05-data-access.md) | +| Look up the `gramps.gen` API an addon may import | [API Reference](06-api-reference.md) | +| Write and run tests | [Testing](07-testing.md) | +| Debug an addon that isn't behaving | [Debug](08-debug.md) | +| Diagnose a common failure mode | [Troubleshoot](09-troubleshoot.md) | +| Pass the static checks (Black, ruff) | [Code Analysis](10-code-analysis.md) | +| Translate your addon's strings | [Internationalization](11-internationalization.md) | +| Package and submit your addon | [Packaging](12-packaging.md) | +| List, announce, and support your published addon | [Community](13-community.md) | +| Port across Gramps versions | [Compatibility](14-compatibility.md) | +| See per-version changes that affect addons | [What's New](15-whats-new.md) | +| Know the rules to follow — and to cite in review | [Rules](16-guidelines.md) | +| See what's planned, or propose a change | [Roadmap](17-roadmap.md) | + +The one page to bookmark is [Rules](16-guidelines.md) — the normative MUST / SHOULD / MAY reference every addon is held to. + +## What an addon can extend (at a glance) + +Almost every part of the Gramps UI is a plugin point. The common kinds: + +| Kind | Adds | Shows up in | +|------|------|-------------| +| **Gramplet** | a lightweight widget over the current selection | Dashboard / sidebar | +| **View** | a full alternative way to browse the tree | main view area | +| **Report** | text or graphical output (PDF, HTML, ODF, …) | Reports menu | +| **Tool** | an operation over the database | Tools menu | +| **Importer / Exporter** | reading or writing an external format | File → Import / Export | +| **Quick View** | a one-call report on a selected object | right-click menus | + +…plus filter rules, sidebars, map providers, relationship calculators, citation formatters, docgen output backends, and more. The full catalogue — with the registration fields and base class each kind needs — is [Addon Kinds](03-addon-kinds.md). + +## Anatomy of an addon + +An addon is a folder under Gramps' user plugin directory — one folder per addon — holding at minimum a registration file and an implementation module: + +| File | Purpose | +|------|---------| +| `.gpr.py` | Registration: id, name, version, Gramps target, kind, entry point | +| `.py` | The implementation Gramps loads on demand | +| `po/` | Translation catalogs (optional) | +| `tests/` | Unit tests (optional, recommended) | + +At startup Gramps scans every `.gpr.py` and builds a metadata catalog from the `register(...)` call(s); the implementation module named by `fname` loads **lazily**, on first use. The consequence to remember: an error in `.gpr.py` hides the addon entirely, while an error in the implementation only surfaces when the addon is invoked. + +The registration declares the Gramps version it targets (`gramps_target_version`) — an addon on `maintenance/gramps60` expects the 6.0 API; see [Compatibility](14-compatibility.md) for cross-version concerns. + +What you build next depends on the **kind** — Gramplet, View, Report, Tool, Importer/Exporter, Quick View, and more — each adding its own registration fields and base class. Choose one in [Addon Kinds](03-addon-kinds.md); the full `.gpr.py` field reference and the discovery model are in [Fundamentals](04-fundamentals.md). + +## Prerequisites + +| Requirement | Why | +|-------------|-----| +| Gramps 6.0 installed and runnable | The target you're developing against | +| Python 3.10+ | Matches Gramps 6.0's minimum | +| A text editor or IDE | Any will do; Gramps doesn't impose one | +| Familiarity with Python imports and packages | Addons are Python modules | + +You do **not** need to build Gramps from source for addon work. Addons load from the user plugin directory and are picked up at next start. + +## Where addons live + +Each addon is a folder under Gramps' user plugin directory, one folder per addon. The exact path is platform-specific; see [the Addons page](https://gramps-project.org/wiki/index.php/6.0_Addons) for the canonical locations. The folder name must be a valid Python import name (no spaces — addons share code via `import `); it need **not** match the registration `id`, which is an independent plugin key ([Rules](16-guidelines.md) → Structure). + +On Gramps 6.0, plugin discovery does **not** follow symlinks — the addon must be physically present under the plugin path, so the development loop is copying (or `rsync`ing) from your working tree on save. + +**Changed in 6.1**: plugin discovery follows symlinks (with realpath-based dedup against symlink loops), so you can `ln -s /` into the user plugin directory and edit in place. Windows users: the 6.1 symlink test is skipped on Windows because the platform's symlink behavior is inconsistent without elevated privileges; the `rsync`/copy loop remains the safe default there. (gramps commit `9443dcbb30` on `maintenance/gramps61`.) + +## Your first addon: a minimal Gramplet + +A *Gramplet* is the lightest-weight addon kind — a sidebar widget. Two files are enough. + +### 1. Create the addon folder + +Make a folder named `HelloGramplet` under the user plugin directory. + +### 2. Add the registration file + +Save this as `HelloGramplet/HelloGramplet.gpr.py`: + +```python +register( + GRAMPLET, + id="HelloGramplet", + name=_("Hello Gramplet"), + description=_("A minimal example Gramplet"), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="hellogramplet.py", + gramplet="HelloGramplet", + gramplet_title=_("Hello"), +) +``` + +The `id` is the addon's stable identifier. `fname` is the implementation module. `gramplet` is the class inside it that Gramps will instantiate. + +### 3. Add the implementation + +Save this as `HelloGramplet/hellogramplet.py`: + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.plug import Gramplet + +_ = glocale.get_addon_translator(__file__).gettext + + +class HelloGramplet(Gramplet): + def init(self): + self.set_text(_("Hello from your first Gramplet!")) +``` + +`init()` is the construction hook — Gramps calls it once when the Gramplet is first shown. The `_ = glocale...` line binds the translation function for this module — see [Translation](#translation) below. + +### 4. Restart Gramps + +Plugin discovery happens at startup. After the restart, the new Gramplet appears under *View → Sidebar* (or the Dashboard, depending on view). + +## Reload / test cycle + +There is no hot-reload for addons. The development loop is: + +1. Edit the source. +2. Sync the change into the plugin directory (or work directly there). +3. Restart Gramps. +4. Observe. + +For faster iteration on non-GUI logic, write a `unittest`-based test alongside the addon and run it without launching Gramps — see [Testing](07-testing.md) for the conventions. + +## Translation + +Wrap every user-visible string in `_()` so it can be translated: + +```python +self.set_text(_("Hello from your first Gramplet!")) +``` + +`_` is set up differently in the two files. In `.gpr.py` it is injected by the plugin loader — just use it, never import it. In the implementation module nothing is injected: bind it explicitly at the top of the file, as the walkthrough's `hellogramplet.py` does: + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale + +_ = glocale.get_addon_translator(__file__).gettext +``` + +Translation catalogues live in a per-addon `po/` directory — optional for a first experiment, required for an addon you intend to share; [Internationalization](11-internationalization.md) covers the workflow. + +## Next steps + +- [Tutorials](02-tutorials.md) — end-to-end walkthroughs per addon kind; read a similar addon's source as your second tutorial ([6.0 Addons](https://gramps-project.org/wiki/index.php/6.0_Addons) lists what exists). +- [Addon Kinds](03-addon-kinds.md) — choose the kind of addon to build; registration fields and base class per kind. +- [Fundamentals](04-fundamentals.md) — every `.gpr.py` field, the discovery model, and the lifecycle hooks the implementation overrides. +- [Testing](07-testing.md) — unit-test conventions and the `tests/` package layout. +- [Addons development](https://gramps-project.org/wiki/index.php/Addons_development) — cross-version porting notes and the wider development reference. diff --git a/docs/addon-development/02-tutorials.md b/docs/addon-development/02-tutorials.md new file mode 100644 index 000000000..60a69f3dc --- /dev/null +++ b/docs/addon-development/02-tutorials.md @@ -0,0 +1,594 @@ +# Tutorials + +[← Previous](01-overview.md) · [Index](01-overview.md) · [Next →](03-addon-kinds.md) + + + +## Overview + +End-to-end walkthroughs that take an author from empty folder to working addon. Each tutorial picks one kind, covers registration, implementation, and the reload cycle, and points at the conventions used to test it. + +Read these in order or skip to the one that matches what you're building — they're independent. They assume you've already followed [the getting-started walkthrough in 01-overview](01-overview.md#your-first-addon-a-minimal-gramplet), so we don't re-explain the user plugin directory or the restart cycle. + +| Tutorial | Kind | What it shows | +|---------------------------|---------------|---------------------------------------------------------------------| +| [A live Gramplet](#a-live-gramplet) | `GRAMPLET` | Reading the DB, refreshing on selection change, signal subscriptions | +| [A simple Tool](#a-simple-tool) | `TOOL` | The Tool / ToolOptions pair, opening a dialog, writing in a `DbTxn` | +| [A text Report](#a-text-report) | `REPORT` | The Report / ReportOptions pair, the docgen abstraction, paragraph styles | +| [A Quick View](#a-quick-view) | `QUICKVIEW` | The `run()` entry point, the Simple Access API, context-menu integration | +| [A custom filter Rule](#a-custom-filter-rule) | `RULE` | Subclassing the namespace Rule base, declaring `labels`, `apply_to_one` | + +For the conceptual map, see [01-overview](01-overview.md). For the full inventory of addon kinds and their registration constants, see [03-addon-kinds](03-addon-kinds.md). + +### A note on tutorial-style code + +The implementation modules below show the smallest code that demonstrates each kind. Two things are deliberately omitted to keep the lesson in focus, and both are **required** for shipped addons: + +- A **GPL-2.0-or-later license header** at the top of every `.py` file. Copy the header from any existing addon, or see [16-guidelines → Coding style](16-guidelines.md#coding-style). +- **Type hints** on public functions and methods (Python 3.10+ syntax — `X | None`, `list[X]`). The tutorials skip them for readability; production addons should include them per [16-guidelines → Coding style](16-guidelines.md#coding-style). + +Both are CI-checked on gramps core PRs (Black formats around the license header; `mypy` verifies the type hints); addons-source doesn't gate on them today but the rules apply to addon code regardless. + +## A live Gramplet + +**Goal.** Build a sidebar Gramplet that reads the active person from the database and shows their direct events, refreshing whenever the active person changes or the database is updated. + +The Hello Gramplet from [the overview's walkthrough](01-overview.md#your-first-addon-a-minimal-gramplet) was static text. This one is dynamic — it subscribes to signals and re-reads the DB on each update. + +### Layout + +Two files in a new folder `PersonEvents/`: + +``` +PersonEvents/ +├── PersonEvents.gpr.py +└── personevents.py +``` + +### `PersonEvents/PersonEvents.gpr.py` + +```python +register( + GRAMPLET, + id="PersonEvents", + name=_("Person Events"), + description=_("Lists the active person's direct events."), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="personevents.py", + gramplet="PersonEventsGramplet", + gramplet_title=_("Events"), + height=200, + expand=True, +) +``` + +`height` and `expand` are Gramplet-specific layout fields; the rest are the same registration shape introduced in [04-fundamentals → The `.gpr.py` registration file](04-fundamentals.md#the-gprpy-registration-file). + +### `PersonEvents/personevents.py` + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.plug import Gramplet + +_ = glocale.get_addon_translator(__file__).gettext + + +class PersonEventsGramplet(Gramplet): + """List the active person's direct events; refresh on changes.""" + + def init(self): + """Build the static parts of the UI once.""" + self.set_use_markup(True) + self.set_text(_("No active person.")) + + def db_changed(self): + """Subscribe to DB signals each time the active DB changes.""" + self.connect(self.dbstate.db, "person-update", self.update) + self.connect(self.dbstate.db, "person-delete", self.update) + self.connect(self.dbstate.db, "event-update", self.update) + + def active_changed(self, handle): + """Active person changed — re-render.""" + self.update() + + def main(self): + """Pull events for the active person and render them.""" + person_handle = self.get_active("Person") + if not person_handle: + self.set_text(_("No active person.")) + return + + person = self.dbstate.db.get_person_from_handle(person_handle) + if person is None: + self.set_text(_("Active person not found.")) + return + + lines = [f"{person.gramps_id}\n"] + for event_ref in person.get_event_ref_list(): + event = self.dbstate.db.get_event_from_handle(event_ref.ref) + if event is None: + continue + date = event.get_date_object() + lines.append(f"{event.get_type()} {date}") + + self.set_text("\n".join(lines)) +``` + +### What's new vs. Hello Gramplet + +- **`db_changed()`** subscribes to DB signals. Using `self.connect(...)` (defined on `Gramplet`) instead of `self.dbstate.db.connect(...)` means Gramps tracks the subscription keys for you and disconnects them automatically when the gramplet closes or the DB swaps out. The forgotten-disconnect bug class is gone. +- **`active_changed(handle)`** is called by Gramps when the user selects a different person in the active view. The default does nothing; calling `self.update()` triggers a redraw. +- **`get_active("Person")`** returns the handle of the active person for the current view, or `None`. It honours navigation context — in a Place view it returns the active place, etc. +- **`set_use_markup(True)`** lets `set_text()` interpret Pango markup (``, ``, …); see [Gramplet textual methods](https://gramps-project.org/wiki/index.php/Gramplets_development#Textual_Output_Methods). + +### Try it + +Drop the folder into your user plugin directory (or symlink it if you're on Gramps 6.1+), restart Gramps, open a tree, and add the Gramplet from the sidebar menu. Click around different people — the displayed events should change with the selection. + +For the API surface this tutorial used (handles, refs, `iter_*`, `commit_*`), see [05-data-access](05-data-access.md). For the signal inventory, see [04-fundamentals → Signals](04-fundamentals.md#signals-addons-reacting-to-changes). + +## A simple Tool + +**Goal.** A menu-launched Tool that scans the database for people with no recorded birth date and shows the list in a dialog. + +Tools differ from gramplets in two ways: they're invoked from the Tools menu (not always visible), and they always carry an Options class — even a tool with no options must register an empty `ToolOptions` subclass. + +### Layout + +``` +MissingBirthDates/ +├── MissingBirthDates.gpr.py +└── missingbirthdates.py +``` + +### `MissingBirthDates/MissingBirthDates.gpr.py` + +```python +register( + TOOL, + id="MissingBirthDates", + name=_("Missing Birth Dates"), + description=_("Lists people with no recorded birth date."), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="missingbirthdates.py", + category=TOOL_ANAL, + toolclass="MissingBirthDates", + optionclass="MissingBirthDatesOptions", + tool_modes=[TOOL_MODE_GUI], +) +``` + +`category=TOOL_ANAL` puts the tool under *Tools → Analysis and Exploration*. Other categories (`TOOL_DBPROC`, `TOOL_DBFIX`, …) are listed in [03-addon-kinds → `TOOL`](03-addon-kinds.md#tool). + +### `MissingBirthDates/missingbirthdates.py` + +```python +from gi.repository import Gtk + +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gui.dialog import OkDialog +from gramps.gui.plug import tool + +_ = glocale.get_addon_translator(__file__).gettext + + +class MissingBirthDates(tool.Tool): + """Scan the DB and report people with no recorded birth date.""" + + def __init__(self, dbstate, user, options_class, name, callback=None): + tool.Tool.__init__(self, dbstate, options_class, name) + + db = dbstate.db + missing = [] + for person in db.iter_people(): + birth_ref = person.get_birth_ref() + if birth_ref is None: + missing.append(person) + continue + event = db.get_event_from_handle(birth_ref.ref) + if event is None or event.get_date_object().is_empty(): + missing.append(person) + + if not missing: + OkDialog( + _("Missing Birth Dates"), + _("Every person has a recorded birth date."), + parent=user.uistate.window, + ) + return + + lines = [f"{p.gramps_id}: {p.get_primary_name().get_name()}" + for p in missing] + OkDialog( + _("Missing Birth Dates"), + _("{n} people with no recorded birth date:\n\n{listing}").format( + n=len(missing), + listing="\n".join(lines), + ), + parent=user.uistate.window, + ) + + +class MissingBirthDatesOptions(tool.ToolOptions): + """No options — placeholder required by the tool framework.""" +``` + +### What's new + +- **`tool.Tool.__init__(self, dbstate, options_class, name)`** — the base-class constructor. The body of `__init__` is *where the tool runs*; there's no separate `run()` method for GUI tools. +- **`MissingBirthDatesOptions`** is required even though we have no options. The `register(...)` call names it via `optionclass`, and Gramps would refuse to load the tool without it. +- **`OkDialog`** is the simplest modal report-back surface; for richer output, build a `Gtk.Dialog` directly (see `gramps/plugins/tool/dumpgenderstats.py` for the standard recipe). + +### Writing data + +If your tool *modifies* the database, all writes go inside a `DbTxn`: + +```python +from gramps.gen.db import DbTxn + +with DbTxn(_("Mark unreferenced media private"), db) as trans: + for media in db.iter_media(): + if not db.find_backlink_handles(media.handle): + media.set_privacy(True) + db.commit_media(media, trans) +``` + +The transaction message is user-visible in the Undo history; translate it. See [05-data-access → Mutating data](05-data-access.md#mutating-data) for the full pattern. + +### Try it + +After restart, the tool appears in *Tools → Analysis and Exploration → Missing Birth Dates*. Run it on `example.gramps` to see the dialog. + +## A text Report + +**Goal.** A simple text report that summarises the database — number of people, number of families, count by gender. Produces the same content through PDF, HTML, ODF, or any other docgen-supported format. + +Reports are the heaviest of the everyday addon kinds. Three pieces work together: + +- A **Report** class that knows how to walk the data and emit it as paragraphs and tables, leaving format details to the docgen. +- An **Options** class that defines user-adjustable options and the paragraph / font styles. +- A **registration** call wiring both into the menu. + +### Layout + +``` +DbSummary/ +├── DbSummary.gpr.py +└── dbsummary.py +``` + +### `DbSummary/DbSummary.gpr.py` + +```python +register( + REPORT, + id="DbSummary", + name=_("Database Summary"), + description=_("Produces a short summary of the family tree."), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="dbsummary.py", + category=CATEGORY_TEXT, + require_active=False, + reportclass="DbSummaryReport", + optionclass="DbSummaryOptions", + report_modes=[REPORT_MODE_GUI, REPORT_MODE_CLI], +) +``` + +`category=CATEGORY_TEXT` makes this a text report — Gramps will offer the user the text-output document backends (PDF, ODF, plain text, …). `require_active=False` because a database summary doesn't need a specific active person. + +### `DbSummary/dbsummary.py` + +```python +from collections import Counter + +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.lib import Person +from gramps.gen.plug import docgen +from gramps.gen.plug.report import MenuReportOptions, Report +from gramps.gen.plug.report import stdoptions + +_ = glocale.get_addon_translator(__file__).gettext + + +class DbSummaryReport(Report): + """A text report summarising the database.""" + + def __init__(self, database, options_class, user): + Report.__init__(self, database, options_class, user) + self.set_locale( + options_class.menu.get_option_by_name("trans").get_value() + ) + self._count() + + def _count(self): + """Walk every Person and tally.""" + self.total = 0 + gender_counts = Counter() + surnames = Counter() + for person in self.database.iter_people(): + self.total += 1 + gender_counts[person.get_gender()] += 1 + primary = person.get_primary_name() + surnames[primary.get_primary_surname().get_surname()] += 1 + self.gender_counts = gender_counts + self.unique_surnames = len(surnames) + self.top_surname = ( + surnames.most_common(1)[0] if surnames else (_("(none)"), 0) + ) + + def write_report(self): + """Emit paragraphs into self.doc.""" + self.doc.start_paragraph("DBS-Title") + self.doc.write_text(self._("Database Summary")) + self.doc.end_paragraph() + + self.doc.start_paragraph("DBS-Normal") + self.doc.write_text( + self._("Total persons: {n}").format(n=self.total)) + self.doc.end_paragraph() + + for gender_code, label in [ + (Person.MALE, _("Males")), + (Person.FEMALE, _("Females")), + (Person.UNKNOWN, _("Unknown gender")), + ]: + self.doc.start_paragraph("DBS-Normal") + self.doc.write_text( + self._("{label}: {n}").format( + label=label, + n=self.gender_counts.get(gender_code, 0), + ) + ) + self.doc.end_paragraph() + + self.doc.start_paragraph("DBS-Normal") + self.doc.write_text( + self._("Unique surnames: {n}").format(n=self.unique_surnames) + ) + self.doc.end_paragraph() + + self.doc.start_paragraph("DBS-Normal") + self.doc.write_text( + self._("Most common surname: {name} ({n})").format( + name=self.top_surname[0], n=self.top_surname[1]) + ) + self.doc.end_paragraph() + + +class DbSummaryOptions(MenuReportOptions): + """Options form and default styles for DbSummaryReport.""" + + def add_menu_options(self, menu): + category = _("Report Options") + stdoptions.add_localization_option(menu, category) + + def make_default_style(self, default_style): + # Title style: 18 pt bold sans-serif, centred, header level 1. + font = docgen.FontStyle() + font.set_size(18) + font.set_type_face(docgen.FONT_SANS_SERIF) + font.set_bold(True) + para = docgen.ParagraphStyle() + para.set_header_level(1) + para.set_alignment(docgen.PARA_ALIGN_CENTER) + para.set_font(font) + para.set_description(_("Style used for the title of the report.")) + default_style.add_paragraph_style("DBS-Title", para) + + # Body style: 12 pt serif. + font = docgen.FontStyle() + font.set_size(12) + font.set_type_face(docgen.FONT_SERIF) + para = docgen.ParagraphStyle() + para.set_font(font) + para.set_description(_("Style used for normal report text.")) + default_style.add_paragraph_style("DBS-Normal", para) +``` + +### What's new + +- **Two classes, one file.** The `register()` call points `reportclass` at the Report and `optionclass` at the Options. +- **`self.doc` is not a file.** It's the live document — a docgen backend instance. The report writes paragraphs and text into it regardless of output format. +- **Paragraph style names are prefixed.** Use `DBS-` (or any short prefix unique to your report) on every style name. Reports get composed into Book reports, where every style name has to be unique across all contributing reports. +- **Localisation is explicit.** `stdoptions.add_localization_option` adds the standard "report locale" option to the form; the report reads it with `self.set_locale(...)` and uses `self._()` for strings that should follow the *report's* chosen locale rather than the UI locale. The leading underscore in `self._` is intentional. +- **`MenuReportOptions`** is the convenient base; for a no-options report, override only `add_menu_options` (to add the locale option) and `make_default_style` (to define paragraph styles). + +### Try it + +After restart, the report appears in *Reports → Text Reports → Database Summary*. Run it through any text document backend (PDF, ODF, plain text) to see the same content reformatted by each. + +For more on the docgen abstraction, see [Report Generation](https://gramps-project.org/wiki/index.php/Report_Generation). For richer reports (tables, multiple paragraph levels, graphical reports using `CATEGORY_DRAW`), see [Report API](https://gramps-project.org/wiki/index.php/Report_API). + +## A Quick View + +**Goal.** A right-click action on a person that lists their siblings — brothers and sisters from every family they're a child in. + +Quick Views are the shortest path to a usable report. There's no class to subclass and no options form to maintain — just a `run()` function and the registration. They're written against the **Simple Access API** (`SimpleAccess`, `SimpleDoc`), which trades some power for very little code. + +### Layout + +``` +Siblings/ +├── Siblings.gpr.py +└── siblings.py +``` + +### `Siblings/Siblings.gpr.py` + +```python +register( + QUICKVIEW, + id="Siblings", + name=_("Siblings"), + description=_("Lists the active person's siblings."), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="siblings.py", + category=CATEGORY_QR_PERSON, + runfunc="run", +) +``` + +`category=CATEGORY_QR_PERSON` puts the entry on the person context menu. `runfunc="run"` names the function Gramps calls. The full set of categories is listed in [03-addon-kinds → `QUICKVIEW`](03-addon-kinds.md#quickview). + +### `Siblings/siblings.py` + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.simple import SimpleAccess, SimpleDoc +from gramps.gui.plug.quick import QuickTable + +_ = glocale.get_addon_translator(__file__).gettext + + +def run(database, document, person): + """Display all siblings of the given person.""" + sdb = SimpleAccess(database) + sdoc = SimpleDoc(document) + + sdoc.title(_("Siblings of {name}").format(name=sdb.name(person))) + sdoc.paragraph("") + + table = QuickTable(sdb) + table.columns(_("Person"), _("Gender"), _("Birth date")) + + own_gid = sdb.gid(person) + for family in sdb.child_in(person): + for child in sdb.children(family): + if sdb.gid(child) == own_gid: + continue + table.row(child, sdb.gender(child), sdb.birth_date(child)) + document.has_data = True + + table.write(sdoc) +``` + +### What's new + +- **`run(database, document, person)`** — the function signature is fixed by the QuickView kind. The third argument is the *selected object* of the category (`CATEGORY_QR_PERSON` → person, `CATEGORY_QR_FAMILY` → family, …). +- **`SimpleAccess`** is the high-level read interface — `sdb.children(family)`, `sdb.birth_date(person)`, `sdb.name(person)`. It hides handle dereferencing, refs, and date formatting. For the full surface, see [Simple Access API](https://gramps-project.org/wiki/index.php/Simple_Access_API). +- **`SimpleDoc`** is the matching write interface — `sdoc.title(...)`, `sdoc.paragraph(...)`, `sdoc.header1(...)`. +- **`QuickTable`** builds an interactive table where each row links back to a real Gramps object — clicking a person opens that person. +- **`document.has_data = True`** tells Gramps the report produced output. When all rows are filtered out, the empty-state path triggers instead. + +### Try it + +After restart, right-click any person in the People view or the person editor. *Quick View → Siblings* appears in the menu. The result opens in a Quick View window; clicking a row in the table opens that person. + +For Quick Views that don't fit the Simple Access surface, you can reach for the full DB API — see [05-data-access](05-data-access.md). The two are complementary; a complex Quick View can use both. + +## A custom filter Rule + +**Goal.** A filter rule "Has at least N children" that the user can add to a custom person filter from the Filter Editor. + +Filter rules are the smallest addon kind by line count and the one with the most reuse: a single rule, written once, drops into every filter the user composes — search, narrative website, reports, gramplets that accept a filter. + +### Layout + +``` +HasNChildren/ +├── HasNChildren.gpr.py +└── hasnchildren.py +``` + +### `HasNChildren/HasNChildren.gpr.py` + +```python +register( + RULE, + id="HasNChildren", + name=_("People with at least N children"), + description=_("Matches people who have at least N children."), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="hasnchildren.py", + ruleclass="HasNChildren", + namespace="Person", +) +``` + +`namespace="Person"` says this rule applies to people. The other namespaces (`Family`, `Event`, `Place`, `Source`, `Citation`, `Repository`, `Media`, `Note`) get their own rules — Gramps' filter editor groups rules by namespace. + +### `HasNChildren/hasnchildren.py` + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale +from gramps.gen.filters.rules import Rule + +_ = glocale.get_addon_translator(__file__).gettext + + +class HasNChildren(Rule): + """Matches people with at least N children.""" + + labels = [_("Minimum count:")] + name = _("People with at least N children") + category = _("Family filters") + description = _("Matches people with at least N children") + + def apply_to_one(self, db, person): + try: + minimum = int(self.list[0]) + except (TypeError, ValueError): + return False + total = 0 + for family_handle in person.get_family_handle_list(): + family = db.get_family_from_handle(family_handle) + if family is None: + continue + total += len(family.get_child_ref_list()) + if total >= minimum: + return True + return False +``` + +### What's new + +- **`labels`** declares the user-prompted arguments — one entry per text box in the filter-editor dialog. The user's typed values arrive on `self.list` in the same order. Always parse defensively; `self.list[0]` is a string straight from the GUI. +- **`name`, `category`, `description`** are class attributes — Gramps reads them off the class (no instance needed) when building the Add Rule dialog. `category` is the section the rule appears under in that dialog. +- **`apply_to_one(self, db, person)`** is the per-object hook. It returns `True` for a match, `False` for a non-match. Gramps calls it for every person in the namespace when applying the filter. On Gramps 6.0 the API is `apply_to_one`; older releases used `apply` (see [gramps/gen/filters/rules/_rule.py:162](https://github.com/gramps-project/gramps/blob/maintenance/gramps60/gramps/gen/filters/rules/_rule.py#L162)). + +### Optional hooks + +- **`prepare(self, db, user)`** — called once before the rule is applied to many objects, on demand. Use it to precompute lookup tables when `apply_to_one` would otherwise repeat expensive work. Pair with `reset()` to release memory afterwards. +- **`allow_regex = True`** — opt the first label into regex input. + +### Try it + +After restart, *Edit → Person Filter Editor → Add → Add Rule* shows "People with at least N children" under *Family filters*. The user types a number in the "Minimum count" field; the rule does the rest. + +The rule is also visible from gramplets like *Filter Gramplet* and as an input to any tool or report that accepts a person filter — no extra work needed; rules are uniform across the framework. + +## See also + +- [01-overview → Your first addon](01-overview.md#your-first-addon-a-minimal-gramplet) — the prerequisites and the development loop these tutorials build on. +- [03-addon-kinds](03-addon-kinds.md) — registration details per kind. +- [04-fundamentals](04-fundamentals.md) — `.gpr.py` fields, signals, `requires_mod`, lifecycle hooks. +- [05-data-access](05-data-access.md) — the DB API patterns used by these tutorials. +- [07-testing](07-testing.md) — how to test what you just wrote without launching Gramps. +- [Report API](https://gramps-project.org/wiki/index.php/Report_API), [Report Generation](https://gramps-project.org/wiki/index.php/Report_Generation) — depth on the docgen abstraction. +- [Simple Access API](https://gramps-project.org/wiki/index.php/Simple_Access_API) — the Quick View read surface. diff --git a/docs/addon-development/03-addon-kinds.md b/docs/addon-development/03-addon-kinds.md new file mode 100644 index 000000000..9abcfa056 --- /dev/null +++ b/docs/addon-development/03-addon-kinds.md @@ -0,0 +1,206 @@ +# Addon Kinds + +[← Previous](02-tutorials.md) · [Index](01-overview.md) · [Next →](04-fundamentals.md) + + + +## Overview + +Gramps doesn't have one "addon" shape — it has 14 of them, each registered with a different `register(KIND, …)` constant and each plugged in at a different extension point. This page is the index over all of them, with the registration constant, the UI location, the base class to subclass, and a pointer onward. Use it to answer the first question every prospective addon author asks: **which kind of thing am I writing?** + +Source of truth for the constants: [`gramps/gen/plug/_pluginreg.py`](https://github.com/gramps-project/gramps/blob/maintenance/gramps60/gramps/gen/plug/_pluginreg.py). + +![Fig. 1 — Where each addon kind plugs into the Gramps UI. Menu-anchored kinds (REPORT, TOOL, IMPORT/EXPORT) appear inline with the menu item that hosts them; panel-anchored kinds (SIDEBAR, VIEW, GRAMPLET, QUICKVIEW, MAPSERVICE, RULE) carry callouts to their surface. The six kinds with no direct UI surface — DOCGEN, DATABASE, RELCALC, THUMBNAILER, CITE, GENERAL — are listed separately. Schematic; relative positions match Gramps 6.0's default layout but are not pixel-accurate.](_media/addon-kinds-ui-map.svg) + +## Kinds at a glance + +| Constant | Where it shows up | Typical use | +|----------------|----------------------------------|------------------------------------------------------------------------------| +| `GRAMPLET` | Dashboard, sidebar, bottombar | Lightweight widget over the current selection | +| `VIEW` | Main view area | A full alternative way to browse the tree | +| `REPORT` | Reports menu | Text / graphical output (PDF, HTML, ODF, …) using the docgen interface | +| `TOOL` | Tools menu | Operates on the database, optionally writing inside a transaction | +| `IMPORT` | File → Import | Reads an external format into the tree | +| `EXPORT` | File → Export | Writes the tree to an external format | +| `DOCGEN` | Report output backends | Adds a new output format / paper backend used by reports | +| `QUICKVIEW` | Right-click context menus | Single-call short report on a selected object (formerly `QUICKREPORT`) | +| `SIDEBAR` | Sidebar navigator | Adds a new sidebar category | +| `MAPSERVICE` | Geography view | Adds a new map tile provider | +| `RELCALC` | Relationships view | Per-locale relationship calculator | +| `RULE` | Filter editor | Adds a new filter rule for an object type | +| `DATABASE` | New tree backend selection | Adds support for another database backend | +| `THUMBNAILER` | Media handling | Adds a thumbnail generator for an additional media format | +| `CITE` | Source citations | Adds a citation formatter style | +| `GENERAL` | (varies) | Catch-all for libraries / pluggable categories (`WEBSTUFF`, `Filters`, …) | + +`QUICKREPORT` is the legacy name for `QUICKVIEW`; the integer constant is identical (`gramps/gen/plug/_pluginreg.py` line 83). New addons use `QUICKVIEW`; existing ones continue to work. + +## Per-kind notes + +The notes below cover the kinds an addon author is likely to write. Kinds with deeper conventions get their own section; the rest are summarised in one paragraph each. For full attribute lists per kind, the authoritative reference is the `expand_*` functions in `_pluginreg.py`. + +### `GRAMPLET` + +**Where it shows up:** docked in the Dashboard, sidebar, or bottombar of any view; can be detached into a floating window. + +**Base class:** subclass `gramps.gen.plug.Gramplet`. Override `init()` (constructor hook, runs once), `main()` (re-run on update), `db_changed()` (called when the active database changes), and `active_changed()` (called when the active person / family / etc. changes). + +**Minimum-viable shape:** + +```python +from gramps.gen.plug import Gramplet + +class MyGramplet(Gramplet): + def init(self): + self.set_text(_("Hello")) +``` + +**Registration:** see [01-overview → Add the registration file](01-overview.md#2-add-the-registration-file) for the full call. Required Gramplet-specific fields are `gramplet` (the class name) and `gramplet_title` (the user-visible tab title). + +**Tutorial:** [02-tutorials → A live Gramplet](02-tutorials.md#a-live-gramplet). + +### `REPORT` + +**Where it shows up:** Reports menu, organised by category. + +**Base class:** subclass `gramps.gen.plug.report.Report`. Override `write_report()` to emit content. Pair with an options class that subclasses `gramps.gen.plug.report.MenuReportOptions` and overrides `add_menu_options()` (to define user-adjustable options) and `make_default_style()` (to define paragraph and font styles). + +**Categories** (`_pluginreg.py` L141–L149): `CATEGORY_TEXT`, `CATEGORY_DRAW`, `CATEGORY_CODE`, `CATEGORY_WEB`, `CATEGORY_BOOK`, `CATEGORY_GRAPHVIZ`, `CATEGORY_TREE`. Text and Draw reports go through the docgen abstraction, so the same report can emit PDF / HTML / ODF without per-format code. + +**Report modes** (`report_modes` field): `REPORT_MODE_GUI` (dialog-driven), `REPORT_MODE_BKI` (book item), `REPORT_MODE_CLI` (command line). Most addons combine GUI + CLI. + +**Tutorial:** [02-tutorials → A text Report](02-tutorials.md#a-text-report). + +### `TOOL` + +**Where it shows up:** Tools menu, optionally categorised. + +**Base class:** subclass a class from `gramps.gui.plug.tool` (typically `Tool` or `BatchTool`). Override the constructor — Gramps passes `(dbstate, user, options_class, name, callback=None)`. Tools that mutate the database **must** do so inside a `DbTxn`. + +**Categories** (`_pluginreg.py` L154–L159): `TOOL_DEBUG`, `TOOL_ANAL`, `TOOL_DBPROC`, `TOOL_DBFIX`, `TOOL_REVCTL`, `TOOL_UTILS`. Choose the one that matches what the tool actually does — `TOOL_DBFIX` for repairs, `TOOL_ANAL` for read-only analysis, `TOOL_UTILS` for generic utilities. + +**Tool modes** (`tool_modes` field, `_pluginreg.py` L183–L184): `TOOL_MODE_GUI` and `TOOL_MODE_CLI`. A pure-data tool should support both so a power user can scriptit. + +**Tutorial:** [02-tutorials → A simple Tool](02-tutorials.md#a-simple-tool). + +### `QUICKVIEW` + +**Where it shows up:** right-click context menus on the selected object in views and editors. + +**Entry point:** a `run(database, document, person_or_family_or_…)` function declared in the implementation module and pointed to by the `runfunc` field. No class subclassing required. + +**Categories** (`_pluginreg.py` L163–L174): `CATEGORY_QR_PERSON`, `CATEGORY_QR_FAMILY`, `CATEGORY_QR_EVENT`, `CATEGORY_QR_SOURCE`, `CATEGORY_QR_PLACE`, `CATEGORY_QR_REPOSITORY`, `CATEGORY_QR_NOTE`, `CATEGORY_QR_DATE`, `CATEGORY_QR_MEDIA`, `CATEGORY_QR_CITATION`, `CATEGORY_QR_SOURCE_OR_CITATION`, `CATEGORY_QR_MISC`. The category determines which context menu the entry appears in. + +Quick Views are deliberately the shortest path to a usable report — written against the `gramps.gen.simple` API (`SimpleAccess`, `SimpleDoc`), they hide most of the docgen complexity. Reach for a full `REPORT` only when you need styles, paragraph layout, or multiple output formats. + +**Tutorial:** [02-tutorials → A Quick View](02-tutorials.md#a-quick-view). + +### `RULE` + +**Where it shows up:** the Add Rule dialog when the user composes a custom filter from the Filter Editor; available wherever filters are. + +**Base class:** subclass the right rule base from `gramps.gen.filters.rules` — pick the namespace-specific base (`gramps.gen.filters.rules.person.Rule`, `…family.Rule`, etc.) that matches the object type your rule applies to. Set the class attributes `name`, `description`, `category`, and `labels` (the user-prompted arguments); implement `apply(db, obj)` to return `True` / `False`. + +**Tutorial:** [02-tutorials → A custom filter Rule](02-tutorials.md#a-custom-filter-rule). + +### `VIEW` + +**Where it shows up:** the main view area; available from the navigator once registered. + +**Base class:** subclass an appropriate view from `gramps.gui.views` (`NavigationView`, `ListView`, `PageView`). Views are the heaviest addon kind — they own the entire display surface and the keyboard / mouse interaction. Most addons should reach for `GRAMPLET` instead and only graduate to `VIEW` when the gramplet outgrows its container. + +**Live examples:** `CombinedView`, `LifeLineChartView`, `QuiltView` — read one before writing your own. + +### `IMPORT` / `EXPORT` + +**Where they show up:** File → Import / Export, with the new format appearing in the format dropdown. + +**Entry point:** a module-level function. Importers receive `(database, filename, user)`; exporters receive `(database, filename, error_dialog, option_box, callback)` (signatures vary slightly by Gramps minor; the safest move is to read a live importer/exporter and copy the shape). + +**Live examples:** the GEDCOM (`gramps/plugins/importer/importgedcom.py`, `…/exporter/exportgedcom.py`) and JSON importers/exporters in core are the canonical references. + +### `DOCGEN` + +**Where it shows up:** as a new output format in any Report's options dialog; not user-launched on its own. + +**Base class:** subclass `gramps.gen.plug.docgen.BaseDoc` (or the text/draw subclasses depending on what kind of output you generate). A DocGen implements the *primitives* — paragraphs, tables, drawing commands — that the abstract Report classes call into. Authors usually only write a new DocGen to add a new output format (e.g. a new word-processor file type); it's a relatively rare addon kind. + +### `SIDEBAR` + +**Where it shows up:** the navigator on the left of the main window; each `SIDEBAR` plugin adds one category. + +**Base class:** subclass `gramps.gui.sidebar.Sidebar`. Core categories (People, Families, Events, …) are themselves implemented this way, so the canonical examples ship in core under `gramps/gui/sidebar/`. + +### `MAPSERVICE` + +**Where it shows up:** the Geography views' map-source dropdown. + +**Base class:** subclass `gramps.plugins.lib.maps.osmgps.MapService` and implement the URL / tile-fetch protocol for your provider. Pure tile adapters — no UI changes — so most are very small. + +### `RELCALC` + +**Where it shows up:** wherever Gramps computes a relationship string (Relationships view, person editor, reports). One `RELCALC` plugin per locale. + +**Base class:** subclass `gramps.gen.relationship.RelationshipCalculator`. The base class supplies all the English-language logic; subclasses override the localised strings and any kinship rules specific to the culture being modelled. + +### `DATABASE` + +**Where it shows up:** the database-backend dropdown in tree creation. + +Adds a fully alternative storage backend implementing the `DbReadBase` / `DbWriteBase` interfaces. By far the heaviest kind — the only current in-tree examples are the BSDDB and SQLite backends themselves. Treat the existence of this kind as "yes, it is possible," not "you should consider writing one." + +### `THUMBNAILER` + +**Where it shows up:** wherever Gramps generates a media thumbnail. + +Adds a generator for one additional media format. Pure-function shape: input file → thumbnail image. Use this when a media type Gramps recognises doesn't have a working thumbnailer in your environment. + +### `CITE` + +**Where it shows up:** the citation style chooser in source / citation editors and reports. + +Adds an alternative citation formatter (Chicago, MLA, Evidence Explained, …). Implements the formatting protocol expected by the source / citation code; cite an existing core formatter (`gramps/plugins/cite/`) for the exact shape on the branch you're targeting. + +### `GENERAL` + +**Where it shows up:** nowhere directly — `GENERAL` is the escape hatch for plugin code that doesn't fit any other kind. Two main uses: + +- **Shared libraries** — code reused across multiple addons. Set `load_on_reg=True` and the file gets imported at startup; everything in it becomes importable to other plugins as `import `. The `libwebconnect` addon, depended on by every Web Connect Pack, is the archetype. +- **Pluggable categories** — `GENERAL` plugins can declare a `category` string; other code can then ask the plugin manager for all `GENERAL` plugins of category `WEBSTUFF` (CSS stylesheets for the narrative website report) or `Filters` (filter-rule providers). New categories are rare; the published ones are documented in [addons-development](https://gramps-project.org/wiki/index.php/Addons_development#Registered_GENERAL_Categories). + +The category `WEBSTUFF` is the one most addon authors meet: addons that ship a stylesheet for the narrative website register as `GENERAL, category="WEBSTUFF"` and the website report picks them up automatically. + +**The plugin-data API.** Three registration fields drive the category machinery. A plugin contributes data either statically (`data = [...]` right in the `.gpr.py`) or dynamically — if the implementation module defines a function named `load_on_reg(dbstate, uistate, plugin)`, Gramps calls it at registration and its return value becomes the plugin's data. A `process = "function_name"` field names a function applied over the accumulated data when a consumer asks for it. Consumers query by category through the plugin manager: + +```python +from gramps.gui.pluginmanager import GuiPluginManager + +plugman = GuiPluginManager.get_instance() +plugman.get_plugin_data("WEBSTUFF") # all data from WEBSTUFF plugins +plugman.process_plugin_data("WEBSTUFF") # same, run through the process function +``` + +Note there is **no automatic loading** of `GENERAL` plugins beyond this: without `load_on_reg=True` the module sits unimported until something imports it explicitly. + +## Multiple kinds in one addon + +A single `.gpr.py` can call `register(...)` more than once. The classic case is a report that also registers a Quick View entry for the same underlying logic (`gramps/plugins/quickview/all_events.py` does this for events). Each `register()` call is independent; only the addon folder / `id` and the implementation file(s) are shared. + +## See also + +- [01-overview](01-overview.md) — what an addon is, file roles, first Gramplet end-to-end. +- [02-tutorials](02-tutorials.md) — per-kind walkthroughs. +- [04-fundamentals](04-fundamentals.md) — the cross-cutting concepts every kind relies on, including [the provided environment](04-fundamentals.md#the-provided-environment) every kind inherits from Gramps' startup. +- [`gramps/gen/plug/_pluginreg.py`](https://github.com/gramps-project/gramps/blob/maintenance/gramps60/gramps/gen/plug/_pluginreg.py) — the authoritative definition of all the constants and `expand_*` attribute lists per kind. +- [6.0 Addons](https://gramps-project.org/wiki/index.php/6.0_Addons) — the canonical catalogue of what already exists per kind; reading a similar addon's source is your fastest second tutorial. diff --git a/docs/addon-development/04-fundamentals.md b/docs/addon-development/04-fundamentals.md new file mode 100644 index 000000000..74639de4f --- /dev/null +++ b/docs/addon-development/04-fundamentals.md @@ -0,0 +1,384 @@ +# Fundamentals + +[← Previous](03-addon-kinds.md) · [Index](01-overview.md) · [Next →](05-data-access.md) + + + +## Overview + +The cross-cutting concerns every addon author hits regardless of which kind they're building. If something in a kind-specific page assumes a piece of background, it's described here. + +![Fig. 1 — Plugin discovery and load sequence. Gramps scans the plugin directory at startup, executes each `register()` call into a metadata-only catalog, and loads the implementation module lazily when the user first invokes the addon.](_media/plugin-discovery.svg) + +Note that the catalog → invoke arrow is dashed: addon implementation modules are *not* loaded at startup. The `.gpr.py` is what runs during discovery; the `fname` module only loads on first use. This is why a registration-time error blocks the whole addon from appearing, but a runtime error in the implementation only surfaces when the user triggers it. + +## The `.gpr.py` registration file + +Every addon ships exactly one `.gpr.py` per folder, executed at startup by Gramps' plugin scanner. Its single job is to call `register(...)` one or more times, declaring the addon's *metadata* — what kind it is, what version of Gramps it targets, which implementation module to load on demand. + +The general shape: + +```python +register( + GRAMPLET, # kind (see 03-addon-kinds) + id="HelloGramplet", # stable identifier — folder name + name=_("Hello Gramplet"), # user-visible label + description=_("A minimal example"), + version="1.0.0", # addon version, X.Y.Z + gramps_target_version="6.0", # which Gramps minor this targets + status=STABLE, # STABLE / BETA / EXPERIMENTAL / UNSTABLE + fname="hellogramplet.py", # implementation module + # kind-specific fields go here + gramplet="HelloGramplet", + gramplet_title=_("Hello"), +) +``` + +### Fields every kind needs + +| Field | Meaning | +|-------------------------|----------------------------------------------------------------------------------| +| `id` | Stable plugin key, unique across addons; need **not** match the folder name | +| `name` | User-visible label, translatable | +| `version` | Addon version, dotted `X.Y.Z` | +| `gramps_target_version` | The Gramps minor this targets, e.g. `"6.0"` | +| `status` | `STABLE`, `BETA`, `EXPERIMENTAL`, or `UNSTABLE` | +| `fname` | The implementation module Gramps loads on first use | + +### Fields most kinds want + +- `description` — shown in the Plugin Manager tooltip. +- `authors`, `authors_email` — credit and contact, both lists. +- `maintainers`, `maintainers_email` — only set if different from authors. +- `help_url` — wiki page name; Gramps prepends the base URL and may add a language extension. Don't wrap in `_()` unless you actually want per-language wiki pages. +- `audience` — `EVERYONE` (default), `EXPERT`, or `DEVELOPER`; filters visibility in the Plugin Manager. The constants live at `_pluginreg.py:75-77` — note `EVERYONE`, not `ALL` (an outdated wiki page documents `ALL`; the code has only ever used `EVERYONE`). + +### Kind-specific fields + +Every kind adds its own. A few examples: + +- `GRAMPLET` adds `gramplet` (class or function name), `gramplet_title`, `height`, `expand`, `navtypes`, `force_update`. +- `REPORT` adds `reportclass`, `optionclass`, `category`, `report_modes`, `require_active`. +- `TOOL` adds `toolclass`, `optionclass`, `category`, `tool_modes`. +- `QUICKVIEW` adds `runfunc`, `category`. + +[03-addon-kinds](03-addon-kinds.md) lists the kind-specific fields per kind. The authoritative reference is the `expand_*` helpers in [`_pluginreg.py`](https://github.com/gramps-project/gramps/blob/maintenance/gramps60/gramps/gen/plug/_pluginreg.py). + +### Multiple registrations per file + +A single `.gpr.py` may call `register(...)` more than once — for example a report that also exposes a quick view, or two related gramplets sharing one implementation module. Each call is independent metadata. + +## Plugin discovery + +Gramps walks the plugin path at startup, executes every `.gpr.py` it finds, and builds an in-memory catalog from each `register()` call. The implementation modules pointed to by `fname` are **not** loaded at this point — they're imported lazily on first invocation. This split matters for diagnostics: + +- A `SyntaxError` or import failure in `.gpr.py` makes the addon disappear entirely from menus — the catalog never got an entry for it. +- A failure inside the implementation module surfaces only when the user triggers the addon, with a traceback in the Plugin Manager and the log window. + +### The plugin path + +Plugin folders are searched under each path Gramps was configured to scan — typically the system-wide plugin dir plus the per-user plugin dir. The per-user dir is the safe one to develop in; system locations generally need elevated permissions and shouldn't be edited directly. The exact paths are platform-specific; [the Addons page](https://gramps-project.org/wiki/index.php/6.0_Addons) lists them. + +### Symlinks + +Plugin discovery's symlink handling changed between 6.0 and 6.1: + +- **Gramps 6.0** — symlinks are **not** followed. An addon symlinked in is invisible. Development loop: copy/`rsync` from working tree on save. +- **Gramps 6.1+** — symlinks **are** followed, with realpath-based dedup so cycles terminate. Symlinking the working tree into the user plugin dir works in place. (Gramps commit [`9443dcbb30`](https://github.com/gramps-project/gramps/commit/9443dcbb30) on `maintenance/gramps61`.) The symlink test is skipped on Windows because the platform's symlink behaviour is inconsistent without elevated privileges; on Windows, a physical copy remains the safe approach even on 6.1+. + +Concrete sync recipes live in [01-overview → Where addons live](01-overview.md#where-addons-live). + +## Names Gramps injects into `.gpr.py` + +The `.gpr.py` runs in a scope where several names are *pre-populated* by the plugin loader. You **must not import** them; Gramps puts them there and an `import` masks them with stale bindings. + +| Injected name | Source | +|------------------------------------------------------------------|-------------------------------------| +| `register` | the loader itself | +| `_` (and `ngettext`) | the addon's local translation | +| Kind constants — `GRAMPLET`, `REPORT`, `TOOL`, … | `gramps.gen.plug._pluginreg` | +| Status constants — `STABLE`, `BETA`, `EXPERIMENTAL`, `UNSTABLE` | `_pluginreg.py:62-65` | +| Audience constants — `EVERYONE`, `EXPERT`, `DEVELOPER` | `_pluginreg.py:75-77` | +| Report category constants — `CATEGORY_TEXT`, `CATEGORY_DRAW`, … | `_pluginreg.py:141-149` | +| Tool category constants — `TOOL_DBPROC`, `TOOL_DBFIX`, … | `_pluginreg.py:154-159` | +| Quick View category constants — `CATEGORY_QR_PERSON`, … | `_pluginreg.py:163-174` | +| Report mode constants — `REPORT_MODE_GUI`, `REPORT_MODE_BKI`, … | `_pluginreg.py` | + +In the implementation module, none of these are injected — the rules are normal Python. Import what you need from `gramps.gen.*` there. + +## The provided environment + +The injected names are one half of what Gramps hands an addon; the other half is process-global. An addon — whatever its kind — is a **guest in Gramps' process**: before the first plugin loads, Gramps' startup (`gramps/grampsapp.py`, with `gramps/gen/utils/grampslocale.py` and `gramps/gen/plug/_manager.py`) has already configured the state the addon runs inside. Each item below is a real temptation, because setting it up yourself is exactly what makes a module work *standalone* — and each one either collides with, or silently hijacks, the running application. The rule is uniform: **the app provides this state at runtime; the test root provides it under test; addon modules touch none of it.** + +| Gramps sets up at startup | The tempting mistake | An addon instead | Documented in | +|---------------------------|----------------------|------------------|---------------| +| **GI version pins** — `gi.require_version("Gtk", "3.0")` / `("Gdk", "3.0")` before any plugin loads | Pinning in the addon module or a test file so bare `unittest` imports work | Never pin; addons-source's repo-root `tests/__init__.py` carries the pins (PR 950) | [07-testing → The GTK-pin contract](07-testing.md#the-gtk-pin-contract) | +| **Locale & translation** — `locale.setlocale(LC_ALL, "")`, gettext domain binding, ICU collators | `gettext.install()` (overwrites the builtin `_` app-wide), `locale.setlocale` for date/number formatting, `locale.strcoll` for sorting | Use the injected `_` in `.gpr.py`; `glocale.get_addon_translator(__file__)` in modules; `glocale.sort_key` for collation | [Translation](#translation) below, [11-internationalization](11-internationalization.md) | +| **Root logger & error reporting** — WARNING-level root logger with stderr/file handlers; in the GUI, `GtkHandler` turns ERROR into the error-report dialog | `logging.basicConfig(...)` or `getLogger().setLevel(...)` in a module or test to "see output" — duplicates handlers and reroutes the error dialog for the whole app | A named module-level logger only: `LOG = logging.getLogger(".MyAddon")` | [Logging](#logging) below, [08-debug → Default log levels](08-debug.md#default-log-levels) | +| **`sys.path`** — the plugin manager adds your addon dir *transiently* at import and pops it after | `sys.path.insert(0, os.path.dirname(__file__))` at module level so sibling/vendored imports resolve standalone — inside Gramps the entry is permanent and global, and a generic `utils.py` shadows every other addon's | Rely on the loader's import semantics; under test, the invocation through the package root provides the path | [09-troubleshoot → Imports and namespace traps](09-troubleshoot.md#imports-and-python-namespace-traps) | +| **The GTK main loop & global GTK state** — one `Gtk.main()` loop, the icon theme, screen-wide CSS, `Gtk.Settings` | `Gtk.main()` / `Gtk.main_quit()` around your own dialog (the standalone-script habit); installing app-wide CSS providers or retheming globally | Use Gramps' dialog and windowing machinery; style your own widgets, never the screen | [16-guidelines → Runtime](16-guidelines.md#runtime) | +| **`sys.excepthook`** — logs unhandled exceptions and, on a `HandleError`, flags the DB for check-and-repair at next start | Installing your own hook for "nicer" error handling — disables crash reporting *and* the DB-repair flag app-wide | Let exceptions propagate; log expected failures through your module logger | [08-debug](08-debug.md) | +| **Environment & user paths** — `GRAMPS_RESOURCES`, `PANGOCAIRO_BACKEND` (Windows), the user config/plugin directories | `os.environ[...] = ...` in module or test-module code; computing Gramps paths from `__file__` | Paths come from `gramps.gen.const`; environment setup belongs to the harness (test root, CI) | [07-testing → Running tests locally](07-testing.md#running-tests-locally) | + +The test-side mirror of this table — the repository-root `tests/__init__.py` reproducing the slice of this environment modules under test need, with test runs going through the repo root so it loads — is the contract in [07-testing → The GTK-pin contract](07-testing.md#the-gtk-pin-contract). + +## Translation + +Every user-visible string in the `.gpr.py` and in the implementation goes through `_()`. The function is set up differently in the two files because the `.gpr.py` runs in the injected-name scope. + +**In `.gpr.py`**: just use `_()`. The loader has already wired it. + +```python +register( + GRAMPLET, + id="HelloGramplet", + name=_("Hello"), + description=_("A minimal example"), + ... +) +``` + +**In the implementation module**: opt into the addon's own translation catalog at the top of the file, then use `_()` normally. + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale + +_ = glocale.get_addon_translator(__file__).gettext +``` + +This binds `_` to translations stored in the addon's own `po/` folder rather than Gramps' core catalog. Without this line, `_()` falls back to the core catalog and your addon-specific strings stay in English regardless of UI language. + +### Plurals + +Use `ngettext(singular, plural, n)` whenever a number is being formatted into a string. Languages with non-trivial plural rules (Russian, Polish, …) need both forms to render correctly. + +```python +msg = ngettext("{n} match", "{n} matches", n).format(n=n) +``` + +### Disambiguating contexts + +When the same English word translates differently in different contexts, add a context hint. Gramps' `_()` accepts `_(msg, context)`; the older `pgettext(context, msg)` form also works but the comma form is preferred because the source remains readable as plain English. + +```python +_("Source", "citation") # vs. _("Source", "person attribute") +``` + +## Logging + +Use a module-level logger; never use `print()` for diagnostics. + +```python +import logging + +LOG = logging.getLogger(".".join(__name__.split(".")[-2:])) +# or simply: +LOG = logging.getLogger(__name__) + +LOG.debug("Reached the interesting branch with n=%d", n) +LOG.warning("Skipping malformed event %s", event.gramps_id) +``` + +Log output flows into: + +- **The Gramps log window** (Help → Log) — visible to the user. +- **stderr** when Gramps is launched with `--debug` or with `GRAMPS_DEBUG=1` set. + +See [08-debug](08-debug.md) for how to enable debug levels per logger. + +## Lifecycle hooks + +Every kind has its own entry points; the shape varies, but the pattern is consistent: a small number of named methods that Gramps calls at specific moments, and you override the ones you need. + +### Gramplets + +Subclass `gramps.gen.plug.Gramplet`. The hooks Gramps calls: + +| Method | When | +|----------------------|-----------------------------------------------------------------------------------| +| `init(self)` | Once, on first show. Build the UI here. Don't read the DB yet — it may not be open. | +| `db_changed(self)` | When the active database changes. Reconnect any signals you wired on the old DB. | +| `active_changed(self, handle)` | When the active person / family / etc. changes. Default is to call `update()`. | +| `main(self)` | The work itself. May be a generator — `yield True` to keep going, `yield False` to stop. | +| `update(self)` | Don't override. Calls `main()` for you; you call `update()` to schedule a redraw. | +| `on_load(self)` / `on_save(self)` | When the gramplet's persistent data is loaded / saved. | + +Inside the class, `self.dbstate.db` is your live database, `self.uistate` is the GUI state. See [05-data-access](05-data-access.md) for what you can do with `self.dbstate.db`. + +### Reports + +Subclass `gramps.gen.plug.report.Report`. The constructor receives `(database, options_class, user)`. Override `write_report()` — that's the single hook Gramps calls. Everything else is plumbing you initialise in `__init__`. + +### Tools + +Subclass from `gramps.gui.plug.tool`. The constructor receives `(dbstate, user, options_class, name, callback=None)` and does the work inline (there's no separate `run()` for non-CLI tools). For CLI mode, `tool_modes=[TOOL_MODE_CLI]` triggers a different entry path. + +### Quick Views + +Plain function: `run(database, document, person_or_family_or_…)`. No class to subclass. Point `runfunc` at it in the registration. + +### Importers / Exporters + +Plain function pointed to by `fname` + the kind's entry-point field. Signature varies by kind and minor; reading a live importer/exporter is the most reliable way to lock down the exact shape on your target branch. + +## Signals: addons reacting to changes + +Gramps' database and UI emit *signals* when state changes. Addons that need to stay in sync — gramplets that refresh on data changes, views that follow the selection — `connect()` to those signals. + +### The minimal pattern + +```python +key = self.dbstate.db.connect("person-update", self.cb_person_changed) +# … later, in teardown … +self.dbstate.db.disconnect(key) +``` + +`connect()` returns an opaque key; pass it to `disconnect()` when the addon shuts down or the database changes. Forgetting to disconnect leaves stale callbacks pointing into freed objects and crashes Gramps sooner or later. + +### The signals that matter most + +| Source | Signal | When | +|------------------------|--------------------------------------------------|--------------------------------------------------------------------| +| `dbstate.db` | `person-add`, `family-add`, `event-add`, … | One object added. Arg: list of handles. | +| `dbstate.db` | `person-update`, `family-update`, … | One object updated. Arg: list of handles. | +| `dbstate.db` | `person-delete`, `family-delete`, … | One object deleted. Arg: list of handles. | +| `dbstate.db` | `person-rebuild`, `family-rebuild`, … | Mass change (import, db repair). No args. | +| `dbstate.db` | `home-person-changed` | Home person changed. No args. | +| `dbstate` | `database-changed` | Active database swapped. Arg: the new db. | +| `dbstate` | `no-database` | No db is open. | +| `uistate` | `nameformat-changed`, `filter-name-changed`, … | Various UI preferences. | +| view's history | `active-changed` | Selected object changed. Arg: the new handle. | + +Pattern: `person-update` / `family-update` / etc. fire one *after* a transaction commits, with a *list* of affected handles. They never fire mid-transaction, so callbacks can safely re-read the DB. + +### Subscribing to "anything changed" + +A common gramplet pattern is "redraw on any structural change to the tree", typically done by wiring `db_changed`: + +```python +def db_changed(self): + self.dbstate.db.connect("person-add", self.update) + self.dbstate.db.connect("person-delete", self.update) + self.dbstate.db.connect("person-update", self.update) + self.dbstate.db.connect("family-add", self.update) + self.dbstate.db.connect("family-delete", self.update) + self.dbstate.db.connect("family-update", self.update) +``` + +For complex subscriptions across many object types, the `CallbackManager` in `gramps.gen.utils.callman` is a higher-level filter that lets you register dictionaries of `{signal: handler}` and tracks keys for `disconnect_all()` on teardown. See [Signals and callbacks](https://gramps-project.org/wiki/index.php/Signals_and_Callbacks) for the full inventory. + +### Signal ordering + +Signals are deferred until a transaction commits and are emitted in a specific order: deletes first, then adds, then updates; within each phase, by object type in the order persons → families → sources → events → media → places → repositories → notes → tags → citations. This deterministic order matters when a single transaction touches related objects (a family merge deletes one family and updates another plus its members); a handler that re-reads the DB on `person-delete` will see a consistent state. + +## Reading and writing the database + +The DB API is covered in depth in [05-data-access](05-data-access.md). The rule worth stating here, where every addon meets it: + +- **Reading** is unrestricted. Any addon may read freely from `self.dbstate.db`. +- **Writing** goes through a transaction. Always: + + ```python + with DbTxn(_("Description for Undo history"), db) as trans: + person = db.get_person_from_handle(handle) + person.set_privacy(True) + db.commit_person(person, trans) + ``` + +The transaction message is user-visible in the Undo history; translate it. + +## Declaring dependencies + +Addons may need Python packages or system tools that aren't part of Gramps' core dependencies. Declare these in the registration so the plugin manager can surface a clear "missing X" message instead of a generic import failure. + +### `requires_mod` — Python modules + +```python +requires_mod = ["PIL", "lxml"] +``` + +Uses the **importable** module name (what you `import`), **not** the PyPI distribution name. PIL not Pillow, lxml fine either way (matches), yaml not PyYAML. Verify before you push: + +```python +from importlib.util import find_spec +assert find_spec("PIL") is not None +``` + +A mismatch shows up the first time the addon's tests run against a clean install — the import fails. Always verify the name with `find_spec` before publishing. + +### `requires_gi` — GObject Introspection bindings + +```python +requires_gi = [("GExiv2", "0.10")] +``` + +A list of `(namespace, version)` tuples. The user has to install these through their OS package manager; Gramps cannot install GI bindings. The version pin must match what your code actually imports — and on gramps61 the version handling for GExiv2 was rewritten (addons-source PR 829), so a `requires_gi` pinned for one branch isn't guaranteed correct on the other. Verify against the target branch's related code before assuming a cherry-pick is correct. + +### `requires_exe` — Executables on PATH + +```python +requires_exe = ["graphviz", "dot"] +``` + +External binaries the user must have installed. Gramps checks PATH for them and surfaces a missing-dependency message. + +### `depends_on` — Other addons + +```python +depends_on = ["libwebconnect"] +``` + +Other addons that must load first. The plugin manager resolves these automatically when the user installs your addon. Circular dependencies break the load and disable the addon — the loader chooses safety over guessing. + +## Configuration and persistent settings + +For settings that should survive between sessions, Gramps' configuration manager handles the file I/O and migration; you only declare the keys. + +```python +from gramps.gen.config import config as configman + +config = configman.register_manager("my_addon") +config.register("section.key1", default_value) +config.register("section.key2", another_default) +config.load() # read existing settings file, if any +config.save() # write defaults out if the file didn't exist +``` + +`config.get("section.key1")` and `config.set("section.key1", value)` read and write at runtime. Gramplets persist via the lifecycle hook: + +```python +def on_save(self): + config.save() +``` + +The settings file lives in the addon's plugin folder by default. For a system-wide config (rare): + +```python +config = configman.register_manager("my_addon", use_config_path=True) +``` + +Other code — another addon, a repro script — can read an addon's settings without re-registering the keys, via `get_manager`: + +```python +from gramps.gen.config import config as configman + +config = configman.get_manager("my_addon") +value = config.get("section.key1") +``` + +## See also + +- [01-overview → Your first addon](01-overview.md#your-first-addon-a-minimal-gramplet) — the first end-to-end Gramplet putting these concepts together. +- [03-addon-kinds](03-addon-kinds.md) — what each kind adds to the registration shape described here. +- [05-data-access](05-data-access.md) — the DB API surface. +- [06-api-reference](06-api-reference.md) — the curated `gramps.gen.*` surface that addons may import. +- [09-troubleshoot](09-troubleshoot.md) — what failure modes look like when one of these conventions is off. +- [Signals and Callbacks](https://gramps-project.org/wiki/index.php/Signals_and_Callbacks) — the standalone wiki page covering signals and the `CallbackManager` in more depth. diff --git a/docs/addon-development/05-data-access.md b/docs/addon-development/05-data-access.md new file mode 100644 index 000000000..3697c9439 --- /dev/null +++ b/docs/addon-development/05-data-access.md @@ -0,0 +1,229 @@ +# Data access + +[← Previous](04-fundamentals.md) · [Index](01-overview.md) · [Next →](06-api-reference.md) + + + +## Overview + +Every addon that does anything useful with a family tree reads or writes through the **database API** — the `DbReadBase` / `DbWriteBase` interface implemented by Gramps' database backends (BSDDB historically, SQLite from 6.0 onward). + +You don't instantiate a database yourself. The plugin loader hands you a `DbState` object; the live database is `dbstate.db`. Everything below is methods on that handle. + +```python +db = dbstate.db # this is your entry point +``` + +The same `db` works for read-only addons (reports, gramplets, quick views) and for tools that mutate data. Mutation goes through transactions; see [Mutating data](#mutating-data) below. + +## Identifying objects: handles vs Gramps IDs + +Every primary object (Person, Family, Event, Place, Source, Citation, Repository, Media, Note, Tag) has **two identifiers**: + +| Identifier | Stable | Format | Used for | +|------------|--------|--------|----------| +| **Handle** | Yes (internal, never reused) | 32-char hex string | Cross-references in the database | +| **Gramps ID** | User-renameable | `I0001`, `F0001`, `E0001`, ... | User-visible labels and external interop | + +**Rule of thumb:** use handles inside your code; show Gramps IDs to the user. Handles never change; Gramps IDs do (the user can edit them, the "Reorder Gramps IDs" tool can rewrite them in bulk). + +```python +# Right: traverse by handle +person = db.get_person_from_handle(handle) + +# Right: show the user a Gramps ID +print(f"Working on {person.gramps_id}") + +# Wrong: traverse by Gramps ID (works, but slower and breaks under reorder) +person = db.get_person_from_gramps_id("I0001") +``` + +Each object class has both lookup methods (`get__from_handle` and `get__from_gramps_id`); see [06-api-reference](06-api-reference.md) for the full list. + +## Reading: one object at a time + +The fastest pattern, when you have a handle in hand: + +```python +person = db.get_person_from_handle(person_handle) +family = db.get_family_from_handle(family_handle) +event = db.get_event_from_handle(event_handle) +``` + +Each returns `None` if the handle isn't in the database (deleted, broken reference). Always guard: + +```python +person = db.get_person_from_handle(handle) +if person is None: + return # silently skip, or raise a HandleError if the caller expects one +``` + +For `HandleError` and friends, import from `gramps.gen.errors`. + +## Reading: iterating all objects + +For reports and surveys you'll want every object of a given type. The database exposes one generator per object class: + +```python +for person in db.iter_people(): + ... + +for family in db.iter_families(): + ... +``` + +These are **generators**, not lists — they stream through the database without loading everything into memory. Don't call `list(db.iter_people())` on a 50,000-person tree unless you have a reason. + +To iterate just the handles (cheaper when you only need to count or filter): + +```python +for handle in db.iter_person_handles(): + ... +``` + +Counts come without iteration: + +```python +db.get_number_of_people() +db.get_number_of_families() +db.get_number_of_events() +``` + +## Following references + +![Fig. 1 — Gramps primary objects and the most-traversed relationships. Edges labelled `Ref` (e.g. `EventRef`, `CitationRef`, `MediaRef`) go through a ref object that carries metadata such as the role or relationship; bare-labelled edges are direct handle references. Notes and Tags can be attached to any primary object and are omitted to keep arrows readable. Reverse traversals — "who refers to this object?" — go through `db.find_backlink_handles()` instead of these forward links; see Backlinks below.](_media/data-model.svg) + +Most addons don't visit objects in isolation — they follow the relationships between them. Gramps' object model exposes references as **handle lists** on the parent object. + +Person → families they're a parent in: + +```python +for family_handle in person.get_family_handle_list(): + family = db.get_family_from_handle(family_handle) + ... +``` + +Person → events: + +```python +for ref in person.get_event_ref_list(): + event = db.get_event_from_handle(ref.ref) + role = ref.get_role() + ... +``` + +`event_ref` carries more than the handle — also the role (Primary, Witness, etc.) and any private flag. Read the ref, then dereference if you need the event itself. + +Family → children: + +```python +for child_ref in family.get_child_ref_list(): + child = db.get_person_from_handle(child_ref.ref) + ... +``` + +The full handle-list / ref-list inventory per object class lives in the [Gramps API docs](https://gramps-project.org/wiki/index.php/Gramps_6.0_Developer_Reference); see also [06-api-reference](06-api-reference.md) for the addon-facing subset. + +## Backlinks: who refers to this object? + +The forward direction (person → events they participated in) lives on the object. The reverse direction (event → people who participated in it) lives on the database: + +```python +for (obj_type, obj_handle) in db.find_backlink_handles(event.handle): + if obj_type == "Person": + person = db.get_person_from_handle(obj_handle) + ... +``` + +`find_backlink_handles` returns `(class_name, handle)` tuples for every primary object that references the given handle. Use it for: + +- Finding all sources that cite a given place +- Finding all people present at a given event +- Detecting orphaned objects (no backlinks → unreferenced) + +Note that `obj_type` is the **class name as a string** (`"Person"`, `"Family"`, ...), not the Python class itself. + +## Filters + +For non-trivial selection (e.g. "all people born in Hamburg between 1850 and 1900"), use Gramps' filter framework rather than hand-rolling predicates: + +```python +from gramps.gen.filters import GenericFilterFactory + +GenericFilter = GenericFilterFactory("Person") +filt = GenericFilter() +filt.add_rule(SomeRule([arg1, arg2])) +handles = filt.apply(db, db.iter_person_handles()) +``` + +Filters compose, cache, and integrate with the GUI's filter sidebar — a report that defines its own filter gets it as a sidebar option for free. The rule catalogue lives under `gramps.gen.filters.rules`; the user-facing counterpart is documented in [Filters](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Filters). + +## Mutating data + +Write addons (mainly tools) modify data through **transactions**. The pattern is always the same: + +```python +with DbTxn(_("Tool name: what it did"), db) as trans: + person = db.get_person_from_handle(handle) + person.set_privacy(True) + db.commit_person(person, trans) +``` + +Three things matter: + +1. **The transaction message is user-visible** in the Undo History. Make it descriptive and translated. +2. **Always `db.commit_(obj, trans)`** after mutating — the object is a copy; commit writes it back. +3. **Group related changes** in one transaction so the user can undo as a single step. + +Creating a new object follows the same shape: + +```python +from gramps.gen.lib import Person, Name + +with DbTxn(_("Add unknown spouse"), db) as trans: + person = Person() + name = Name() + name.set_surname(surname) + person.set_primary_name(name) + person.gramps_id = db.find_next_person_gramps_id() + db.add_person(person, trans) +``` + +`find_next__gramps_id()` allocates an unused ID; `add_()` inserts and assigns the handle. + +## Testing data access + +Two complementary approaches: + +- **Real-data tests** — load `example.gramps` (shipped with Gramps, canonical test fixture) and exercise your code against it. Best for catching real-world data quirks (cross-typed backlinks, ID normalisation, unusual character sets). See [07-testing](07-testing.md). +- **Mocked tests** — substitute the database with a stub that returns fixed objects. Best for tight unit-test loops that don't need a database on disk. + +The lesson, learned the hard way: mocked DB tests can pass while the real-DB code is broken, because the mock doesn't reproduce the cross-typed backlinks and ID quirks of a populated tree. Prefer example.gramps for anything that traverses the DB; reserve mocks for pure helpers. + +## Performance notes + +The database API is fast enough that most addons don't need to think about performance. When you do: + +- Iterating **handles** is cheaper than iterating **objects** — only dereference when you need the object's contents. +- `get_number_of_()` is O(1); `len(list(db.iter_()))` is O(n). +- Backlinks aren't free — they read an index but still scan it. Don't call `find_backlink_handles` in a tight inner loop. +- The 5.x → 6.0 SQLite backend is roughly comparable to BSDDB for reads, faster for writes. Avoid backend-specific assumptions; addons should work on either. + +## See also + +- [04-fundamentals](04-fundamentals.md) — the plugin lifecycle that wraps this DB access in +- [06-api-reference](06-api-reference.md) — the addon-facing API surface +- [07-testing](07-testing.md) — testing strategies, real-data vs mocks +- [Using database API](https://gramps-project.org/wiki/index.php/Using_database_API) — the standalone wiki reference, covers backends and internals in more depth +- [Gramps Developer Reference](https://gramps-project.org/wiki/index.php/Gramps_6.0_Developer_Reference) — the full API docs diff --git a/docs/addon-development/06-api-reference.md b/docs/addon-development/06-api-reference.md new file mode 100644 index 000000000..fec5d9b04 --- /dev/null +++ b/docs/addon-development/06-api-reference.md @@ -0,0 +1,209 @@ +# API Reference + +[← Previous](05-data-access.md) · [Index](01-overview.md) · [Next →](07-testing.md) + + + +## Overview + +The curated `gramps.gen.*` surface addons are allowed to import. `gen` is the self-contained core submodule (it must not import from `gui` or `plugins`); importing only from `gen` keeps an addon portable across UI variants and testable without a display. + +This page is a navigator, not a generated API dump. For exhaustive signatures, read the source of the module referenced — the [upstream Sphinx docs](https://gramps-project.org/docs/) carry the same information formatted for browsing. + +## Allowed surface + +### Database + +| Module / class | Notes | +|-------------------------------------------|--------------------------------------------------------------------| +| `gramps.gen.db.base.DbReadBase` | Read-only DB interface — what addon code typically receives | +| `gramps.gen.db.base.DbWriteBase` | Mutation interface; reach via `db` after `with DbTxn(...) as trans`| +| `gramps.gen.db.txn.DbTxn` | Transaction context manager — required for every write | +| `gramps.gen.db.utils.open_database` | Open a tree by path; used in repro scripts and tests | +| `gramps.gen.db.exceptions` | DB-layer exception hierarchy | + +See [05-data-access](05-data-access.md) for the addon-facing patterns that use this surface. + +### Object model + +| Module | Notes | +|-------------------------------------|------------------------------------------------------------------------| +| `gramps.gen.lib` | Every primary class: `Person`, `Family`, `Event`, `Place`, `Source`, `Citation`, `Repository`, `Media`, `Note`, `Tag`. Plus value classes (`Name`, `Date`, `Address`, `Surname`, `EventRef`, …). | +| `gramps.gen.lib.person.Person` | The gender constants (`Person.MALE`, `Person.FEMALE`, `Person.UNKNOWN`) are class attributes | + +The full inventory is large; the cheapest reference is the source under [`gramps/gen/lib/`](https://github.com/gramps-project/gramps/tree/maintenance/gramps60/gramps/gen/lib). Every primary class has matching `get_*` / `set_*` accessors; relationships are exposed as handle lists (`get_family_handle_list`) or ref lists (`get_event_ref_list`, `get_child_ref_list`). + +### Types and IDs + +| Module | Notes | +|-----------------------|---------------------------------------------------------------------------------------------| +| `gramps.gen.types` | `PersonHandle`, `FamilyHandle`, …, `PersonGrampsID`, `FamilyGrampsID`, … | + +Prefer these over bare `str` in addon code that handles either kind of identifier. It documents intent for the next reader and makes mistakes (handle vs ID) catchable with `mypy`. See [16-guidelines → Coding style](16-guidelines.md#coding-style). + +### Errors + +| Module | Use | +|-------------------------------------|----------------------------------------------------------------------| +| `gramps.gen.errors` | Raise existing exceptions here before inventing new classes | +| `gramps.gen.errors.HandleError` | Invalid or missing handles | +| `gramps.gen.db.exceptions` | DB-layer-specific exceptions | + +### Plugin base classes + +| Class / module | Used by | +|---------------------------------------------------------|------------------------------------| +| `gramps.gen.plug.Gramplet` | `GRAMPLET` addons | +| `gramps.gen.plug.report.Report` | `REPORT` addons | +| `gramps.gen.plug.report.MenuReportOptions` | Options form for report addons | +| `gramps.gen.plug.report.stdoptions` | Pre-built options like locale chooser | +| `gramps.gen.plug.docgen.BaseDoc` | `DOCGEN` addons (base) | +| `gramps.gen.plug.docgen.TextDoc` | Text reports | +| `gramps.gen.plug.docgen.DrawDoc` | Graphical (drawing) reports | +| `gramps.gen.plug.docgen.GVDoc` | Graphviz-based reports | +| `gramps.gen.plug.docgen.FontStyle`, `ParagraphStyle` | Style definitions for text reports | +| `gramps.gen.plug.docgen.PaperStyle`, `PaperSize` | Page geometry for graphical reports | +| `gramps.gen.plug.menu` | Options-form widgets | +| `gramps.gen.filters.rules.Rule` (and namespace bases) | `RULE` addons | +| `gramps.gen.simple.SimpleAccess`, `SimpleDoc` | Quick Views | + +Most `gramps.gui.*` classes are *internal*; addons that import from there will break across Gramps versions. The exceptions used in this manual's tutorials — `gramps.gui.plug.tool.Tool`, `gramps.gui.plug.quick.QuickTable`, `gramps.gui.dialog.OkDialog` — are documented because every existing Tool / Quick View in core uses them, but they are nevertheless GUI-coupled. Pure logic factored out into modules that import only from `gen` stays unit-testable without a display. + +### Report categories + +For `REPORT` addons, register with one of these category constants (see [`_pluginreg.py:141-149`](https://github.com/gramps-project/gramps/blob/maintenance/gramps60/gramps/gen/plug/_pluginreg.py)): + +| Category | Docgen interface | Notes | +|------------------------|------------------------|--------------------------------------------------------| +| `CATEGORY_TEXT` | `TextDoc` | Text reports — PDF, HTML, ODF, plain text | +| `CATEGORY_DRAW` | `DrawDoc` | Graphical reports drawn at exact coordinates | +| `CATEGORY_GRAPHVIZ` | `GVDoc` | Graphviz / DOT input — laid out by graphviz | +| `CATEGORY_WEB` | (direct file I/O) | Narrative website — writes HTML/CSS directly to files | +| `CATEGORY_BOOK` | `TextDoc` + `DrawDoc` | A composition of Text and Draw reports | +| `CATEGORY_TREE` | `DrawDoc` | Genealogical tree-chart layouts | +| `CATEGORY_CODE` | (none) | Catch-all for reports that don't fit elsewhere | + +Only `CATEGORY_TEXT` and `CATEGORY_DRAW` participate in `CATEGORY_BOOK`. + +### Document API: structure at a glance + +The three docgen interfaces have distinct hierarchies. Knowing which container nests what saves a long trip through the source. + +**`TextDoc` — sequential text layout, paginated by the backend.** + +``` +Document +├── Paragraph +├── Pagebreak +├── Table +│ └── Row +│ └── Cell +│ ├── Paragraph +│ └── Image +└── Image +``` + +Paragraph styles drive titles, body text, list entries. The backend or external viewer handles pagination, except where a manual `Pagebreak` is inserted. Index marks attach to text within a paragraph (`gramps.gen.plug.docgen.IndexMark`), feeding the table of contents in Book reports. + +**`DrawDoc` — exact-coordinate graphics on a frame.** + +``` +Document +└── Frame + ├── Line + ├── Polygon + ├── Box + └── Text +``` + +The frame is the drawing surface; elements get placed by coordinates supplied by the report. The origin is the top-left of the usable area (page minus margins). Graphical reports need to honour `PaperStyle.get_usable_width()` / `…_height()` — drawing into the margins is a contract violation. + +**`GVDoc` — graphviz model.** + +``` +Document +└── Subgraph + ├── Node + ├── Link + └── Comment +``` + +The report defines nodes, links, and comments; layout is the external graphviz binary's job. This is why `requires_exe=["dot"]` appears on Graphviz-based addons. + +### Paper geometry (Draw / Tree only) + +`gramps.gen.plug.docgen.PaperStyle` holds: + +- the paper size (a `PaperSize` instance), +- margins, +- orientation (portrait / landscape). + +Convenience accessors `get_usable_width()` and `get_usable_height()` return the drawing-area dimensions (paper size minus margins, in orientation order — width is always horizontal). Text reports don't need to read these; the backend paginates around them. + +### Locale and translation + +| Module / class | Use | +|-------------------------------------------------------------|------------------------------------------------------------| +| `gramps.gen.const.GRAMPS_LOCALE` (alias `glocale`) | The live locale; entry point for `_()` injection | +| `glocale.get_addon_translator(__file__).gettext` | Bind `_` to the addon's own `po/` catalog | +| `gramps.gen.utils.grampslocale.GrampsLocale` | Instantiate directly to pin a locale in repro scripts | +| `glocale.translation.ngettext` | Plural-aware translation | +| `glocale.translation.sgettext` | Strip translator-hint prefix; used with `"hint | msg"` form| + +See [04-fundamentals → Translation](04-fundamentals.md#translation) for the addon-side opt-in, and [08-debug → Reproduction scripts that bypass the GUI](08-debug.md#reproduction-scripts-that-bypass-the-gui) for the `GrampsLocale(localedir, languages)` pattern in repros. + +### Filters and selection + +| Module / class | Use | +|---------------------------------------------------------|------------------------------------------------| +| `gramps.gen.filters.GenericFilterFactory` | Construct a filter for a namespace | +| `gramps.gen.filters.rules` | The rule catalogue (one subpackage per namespace) | +| `gramps.gen.filters.rules..Rule` | Base class to subclass when writing a custom rule (see [02-tutorials](02-tutorials.md#a-custom-filter-rule)) | + +The modern rule entry point is `apply_to_one(db, obj)` (see [`_rule.py:162`](https://github.com/gramps-project/gramps/blob/maintenance/gramps60/gramps/gen/filters/rules/_rule.py#L162)). Older code used `apply()`. + +### Logging + +| Module / class | Use | +|---------------------------|----------------------------------------------------| +| `logging.getLogger(__name__)` | Module-level logger; see [04-fundamentals → Logging](04-fundamentals.md#logging) | + +There's nothing addon-specific to import here; addons use stdlib `logging` exactly like Gramps' own modules do. + +### Simple Access (Quick Views) + +| Class | Use | +|---------------------------------------------|----------------------------------------------------------------| +| `gramps.gen.simple.SimpleAccess` | High-level DB read interface — hides handles and refs | +| `gramps.gen.simple.SimpleDoc` | Matching write interface — `title`, `paragraph`, `header1`, … | +| `gramps.gui.plug.quick.QuickTable` | Clickable result table (GUI-coupled; QuickView-only) | + +See [02-tutorials → A Quick View](02-tutorials.md#a-quick-view) for the standard pattern. + +## What's NOT API + +Anything under `gramps.gui.*` or `gramps.plugins.*` is internal to the shipped distribution; addons that import from there break across Gramps versions. The exceptions (Tool / Quick View / Dialog) are documented above and unavoidable for those addon kinds, but pure logic should be factored out behind a `gen.*`-only boundary so it stays unit-testable without a display. + +If you find yourself reaching into `gramps.gui.*` or `gramps.plugins.*` for something that *isn't* tied to GUI display, the right move is usually to ask upstream to promote what you need into `gen`. The [committing policies wiki page](https://www.gramps-project.org/wiki/index.php/Committing_policies) and the gramps-devel mailing list are the channels. + +## See also + +- [03-addon-kinds](03-addon-kinds.md) — which kinds use which base classes. +- [04-fundamentals](04-fundamentals.md) — the cross-cutting concepts (logging, translation, signals) backed by this surface. +- [05-data-access](05-data-access.md) — patterns over the DB API. +- [14-compatibility](14-compatibility.md) — what changes across Gramps versions in this surface. +- [Report API](https://gramps-project.org/wiki/index.php/Report_API), [Report Generation](https://gramps-project.org/wiki/index.php/Report_Generation) — standalone wiki references for the docgen subsystem. +- [Simple Access API](https://gramps-project.org/wiki/index.php/Simple_Access_API) — the standalone wiki page for `SimpleAccess` / `SimpleDoc`. +- [Gramps Developer Reference](https://gramps-project.org/docs/) — upstream Sphinx-generated API docs. diff --git a/docs/addon-development/07-testing.md b/docs/addon-development/07-testing.md new file mode 100644 index 000000000..907eb69ef --- /dev/null +++ b/docs/addon-development/07-testing.md @@ -0,0 +1,302 @@ +# Testing + +[← Previous](06-api-reference.md) · [Index](01-overview.md) · [Next →](08-debug.md) + + + +## Overview + +How to test an addon without launching the GUI on every iteration — the test framework, the layout conventions, the fixtures that work, and the platform-aware rules that keep tests portable across Linux, Windows, and Mac. + +A working test suite is what makes an addon **maintainable across Gramps releases**. The matrix of (Gramps version × OS) makes manual testing impossible at scale; the per-OS prefix conventions below let a single CI matrix verify your addon against every supported combination automatically. + +## Framework: stdlib `unittest` + +Use stdlib `unittest`. Don't use pytest. + +Gramps itself standardises on `unittest` (subclasses of `unittest.TestCase`), which keeps addon tests contributable upstream without a framework-conversion step. Mixing pytest features (fixtures, parametrise, plugins) breaks contribution upstream where pytest isn't installed. + +```python +import unittest + + +class MyAddonTests(unittest.TestCase): + def test_handles_empty_input(self): + # ... + self.assertEqual(result, expected) + + +if __name__ == "__main__": + unittest.main() +``` + +### Class header convention + +The "class header navigation comment" rule from gramps' AGENTS.md is unconditional — it applies to `unittest.TestCase` subclasses too. PR 2326 round 2 caught the omission: + +```python +# ------------------------------------------------------------ +# +# MyAddonTests +# +# ------------------------------------------------------------ +class MyAddonTests(unittest.TestCase): + ... +``` + +## Layout + +Each addon ships its tests in a `tests/` subpackage: + +``` +MyAddon/ +├── MyAddon.gpr.py +├── MyAddon.py +└── tests/ + ├── __init__.py # marker — see below + └── test_myaddon.py +``` + +### Why `tests/__init__.py` exists + +The marker is **hygiene, not a bug fix**. Python 3.3+'s implicit namespace packages (PEP 420) mean a directory without `__init__.py` is still importable; dotted-path loading (`python3 -m unittest MyAddon.tests.test_myaddon`) works either way. But: + +1. **Explicit beats implicit.** "It works" is currently true by accident of invocation. The same code breaks the moment something uses `discover` or assumes regular packages. +2. **Explicit — and empty.** Suite-wide test setup (the GI version pins, warning filters) lives at the *repository* root's `tests/__init__.py`, not per addon — see the next section. The per-addon marker stays empty; it is packaging hygiene, and a home for genuinely addon-local setup only if one ever appears. + +The convention crystallises as: every addon's `tests/` **should** have an `__init__.py`; the addon directory itself **should not**. + +The asymmetry matters. The addon directory must remain a plain namespace dir — Gramps' plugin loader puts the addon dir on `sys.path` and imports `.py` by name. Making the addon dir a regular package can disturb plugin loading (and the [Mantis 12691](https://gramps-project.org/bugs/view.php?id=12691) namespace trap lives in exactly this area). The `tests/` subfolder has no such constraint, so making it an explicit package is free. + +This is what [addons-source PR 930](https://github.com/gramps-project/addons-source/pull/930) (Gary Griffin) is moving toward. + +## The GTK-pin contract + +Gramps establishes the GObject-introspection environment **once, at startup, before any plugin loads**: `gi.require_version("Gtk", "3.0")` and `gi.require_version("Gdk", "3.0")` run in `gramps/grampsapp.py` and `gramps/gen/constfunc.py`. Every addon module therefore does `from gi.repository import Gtk` into an already-pinned namespace — inside Gramps, the pin is never the addon's job. + +**The trap.** Run that module under bare `unittest` and the import warns (or resolves a different GTK) because nothing has pinned yet. The tempting fix is to copy the `gi.require_version` call into the addon module or the test file. Tests now pass — but the pin also executes inside Gramps, where it is redundant at best and a hard failure the moment the hardcoded pin and the version Gramps runs diverge: `gi.require_version` raises `ValueError` once the namespace is already loaded at a different version. That is the *works-in-tests, breaks-in-Gramps* failure mode, and it is invisible to CI because CI only runs the tests. + +**The contract**, in two halves: + +1. **Modules never pin.** No `gi.require_version` in the addon module or in any test file — the environment is provided *to* them, in both contexts. Redundant pins are safe to remove from files you are already touching (the Themes addon's `tests/__init__.py` cleanup is the precedent), but don't churn files you aren't otherwise changing. +2. **The repository root provides what Gramps provides.** addons-source carries the pins **once**, in the repo-root `tests/__init__.py` (addons-source PR [950](https://github.com/gramps-project/addons-source/pull/950)): the repo-root suite run and the CI runners import that package before any test module, pinning the whole suite to the GTK 3 / GDK 3 stack a real Gramps session uses (it also silences the locale warnings that uncompiled source-tree addons legitimately emit). The per-addon `MyAddon/tests/__init__.py` stays **empty** — see the previous section. + +The one thing a GUI-touching test module may still need is a presence guard for hosts with no PyGObject at all: + +```python +try: + import gi +except ImportError as err: + raise unittest.SkipTest("PyGObject not available: %s" % err) +``` + +**The corollary: run from the repository root.** The pins execute when the root `tests` package loads — the repo-root suite run and CI's per-addon runners do that. Run your own invocations from the addons-source root too (the dotted-path form below), and never run a test file by filesystem path (`python3 MyAddon/tests/test_myaddon.py`) — the shortcut that pushes pins back into the modules, and that bypasses the namespace-package semantics the loading section below relies on. + +The GI pins are one instance of a wider rule: everything process-global that Gramps' startup owns — locale, the root logger, `sys.path`, the GTK main loop, `sys.excepthook`, environment variables — follows the same contract. The full startup surface, with the per-item temptations and alternatives, is tabulated in [04-fundamentals → The provided environment](04-fundamentals.md#the-provided-environment). + +## Filename conventions (addons-source CI) + +addons-source's CI workflow filters tests by **filename prefix** to scope them per platform: + +| Prefix | Where it runs | +|-------------------------|------------------------------------------------| +| `test_*.py` | All platforms (Linux + Windows) | +| `test_linux_*.py` | Linux only | +| `test_windows_*.py` | Windows only | +| `test_integration_*.py` | Linux only — full-pipeline / DB-backed | + +The Ubuntu runner skips `test_windows_*`; the Windows runner skips both `test_linux_*` and `test_integration_*`. Both runners include the platform-neutral `test_*.py` files. + +**Pick the prefix that matches the test's portability**, not the platform you happen to be developing on. A test that exercises POSIX file paths goes under `test_linux_*`; a test that exercises win32 locale handling goes under `test_windows_*`; everything else, the plain `test_*.py` prefix. + +CI's workflow file is authoritative: [addons-source/.github/workflows/ci.yml](https://github.com/gramps-project/addons-source/blob/maintenance/gramps60/.github/workflows/ci.yml). + +## Loading: dotted path, not `discover` + +Upstream CI loads tests by **dotted path**: + +```bash +python3 -m unittest MyAddon.tests.test_myaddon +``` + +Not by `discover` from inside an addon's `tests/` directory, and never by filesystem path. Dotted-path loading from the repo root surfaces the namespace-package trap. Bug 12691 — `from import ` binding the submodule instead of the class — only shows up under dotted-path loading. `discover`-based loading walks files by *filename*, hiding the import-resolution issue. Mirroring CI's invocation locally catches what CI catches. + +Locally, from the `addons-source` root, the same invocation works: + +```bash +# Run one test module +python3 -m unittest MyAddon.tests.test_myaddon + +# Run every test in the addon's tests/ package +python3 -m unittest discover -s MyAddon/tests -t . +``` + +The discover form here works because the addon directory is the import root — the namespace-package trap shows up only when an *individual addon module* mis-imports itself. + +## Mocked vs `example.gramps`-backed tests + +Two complementary strategies. They're not alternatives. + +### Mocked unit tests + +Fast, no DB on disk, suitable for tight branch-coverage of pure logic. Substitute the database with a stub that returns fixed objects: + +```python +import unittest +from unittest.mock import MagicMock + + +class HappyPathTests(unittest.TestCase): + def test_skips_people_without_birth(self): + person = MagicMock() + person.get_birth_ref.return_value = None + + result = pure_logic(person) + + self.assertEqual(result, expected) +``` + +The MagicMock approach has a built-in failure mode: it returns something for *every* method call, so a typo'd method name appears to work. Real DB code that fails on the next call will pass the mocked test. This is the bug the next strategy catches. + +### `example.gramps`-backed tests + +`example.gramps` ships with the Gramps source under `example/gramps/example.gramps`. It's the canonical fixture triage and developers reproduce against; loading it produces a real populated database with the cross-typed backlinks, ID normalisations, and absent optional fields that real users hit. + +```python +import os +import unittest +from gramps.gen.db.utils import open_database + + +class IntegrationTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.db = open_database( + os.path.expanduser("~/path/to/gramps/example/gramps/example.gramps") + ) + + def test_handles_real_data(self): + result = code_under_test(self.db) + self.assertGreater(len(result), 0) +``` + +Name these `test_integration_*.py` so CI scopes them to Linux only (loading a real DB is heavier, and Windows CI's Gramps setup is separately constrained — see [14-compatibility → Windows toolchain migrated to UCRT64](14-compatibility.md#windows-toolchain-migrated-to-ucrt64)). + +### Choosing between them + +| Use the mock when | Use `example.gramps` when | +|---------------------------------------------------|-----------------------------------------------------------| +| The function under test takes pure inputs | The function traverses the DB | +| You're covering many input shapes (loop / branch) | You're verifying *one* real-world scenario | +| You need sub-millisecond turnaround | You need real-data shape (backlinks, IDs, optional refs) | + +The lesson, learned the hard way: mocked tests can pass while real-DB tests fail, because the mock doesn't model what production data looks like. + +## Tests must run without `requires_mod` deps + +A hard constraint, set by Gary Griffin (2026-05-16): addon tests must run cleanly without the addon's `requires_mod` dependencies installed in the Python that runs them. Mac contributors can't easily install addon deps into the Gramps Python on macOS, and there's no Gramps debug-mode equivalent on Mac to work around it. + +Two ways to honour this: + +### Mock at the import boundary + +```python +import sys +from unittest.mock import MagicMock + +# Stand in for an optional dep before importing the addon. +sys.modules.setdefault("PIL", MagicMock()) +sys.modules.setdefault("PIL.Image", MagicMock()) + +from MyAddon.MyAddon import code_under_test +``` + +Cleaner than try/except, and the test asserts the addon's behaviour **with the dep present** — what almost every real user sees. + +### Skip cleanly + +When mocking is impractical (e.g. the dep is core to the function under test), skip without erroring: + +```python +import unittest +from importlib.util import find_spec + + +@unittest.skipUnless(find_spec("PIL"), "Pillow not installed") +class PhotoTaggingTests(unittest.TestCase): + def test_loads_jpeg(self): + ... +``` + +A failed import at module load — instead of a `skipUnless` — turns into a test error on the Mac runner, blocking the CI suite. + +## What to test + +Mandatory: + +- **The bug a fix closes.** Every bug fix ships with a test that fails pre-fix and passes post-fix. At PR level this is a [16-guidelines MUST](16-guidelines.md#contributor-workflow): the regression test, or an explicit "no test because X" rationale plus a manual repro — "add the test later" is not an option. Doc-only PRs are the only exception. + +Strongly recommended: + +- **One happy-path call** through the addon's main entry point. The smoke test that catches the next breakage. +- **One real-data scenario** against `example.gramps` for any DB-traversal code. + +Optional but valuable: + +- **Edge cases** the function explicitly handles: empty DB, missing optional fields, IDs at the boundaries of normalisation. + +What *not* to test: + +- The Gramps API itself. If `db.get_person_from_handle(h)` returns `None` for a missing handle, that's Gramps' contract; your test exercises that **your code handles `None`**, not that Gramps returns it. + +## What the test catches that the GUI doesn't + +A test surfaces failure modes the GUI cycle hides: + +- **The namespace-package trap** (bug 12691) — surfaces under dotted-path loading. +- **`requires_mod` typos** — `from import …` would fail import; surfaces immediately at test load. +- **DB-shape assumptions** — the cross-typed-backlinks / ID-norm issues that mocked tests miss. +- **Per-OS regressions** — running on both runners. + +See [09-troubleshoot](09-troubleshoot.md) for the symptoms-to-cause mapping for these classes of failure. + +## Running tests locally + +From the `addons-source` checkout root: + +```bash +# Run one addon's tests +python3 -m unittest discover -s MyAddon/tests -t . + +# Or invoke a single test module by dotted path (mirrors CI's invocation) +python3 -m unittest MyAddon.tests.test_myaddon +``` + +Run from the addons-source root, and never invoke a test file by filesystem path — see [the GTK-pin contract](#the-gtk-pin-contract). + +The Python that runs the tests needs `gramps` importable. The simplest setup is `PYTHONPATH=/path/to/gramps python3 -m unittest …`; if Gramps is installed system-wide, the import resolves without `PYTHONPATH`. + +On Windows, run from the MSYS2 UCRT64 shell against a UCRT64-installed Gramps — the AIO build for Gramps 6.1+ targets UCRT64; Gramps 6.0 isn't Windows-tested upstream. See [14-compatibility → Windows toolchain migrated to UCRT64](14-compatibility.md#windows-toolchain-migrated-to-ucrt64). + +## See also + +- [04-fundamentals → Logging](04-fundamentals.md#logging) — `LOG` setup that tests assert against. +- [05-data-access → Testing data access](05-data-access.md#testing-data-access) — DB-API patterns to exercise. +- [08-debug](08-debug.md) — turning a repro script into a test. +- [09-troubleshoot](09-troubleshoot.md) — the symptoms these tests catch in CI rather than production. +- [10-code-analysis](10-code-analysis.md) — what the static checkers verify before tests run. +- [16-guidelines → Testing](16-guidelines.md#testing) — normative rules. +- [Mantis 12691](https://gramps-project.org/bugs/view.php?id=12691) — the canonical namespace-package trap that motivates dotted-path loading. +- [addons-source PR 930](https://github.com/gramps-project/addons-source/pull/930) — `tests/__init__.py` convention. diff --git a/docs/addon-development/08-debug.md b/docs/addon-development/08-debug.md new file mode 100644 index 000000000..b12f46970 --- /dev/null +++ b/docs/addon-development/08-debug.md @@ -0,0 +1,184 @@ +# Debug + +[← Previous](07-testing.md) · [Index](01-overview.md) · [Next →](09-troubleshoot.md) + + + +## Overview + +How to see what an addon is actually doing — where it logs, how to enable verbose output, and the patterns for reproducing a problem without sitting through a full Gramps launch cycle each time. + +Most addon bugs are reachable through three escalating tools, in order: read the log window, enable per-logger debug output, or write a tight repro script that bypasses the GUI entirely. The heavier tools (pdb, gdb) are documented at the bottom for the cases where the lighter ones don't suffice. + +## Where addon output goes + +Two surfaces, both populated by the same logging calls: + +| Surface | When you see it | +|------------------------|------------------------------------------------------------------------------------------------| +| **Gramps log window** | Help → Log. Always populated. Visible to the user. | +| **stderr / terminal** | Whatever shell launched Gramps. Populated only when Gramps is run from a terminal. | + +The logging module that backs both is the stdlib `logging`; an addon's module-level logger feeds in like any other: + +```python +import logging +LOG = logging.getLogger(__name__) + +LOG.debug("Computed candidate set: %s", candidates) +LOG.info("Processed %d people", n) +LOG.warning("Skipping malformed event %s", event.gramps_id) +LOG.error("Could not parse %s", filename) +``` + +`__name__` for an addon resolves to the addon's `id` (e.g. `"MyAddon.myaddon"`), so the logger inherits the addon's name naturally — useful for per-logger filtering below. + +**Don't use `print()`.** It bypasses both surfaces and breaks under windowed launches that have no terminal attached. The [16-guidelines](16-guidelines.md#runtime) page makes this a hard rule. + +## Default log levels + +Gramps configures the root logger at `WARNING` by default; `DEBUG` and `INFO` are filtered. Two ways to lower the bar: + +### `--debug=` + +Launch Gramps with the `--debug` flag to enable `DEBUG` for one named logger: + +```bash +gramps --debug=MyAddon +gramps --debug=MyAddon.myaddon # narrower +gramps --debug=gramps.gen.db # for DB internals +``` + +Pass it more than once to enable several loggers. The flag is strictly opt-in per logger — that's why you set it on the launch command, not in code. Other loggers stay quiet, so you're not swimming in noise. + +### Module-level override (development only) + +When iterating tightly, drop a one-line override at the top of the implementation module: + +```python +import logging +logging.getLogger(__name__).setLevel(logging.DEBUG) +``` + +Remove before committing — published addons should rely on `--debug=…` so users aren't forced into verbose output. + +## Reproduction scripts that bypass the GUI + +Restarting Gramps to test a one-line change burns minutes. For anything that *can* be tested without the GUI, write a tight repro script that instantiates the addon's testable pieces directly. + +The pattern looks like this: + +```python +# repro_.py — run with `python3 repro_.py`. +import os, sys +sys.path.insert(0, os.path.expanduser("~/path/to/gramps")) + +from gramps.gen.const import GRAMPS_LOCALE +from gramps.gen.utils.grampslocale import GrampsLocale +from gramps.gen.db.utils import open_database + +# Pin the locale without touching system locale config. +glocale = GrampsLocale( + localedir=os.path.expanduser("~/path/to/gramps/po"), + languages=["fi"], # the language under test +) + +db = open_database("example.gramps") +# … exercise the buggy code path … +``` + +`GrampsLocale(localedir, languages)` is the key escape hatch — it bypasses both `LANGUAGE` env-var setup and `locale-gen`-style OS config, neither of which is needed for an in-process test. The pattern came out of triaging [Mantis 14100](https://gramps-project.org/bugs/view.php?id=14100) (Finnish month-inflection crash). + +For DB-traversal code, the canonical fixture is `example.gramps` shipped with Gramps source. Real-data tests against `example.gramps` catch bugs mocked DBs miss — see [05-data-access → Testing data access](05-data-access.md#testing-data-access). + +The same pattern, formalised as a `unittest.TestCase`, becomes a regression test. See [07-testing](07-testing.md). + +## In-app diagnostics: `PrerequisitesCheckerGramplet` + +When a user reports an addon misbehaving, the first triage step is *"what's the running environment?"* The PrerequisitesCheckerGramplet (an addon itself) lists every optional dependency Gramps detects and the version it found. Asking a reporter to install it, run it, and paste the output is the fastest baseline. + +It also surfaces missing GI bindings, which addons typically declare via `requires_gi` — a `requires_gi=[("GExiv2", "0.10")]` that fails silently is almost always something the PrerequisitesCheckerGramplet output would have surfaced. + +[Mantis 13966](https://gramps-project.org/bugs/view.php?id=13966) (active_page None on tree close) was a teardown-order bug *in* this gramplet; the fix lives in addons-source PR 913. + +## Platform notes + +**Linux** — the standard environment. `gramps --debug=…` works as documented; logging surfaces in the terminal that launched Gramps, plus the in-app log window. + +**Windows** — debug flags work the same way, but the launcher is typically a `.bat` or `.exe` shortcut rather than a terminal command. Launch from MSYS2 UCRT64 (`/ucrt64/bin/gramps`) to get a terminal attached for stderr. Some bugs reproduce only on Windows; when reporting one, include the Gramps version, the MSYS2 UCRT64 toolchain version, and the exact reproduction steps in the Mantis ticket so a Windows-equipped maintainer can confirm. + +**macOS** — there is **no Gramps debug mode equivalent on Mac**, and contributors typically can't install addon dependencies into the Gramps Python (Gary Griffin, 2026-05-16). This shapes two testing-side decisions: tests must run cleanly without `requires_mod` deps installed (see [07-testing](07-testing.md)), and a Mac repro that needs GUI inspection usually requires triaging via screenshots the reporter pastes into the Mantis ticket. + +## Heavier tools + +When the lighter approaches aren't enough. + +### `pdb` (Python debugger) + +Drop a breakpoint at the line of interest: + +```python +breakpoint() # Python 3.7+; same as `import pdb; pdb.set_trace()` +``` + +Launch Gramps from a terminal; when execution reaches the breakpoint, Gramps freezes and you get an interactive `(Pdb)` prompt. Commands: `n` (next line), `s` (step into), `c` (continue), `l` (list source), `p expr` (print). Full reference: [Python pdb docs](https://docs.python.org/3/library/pdb.html). + +### `python -m trace` + +For "where on earth does the crash come from?": + +```bash +python3 -m trace -t /path/to/Gramps.py >/tmp/trace.out +``` + +Produces every executed line of Python in the file. Huge, but `grep` of the last hundred lines often pins the crash site. + +### `gdb` (C debugger) + +For segfaults coming from C libraries (GTK, GObject Introspection): + +```bash +gdb python3 +(gdb) run /path/to/Gramps.py +# … reproduce the crash … +(gdb) bt # Python+C backtrace; the C frames pin the C-side cause +``` + +To trap GTK warnings as hard errors: + +```bash +G_DEBUG=fatal-warnings gdb python3 +``` + +Then `r /path/to/Gramps.py`; any GTK warning aborts with a backtrace showing the originating call. + +Addons rarely need `gdb` — segfaults that bubble up from C are usually GI binding issues (wrong typelib version, missing `gir1.2-*` package). When you do hit one, the C backtrace plus the user's `PrerequisitesCheckerGramplet` output typically point at the missing package. + +### Profiling + +`gramps.gen.utils.debug.profile` is a convenience wrapper around `cProfile`. Replace the call you want to profile: + +```python +from gramps.gen.utils.debug import profile + +def cb_save(self, *obj): + profile(self.save, *obj) +``` + +On the next save, a profile report goes to stdout: per-function call counts and cumulative time. Useful when a gramplet's `main()` is unexpectedly slow. + +## See also + +- [04-fundamentals → Logging](04-fundamentals.md#logging) — the conventions for setting up the logger in the first place. +- [07-testing](07-testing.md) — formalising a repro script into a regression test. +- [09-troubleshoot](09-troubleshoot.md) — symptom-first guide to the failure modes these tools surface. +- [16-guidelines](16-guidelines.md) — the rules around logging and diagnostics (logger over `print`, etc.). +- [Debugging Gramps](https://gramps-project.org/wiki/index.php/Debugging_Gramps) — the standalone wiki page; primary scraped source. +- [Logging system](https://gramps-project.org/wiki/index.php/Logging_system) — the deeper reference for Gramps' logging configuration. diff --git a/docs/addon-development/09-troubleshoot.md b/docs/addon-development/09-troubleshoot.md new file mode 100644 index 000000000..19bb7d9a0 --- /dev/null +++ b/docs/addon-development/09-troubleshoot.md @@ -0,0 +1,213 @@ +# Troubleshoot + +[← Previous](08-debug.md) · [Index](01-overview.md) · [Next →](10-code-analysis.md) + + + +## Overview + +The failure modes that bite first-time addon authors, organised by symptom. Each entry is "what you see → why → what to do." Read this sideways: jump to the symptom that matches what you're seeing, follow the link out to the relevant chapter for the fix in depth. + +For technique-level coverage (pdb, gdb, profilers), see [08-debug](08-debug.md). For the normative rules an addon must satisfy, see [16-guidelines](16-guidelines.md). + +## Loading and discovery + +### "My addon doesn't appear in any menu." + +The addon failed to register. Three usual causes, in order of likelihood: + +1. **`.gpr.py` raised at import.** Plugin discovery executes every `.gpr.py` at startup; a `SyntaxError` or import failure there silently drops the addon from the catalog. Launch from a terminal to see the traceback on stderr, or check the Gramps log window (Help → Log) for the failure entry. + +2. **`gramps_target_version` mismatch.** A `6.0` addon won't load in 6.1, and vice versa. Plugin discovery silently skips the registration entry. See [14-compatibility → `gramps_target_version` semantics](14-compatibility.md#gramps_target_version-semantics). + +3. **`id` doesn't match the folder name.** The addon's folder name and the `id` argument to `register(...)` must be identical. Gramps does not match by content — it matches by folder name and verifies against `id`. A mismatch silently drops the entry. + +The fastest check: in a Python REPL with `gramps` on `sys.path`, `exec(open("MyAddon/MyAddon.gpr.py").read())`. If it raises, you have your cause; if it returns silently and there's no entry, your `register()` call is being filtered out. + +### "My edits to the plugin file disappeared on restart." + +The user plugin directory (`~/.local/share/gramps/gramps60/plugins/…`) is the auto-sync **target**. Edits there are silently overwritten on the next save from `addons-source/`. + +**The fix.** Edit in `addons-source//` and let the sync flow do its job — see [12-packaging → Editing `addons-source/`, not the live plugin directory](12-packaging.md#editing-addons-source-not-the-live-plugin-directory). + +On Gramps 6.1+ Linux/macOS, symlinking the working tree into the user plugin directory once eliminates the copy step (commit `9443dcbb30`); on Gramps 6.0 and on Windows generally, the copy / `rsync` loop remains. + +### "The addon's folder is there but Gramps doesn't load it." + +The most common variants of the previous symptom, when ruled out: + +- **6.0 only**: the folder is reached via a symlink. Gramps 6.0 plugin discovery does **not** follow symlinks; use a physical copy or upgrade to 6.1+. (See [14-compatibility → Plugin discovery follows symlinks](14-compatibility.md#plugin-discovery-follows-symlinks).) +- **Windows, any version**: same as above — the 6.1 symlink test is skipped on Windows because the platform's symlink behaviour is inconsistent without elevated privileges. Physical copy. +- **`.gpr.py` not at top level of folder**: the registration file has to be `/.gpr.py`, not in a subfolder. + +## Imports and Python namespace traps + +### "`from import ` binds the submodule, not the class." + +The classic Gramps namespace-package trap. + +The addon folder is a *namespace package* (PEP 420), so importing `` gives you the package, not the class inside the like-named module. Code that worked under `discover`-based test loading breaks under dotted-path loading because the resolution path is different. + +**The fix.** Use the explicit submodule form: + +```python +# Wrong: binds the package (silently — until you try to use the class) +from MyAddon import MyAddon + +# Right: binds the class inside the module +from MyAddon.MyAddon import MyAddon +``` + +Mantis bug [12691](https://gramps-project.org/bugs/view.php?id=12691) is the canonical case. Upstream CI loads addon tests by dotted path rather than `discover` exactly to surface this trap; see [07-testing](07-testing.md). + +### "`requires_mod` declares `Pillow` but Gramps says it's missing." + +`requires_mod` takes the **importable** module name, not the PyPI distribution name: + +| PyPI name | Importable name | +|----------------|-----------------| +| `Pillow` | `PIL` | +| `PyYAML` | `yaml` | +| `lxml` | `lxml` | +| `python-dateutil` | `dateutil` | +| `Beautifulsoup4` | `bs4` | + +**The check.** Before pushing, verify on a system with the package installed: + +```python +from importlib.util import find_spec +assert find_spec("PIL") is not None +``` + +If `find_spec` returns `None`, the name in `requires_mod` is wrong. + +### "`requires_gi` declaration is fine on 6.0, broken on 6.1." + +GExiv2's version handling was rewritten on `maintenance/gramps61` only (addons-source PR [829](https://github.com/gramps-project/addons-source/pull/829)). An addon with `requires_gi=[("GExiv2", "0.10")]` that works on 6.0 may need a different pin on 6.1. + +**The fix.** Read the EditExifMetadata addon's GExiv2 code on the target branch before assuming a pin transfers. See [14-compatibility → GExiv2 version handling rewritten](14-compatibility.md#gexiv2-version-handling-rewritten). + +## Database access + +### "My addon iterates the DB but raises `KeyError` halfway through." + +The DB contains a reference to a handle that no longer resolves. This happens in real-world data; mocked tests don't exhibit it because mocks always return the same fixed set. + +**The fix.** Always guard handle dereferences: + +```python +event = db.get_event_from_handle(handle) +if event is None: + continue # silently skip dangling reference +``` + +See [05-data-access → Reading: one object at a time](05-data-access.md#reading-one-object-at-a-time) for the pattern. The same shape applies to every `get__from_handle` call. + +### "Backlinks return `(class_name, handle)` not `(class, handle)`." + +`db.find_backlink_handles(handle)` yields tuples whose first element is the **class name as a string** (`"Person"`, `"Family"`, …), not the Python class itself. The most common bug here is `isinstance` checks that never match. + +```python +# Wrong: +for cls, h in db.find_backlink_handles(handle): + if cls is Person: # always False — cls is "Person" + ... + +# Right: +for type_name, h in db.find_backlink_handles(handle): + if type_name == "Person": + ... +``` + +### "The fix worked in my mocked test but breaks on `example.gramps`." + +Real data has shapes the mock doesn't model: + +- **Cross-typed backlinks** — a Source can be backlinked from a Person, a Family, an Event, a Place, a Media, a Note, a Citation, and a Repository. Mocks tend to model only the type the test author was focused on. +- **ID normalisation** — `I0001` vs `I0021` vs `I12345`. A regex that matches the mock's 4-digit IDs misses the real data's variable-width IDs. +- **Optional fields actually being absent** — `person.get_birth_ref()` returns `None` in real data far more often than in mocks. + +**The fix.** Add an `example.gramps`-backed test alongside the mock. See [05-data-access → Testing data access](05-data-access.md#testing-data-access) and [07-testing](07-testing.md). + +## Translation and locale + +### "My addon translates fine on Linux, not on Windows" (or vice versa). + +The two platforms set up the locale differently: + +- **Linux** — needs `locale-gen` for the language and the `LANGUAGE` env var set (not just `LANG`). +- **Windows** — reads `LANG` directly via `win32locale.py`, no OS locale config needed. + +**The fix in repro scripts.** Sidestep both by instantiating `GrampsLocale(localedir, languages)` directly — see [08-debug → Reproduction scripts that bypass the GUI](08-debug.md#reproduction-scripts-that-bypass-the-gui). + +**The fix in production.** Make sure the per-addon `.po` files compile cleanly on both platforms (`make.py compile ` in addons-source). A `.mo` file that's missing or malformed will silently fall back to English on whichever platform fails to load it. + +### "Strings I marked with `_()` aren't translated." + +You've forgotten to bind `_` to the addon's catalog. At the top of the implementation module: + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale + +_ = glocale.get_addon_translator(__file__).gettext +``` + +Without that line, `_()` falls back to Gramps' core catalog rather than the addon's own `/po/` translations. The strings stay English regardless of UI language. See [04-fundamentals → Translation](04-fundamentals.md#translation). + +## Testing + +### "Tests pass locally, fail in CI." + +Three common causes: + +1. **Filename prefix wrong for the platform.** `test_linux_*.py` is skipped on the Windows runner, `test_windows_*.py` is skipped on Linux. A test you intended as cross-platform but accidentally named with a prefix runs only where the prefix points. See [07-testing → Filename conventions](07-testing.md#filename-conventions-addons-source-ci). + +2. **`requires_mod` deps assumed in tests.** Addon tests must be runnable without the addon's `requires_mod` dependencies installed in the test Python — Mac contributors can't easily install addon deps into the Gramps Python (Gary Griffin, 2026-05-16). Mock at the import boundary or skip cleanly. + +3. **Test loaded by dotted path surfaces the namespace trap.** Local `discover` from `tests/` would hide the `from import ` bug; CI loads by dotted path (`.tests.`), which exposes it. See bug 12691. + +### "PR's pre-commit passed but CI is red." + +Pre-commit catches static checks only. Test failures (e.g. an import that breaks at module load) surface in CI's actual unit-test run, not in pre-commit. After pushing, watch the PR's checks until they finish: + +```bash +gh pr checks --watch +``` + +See [16-guidelines → Verification before commit](16-guidelines.md#verification-before-commit). + +## Pull-request shape + +### "`.gpr.py` version bump rejected on PR." + +addons-source PRs do **not** bump the addon's `version` field. The maintainer manages versions centrally. Leave the `version = "…"` line in `.gpr.py` untouched. (Tripped on addons-source PR 911.) + +### "PR sits without review." + +Two things to check: + +1. **Branch target.** `addons-source` PRs target `maintenance/gramps60`, not `master`. Gary cherry-picks forward to `gramps61`. Core PRs target `maintenance/gramps61`. A PR against the wrong branch may sit untouched waiting for retargeting. See [16-guidelines → Contributor workflow](16-guidelines.md#contributor-workflow). + +2. **PR body shape.** Reviewers expect a **`**User impact:**`** opener (before Root cause), then *Summary / What to look at / Root cause / Fix / Verification* (the #106 format). A PR body that leads with internals instead of the user-visible effect often returns without a substantive review until it conforms. + +### "PR was rejected as duplicate." + +You wrote a fix without checking that upstream already had one in flight. The pre-flight check is [16-guidelines → Contributor workflow](16-guidelines.md#contributor-workflow) — the bullets about "check upstream isn't ahead" and "if a PR already exists, VERIFY it, do not duplicate." Searching by **affected file path**, not just the bug number, is the part that catches the most duplicates. + +## See also + +- [08-debug](08-debug.md) — technique-level coverage for reproducing what these symptoms describe. +- [07-testing](07-testing.md) — the test conventions that catch many of these symptoms before they reach a user. +- [12-packaging](12-packaging.md) — the source-to-distribution flow, where the "edits disappeared" trap lives. +- [14-compatibility](14-compatibility.md) — the 6.0 vs 6.1 deltas behind several entries here. +- [16-guidelines](16-guidelines.md) — normative reference; this chapter describes what *goes wrong*, that chapter describes what *must hold*. +- [Mantis bug tracker](https://gramps-project.org/bugs) — where the recurring failures get filed. diff --git a/docs/addon-development/10-code-analysis.md b/docs/addon-development/10-code-analysis.md new file mode 100644 index 000000000..1ebb28ea4 --- /dev/null +++ b/docs/addon-development/10-code-analysis.md @@ -0,0 +1,235 @@ +# Code Analysis + +[← Previous](09-troubleshoot.md) · [Index](01-overview.md) · [Next →](11-internationalization.md) + + + +## Overview + +What automated checks run against addon code, locally and in CI, and how to keep an addon passing them. The goal is "PR opens green" — every check below catches a class of issue cheaper than a maintainer review round. + +The checks vary by repo. Two combinations matter; the cheat sheet: + +| Check | gramps core | addons-source | +|----------------------------------------|------------------------|-------------------| +| Black formatting (`--check --diff`) | pre-commit + CI | — | +| `mypy` static types | pre-commit + CI | — | +| `ruff` E9 / F63 / F7 / F82 | — | pre-commit + CI | +| `python -m py_compile` / `ast.parse` | — | pre-commit + CI | +| `msgfmt` on `po/*.po` | upstream build | upstream build | +| `pylint` ≥ 9 on new files | manual; not gated | not enforced | + +`addons-source` does **not** enforce Black today; gramps core does. This is the most common surprise for authors moving between the two. See [Black](#black) below. + +## Pre-commit + +There is no published upstream `.pre-commit-config.yaml` for `addons-source`; addon authors can install their own locally to mirror the CI gates above (ruff, py_compile), but this is convenience tooling, not an upstream requirement. The authoritative checks live in `addons-source/.github/workflows/ci.yml`; if your local pre-commit and CI disagree, CI wins. + +For gramps core, the upstream pre-commit config under the `gramps/` repo covers Black and `mypy`; install with the standard `pre-commit install` flow from the repo root. + +## Black + +[Black](https://black.readthedocs.io/) is an opinionated Python formatter. Where enforced, the CI lint job is `psf/black@stable --check --diff`; a violation fails the build and blocks merging. + +**Where enforced.** gramps core (`maintenance/gramps61` and `master`) — both pre-commit and CI. addons-source does *not* enforce Black today; PRs there go through without formatting checks. + +**The trap on gramps core.** Tiny diffs that look harmless trip the gate: + +- A mid-module-import blank line. +- A multi-line `.append()` collapsed to one line. + +PR 2326 tripped this on `cli/clidbman.py` and a new test file; the fix was a Black-cleaned force-push rebase. Run `black --check` on the changed files before pushing: + +```bash +git diff --name-only --diff-filter=ACMR origin/master...HEAD \ + | grep '\.py$' \ + | xargs --no-run-if-empty black --check --diff +``` + +## `mypy` + +Gramps core's CI runs `mypy` against the tree; type errors block the build. `*.gpr.py` plugin registration files are excluded (they run in the injected-name scope and would otherwise complain about `register`, `_`, etc.). + +This applies to gramps core only. addons-source PRs don't run `mypy`; addon Python doesn't ship type hints by default. Where an addon does add type hints, prefer the 3.10+ shape (`X | None`, `list[X]`) per [16-guidelines → Coding style](16-guidelines.md#coding-style). + +## `ruff` E9 / F63 / F7 / F82 + +addons-source's pre-commit and CI run `ruff` with a tight rule selection: + +| Code | Catches | +|-------|------------------------------------------------------------------------| +| `E9*` | Syntax errors | +| `F63` | Comparison and membership operator mistakes (`is not` vs `not is`) | +| `F7` | Imports inside dead code, syntax-level structural issues | +| `F82` | Undefined names | + +It's a syntax-and-undefined-names net, not a style enforcer — the goal is "code that *imports*", which is what gets you past the plugin-discovery gate. + +Local invocation: + +```bash +ruff check --select E9,F63,F7,F82 / +``` + +The undefined-name rule (`F82`) is the most useful single check for addon authors — `Pillow` typo'd as `Pilllow`, `gramps.gen.plugin` typo'd as `plugn`, the kind of typo that produces a silent skip in the plugin manager and no traceback. `ruff F82` catches them. + +A lint flag is a symptom, not the bug. Don't just add a `# noqa` or a defensive import to silence `F82` — read the enclosing function first. An undefined name in shipping code usually means dead or broken code. + +## `python -m py_compile` / `ast.parse` + +Both pre-commit and addons-source CI compile every changed `.py`: + +```bash +python -m py_compile /*.py +``` + +`ast.parse` is the more lenient check (won't import code, just parses); both exist because a file that compiles can still fail to import (`NameError` at module-level, missing dep). The compile pass is the absolute floor — failing it means the addon can't even register. + +## `msgfmt` on `po/*.po` + +Per-addon `.po` files have to compile to `.mo` cleanly for translations to take effect at runtime. A malformed catalog — mismatched `%s` substitutions, unclosed plural-form expression — silently falls back to English on the platform where compilation fails. Run `msgfmt -c` on every per-addon catalog before publishing. + +Local check: + +```bash +make.py gramps60 compile +``` + +`make.py compile` wraps `msgfmt` with the right paths; a failure prints the offending file and line. See [12-packaging → The localisation flow](12-packaging.md#the-localisation-flow). + +[Mantis 14234](https://gramps-project.org/bugs/view.php?id=14234) (lxml `ngettext` newline fix; addons-source PR 907) is the canonical example — a single misplaced newline in a plural form, caught by a `msgfmt -c` pass. + +## `pylint` + +Gramps' programming guidelines call for pylint ≥ 9 on new files and "changes to existing files shall not reduce the pylint score" — but this is **not** gated by CI. It's developer guidance, not a hard check. + +`pylint` doesn't run on addon code by default. When you do run it locally: + +```bash +pylint --disable=missing-docstring /.py +``` + +Run from the addon's parent directory so `pylint`'s import resolution finds it as a package. + +## Verifying `requires_mod` + +`requires_mod` takes the **importable** module name, not the PyPI distribution name (see [09-troubleshoot → `requires_mod` declares `Pillow`…](09-troubleshoot.md#requires_mod-declares-pillow-but-gramps-says-its-missing)). Before pushing, on a system with the dependency installed: + +```python +from importlib.util import find_spec +for mod in ["PIL", "lxml", "dateutil"]: + assert find_spec(mod) is not None, mod +``` + +This is a manual check, not a CI gate. It's listed here because the failure mode it catches — silent skip in the plugin manager — looks exactly like a `ruff F82` symptom but happens at a different layer. + +## Coding-standard rules worth running locally + +The full standard lives in `../gramps/AGENTS.md` and applies to all Gramps-related Python. The mechanical checks above cover formatting and syntax; the rules below need a manual pass. + +### Import grouping + +Three sections, each with a comment header: + +```python +# ------------------------------------------------------------------------- +# +# Standard Python modules +# +# ------------------------------------------------------------------------- +import os +import logging + +# ------------------------------------------------------------------------- +# +# GTK/Gnome modules +# +# ------------------------------------------------------------------------- +from gi.repository import Gtk + +# ------------------------------------------------------------------------- +# +# Gramps modules +# +# ------------------------------------------------------------------------- +from gramps.gen.db.base import DbReadBase +from .mymodule import MyClass +``` + +Existing code that doesn't follow this stays as-is; new code does. + +### Callback names + +Callbacks are prefixed `cb_`: + +```python +def cb_save(self, *args): + ... +``` + +`pylint` also avoids the `W0613: Unused argument` warning for `cb_*`-prefixed methods, which is convenient for GTK signal handlers that receive arguments they don't use. + +### Class headers + +Every class — including `unittest.TestCase` subclasses — carries a navigation comment header: + +```python +# ------------------------------------------------------------ +# +# MyClass +# +# ------------------------------------------------------------ +class MyClass: + ... +``` + +This is for finding the class when multiple classes share a file, not for documentation. Sphinx-style docstrings handle the documentation. + +### Member-name conventions + +- `__private` (two underscores) — class-only access. +- `_protected` (one underscore) — class and subclass access. + +PEP 8 with one local addition: a space after every comma. + +### TAB stops + +No TABs in Python. Indentation is 4 spaces. Where TABs are unavoidable (Makefiles), they're at columns 9, 17, 25, … (equivalent to 8 spaces). Don't set your editor's TAB stops to 4 — that "fixes" indentation by making TABs invisible and produces files that look right but parse wrong. + +## Running everything locally before pushing + +A pragmatic checklist before opening a PR: + +```bash +# Addons-source PRs: +ruff check --select E9,F63,F7,F82 / +python -m py_compile /*.py +make.py gramps60 compile # exercises msgfmt +python -m unittest discover -s /tests -t . # tests + +# Gramps core PRs add: +black --check --diff .py +mypy +GRAMPS_RESOURCES=. python3 -m unittest discover -p "*_test.py" +``` + +See [12-packaging](12-packaging.md) for `make.py` setup and [07-testing](07-testing.md) for the test-loading conventions. + +## See also + +- [04-fundamentals](04-fundamentals.md) — the conventions the static checks verify. +- [07-testing](07-testing.md) — the runtime checks that complement static analysis. +- [09-troubleshoot → "PR's pre-commit passed but CI is red"](09-troubleshoot.md#prs-pre-commit-passed-but-ci-is-red) — the most common code-analysis-related symptom. +- [12-packaging](12-packaging.md) — `make.py` invocations. +- [16-guidelines → Coding style](16-guidelines.md#coding-style), [16-guidelines → Verification before commit](16-guidelines.md#verification-before-commit) — normative rules. +- [Programming guidelines](https://gramps-project.org/wiki/index.php/Programming_guidelines) — the standalone wiki page; primary scraped source. +- `../gramps/AGENTS.md` — the full Python coding standard, inherited from gramps core. diff --git a/docs/addon-development/11-internationalization.md b/docs/addon-development/11-internationalization.md new file mode 100644 index 000000000..5405ae82e --- /dev/null +++ b/docs/addon-development/11-internationalization.md @@ -0,0 +1,180 @@ +# Internationalization + +[← Previous](10-code-analysis.md) · [Index](01-overview.md) · [Next →](12-packaging.md) + +## Overview + +Gramps is a highly globalized application, and addons should be fully translatable to support users worldwide. This guide covers how to prepare your addon for internationalization (i18n), manage translation strings using `gettext`, and package translations with your addon. + +See [the addon development overview](01-overview.md) for where this fits into the broader addon lifecycle. + +## Working example + +To make strings in your addon translatable, you need to mark them using the standard translation functions, extract them into a template (`.pot`), and provide translations (`.po`). + +### Registration + +In your `*.gpr.py` file, the strings you provide for `name`, `description`, etc., should use `_()` so they can be extracted by Gramps' build tools. + +You do not need to import `_` in the `.gpr.py` file — Gramps' plugin registration loader pre-defines it to use your locale translations. Just mark strings with `_("TEXT")` and supply a translation in your `.po` file. + +```python +# exampleaddon.gpr.py +register( + KIND, + id="ExampleAddon", + name=_("Example Addon"), + description=_("A sample addon to demonstrate internationalization."), + version="1.0.0", + gramps_target_version="6.0", + status=STABLE, + fname="exampleaddon.py", +) +``` + +### Implementation + +Inside your Python implementation, you must set up your addon's translation domain or use the core Gramps translation tools if contributing to the main repository. + +```python +# exampleaddon.py +import os +from gramps.gen.plug import Gramplet + +# Typical setup for an external addon to manage its own translation domain +from gramps.gen.const import GRAMPS_LOCALE as glocale +_ = glocale.get_addon_translator(__file__).gettext + +class ExampleAddon(Gramplet): + def init(self): + # A simple translated string + message = _("Welcome to the Example Addon!") + self.set_text(message) + + def show_items(self, count): + # Using ngettext for proper pluralization + ngettext = glocale.get_addon_translator(__file__).ngettext + msg = ngettext( + "Found %d item.", + "Found %d items.", + count + ) % count + print(msg) +``` + +## Translating UI Files (Glade) + +Gramps' addon translation tools only automatically extract and manage Python strings. If your addon uses a Glade (`.ui` / `.glade`) file for its interface, those strings will not be picked up by the standard addon translation workflow. The recommended pattern is to mark the Glade strings as translatable (so they show up for translators), then override the label at runtime from Python so they get translated through your addon's gettext domain. + +1. Give the relevant widget a meaningful `id` in the `.glade` file (not the autogenerated `label3`-style id), so your Python code can look it up: + + ```xml + + place|Name: + + ``` + + The `place|` prefix is a translator context hint (see [String Marking Rules](#string-marking-rules)) — it tells the translator which sense of "Name" you mean, and is stripped before display. + +2. In the corresponding dialog's `__init__`, override the label with the runtime-translated string: + + ```python + PLACE_NAME = _("place|Name:") + + # inside __init__: + self.get_widget("place_name_label").set_label(PLACE_NAME) + ``` + + The exact setter depends on the widget — `GtkLabel` uses `set_text`, `GtkButton` uses `set_label`, etc. + +3. Re-run `make.py … init` so the new string lands in `template.pot`, then translate and test. + +## String Marking Rules + +| Function | Meaning / Usage | +|----------|---------| +| `_("...")` | Standard string translation. Marks a string for extraction and translates it at runtime. | +| `N_("...")` | Marks a string for extraction but *does not* translate it at runtime. Useful for defining lists of strings that will be translated later when displayed. | +| `ngettext("Singular", "Plural", n)` | Translates a string while applying the correct pluralization rules for the target language based on the integer `n`. | +| `_("Context\|String")` | The Gramps convention for translator context. Prefix the user-facing string with a short hint plus a pipe (`\|`). The translator sees the hint in the `.po` file and renders only the post-pipe portion. The same `_` handles it — no special function call is needed. The two-arg form `_("String", "Context")` works equivalently and dispatches to `pgettext` under the hood. | + +**Canonical context example:** the English word "Title" can mean the title of a *book* or the nobility *title* of a person. In many languages these need different translations. Mark them as: + +```python +_("book|Title") +_("person|Title") +``` + +Translators see the hint, drop the `prefix|` part, and translate the two senses independently. This is the form used throughout `addons-source` today; don't reach for `pgettext` or `sgettext` directly — go through `_`. + +**Note on obsolete functions:** In older versions of Gramps (pre-Gramps 4), you may have seen `lgettext`, `ugettext`, `lngettext`, and friends. The `l*` variants returned strings encoded according to the current locale (bytes, not text), and the `u*` variants existed only to force Unicode output under Python 2. With Python 3, all strings are Unicode by default, so both families became redundant. Use `_` (i.e. `gettext`) and `ngettext` — they always return translated strings as Python `str`. `sgettext` and `pgettext` also exist as internal helpers, but addon code should go through `_`. + +## Weblate (Gramps 6.0+) + +> **Gramps 6.0 Weblate Workflow:** Starting with Gramps 6.0 (and *only* 6.0), addon translations can be done collaboratively on the Gramps Weblate platform. The `Third-party Addons` component contains aggregated translations for every addon. If your addon is hosted in the official repository, you do not need to manually manage `.po` files. + +## Managing Translations Manually with `make.py` + +If you are managing translations manually (or for older Gramps versions), the Gramps `addons-source` repository provides a `make.py` script to manage the entire lifecycle of your translations. This script relies on the standard `gettext` tools. + +Assuming you are in the `addons-source` directory and your addon is named `ExampleAddon`, here is the workflow: + +### 1. Extracting Strings (Template Generation) + +To extract all marked strings from your Python files and generate the `template.pot` file: +```bash +python3 make.py gramps60 init ExampleAddon +``` +This command parses your addon, creates necessary subdirectories (like `po/`), and writes the base `.pot` template. + +### 2. Adding a New Language + +To initialize a translation file for a specific locale (e.g., French `fr`): +```bash +python3 make.py gramps60 init ExampleAddon fr +``` +This creates a new, empty `po/fr-local.po` file based on your template. A translator can now open this `.po` file in a tool like Poedit to provide translations. + +### 3. Updating Translations + +If you modify your Python code and add new strings, you must update your templates and existing language files: +```bash +python3 make.py gramps60 update ExampleAddon fr +``` +This synchronizes the existing `.po` file with the latest `template.pot` without destroying existing translations. + +### 4. Compiling Translations + +When testing locally or preparing to package, compile the human-readable `.po` files into binary `.mo` files (which are placed in `locale//LC_MESSAGES/.mo`): +```bash +python3 make.py gramps60 compile ExampleAddon +``` +*Note: To compile all projects in your local repository at once, use `compile all` instead of `ExampleAddon`.* + +Before committing a hand-edited `.po`, run a quick syntax sanity check: +```bash +msgfmt -c po/fr-local.po +``` +This catches malformed headers, missing/mismatched format placeholders, and broken plural forms without going through the full `make.py` pipeline. + +### 5. Building for Release + +When your addon is ready, the build command will package everything, including the compiled translations, into a `.tgz` archive: +```bash +python3 make.py gramps60 build ExampleAddon +``` + +## Implementation notes + +- **Do not use f-strings or `.format()` inside the translation wrapper:** Translation tools like `xgettext` cannot extract dynamically generated strings. You must use old-style `%` formatting or translate the static template string first before calling `.format()`. + - **Bad:** `_(f"User {name}")` + - **Good:** `_("User %s") % name` +- **Context is key:** If a word can mean multiple things (e.g., "Date" as a fruit vs. "Date" as a calendar day), consider adding translation context comments so translators know how to interpret it. +- **Extraction:** Addons distributed in the `gramps-addons` repository have their translation strings automatically extracted into a `.pot` file by the Gramps translation infrastructure. + +## See also + +- [Addon Development overview](01-overview.md) +- [Coding for translation](https://gramps-project.org/wiki/index.php/Coding_for_translation) — the core-side counterpart to this page; covers conventions for marking strings in Gramps itself. +- [Translating Gramps](https://gramps-project.org/wiki/index.php/Translating_Gramps) — general guidelines for translators (`.po` headers, plural forms, context, mnemonics). +- [Python `gettext` documentation](https://docs.python.org/3/library/gettext.html) — primary reference for `gettext`, `ngettext`, and the `GNUTranslations` class that backs them. diff --git a/docs/addon-development/12-packaging.md b/docs/addon-development/12-packaging.md new file mode 100644 index 000000000..4dac98850 --- /dev/null +++ b/docs/addon-development/12-packaging.md @@ -0,0 +1,285 @@ +# Packaging + +[← Previous](11-internationalization.md) · [Index](01-overview.md) · [Next →](13-community.md) + + + +## Overview + +From "works on my machine" to "users can install it from the addon manager." This chapter is the source-to-distribution pipeline: how `addons-source` becomes a `.addon.tgz` in `addons`, how the in-app addon manager picks it up, and what to send upstream. + +The normative *rules* a submission must satisfy (branch targeting, version-field discipline, PR body shape, Mantis trailers) live in [16-guidelines](16-guidelines.md). This page covers the *workflow* — what to run, what files appear, where they end up. + +## The three repositories + +![Fig. 1 — The source-to-distribution pipeline. Authors edit in `addons-source/`; `make.py build` packages each addon into `addons/grampsXY/download/.addon.tgz` and `make.py listing` refreshes `addons/grampsXY/listings/*.json`; the in-app addon manager fetches both over HTTPS and installs to the user's plugin directory. Edits in the user plugin dir are not pushed back — the flow is one-way only.](_media/packaging-pipeline.svg) + +Gramps addons live across three repositories. You'll have all three cloned side-by-side under one base directory: + +``` +base/ +├── gramps/ # gramps-project/gramps — Gramps itself; source of truth for the API +├── addons-source/ # gramps-project/addons-source — addon source code, one folder per addon +└── addons/ # gramps-project/addons — built distribution, one folder per Gramps version +``` + +| Repo | What's there | You edit? | +|-----------------|-------------------------------------------------------------------------------------|------------------------------------| +| `gramps` | The Gramps source tree; provides `GRAMPSPATH` for the build | No (unless writing a core change) | +| `addons-source` | The addon source: `/.gpr.py`, `/.py`, `po/`, `tests/` | **Yes — author addons here** | +| `addons` | Built `.addon.tgz` packages and listing JSON, organised by Gramps minor | No — `make.py` writes to it | + +`addons` is the **output**. The in-app addon manager hits its HTTPS mirror to fetch listings and downloads. Editing files in `addons` directly does nothing — the next `make.py` run overwrites them. + +### Branch directory split inside `addons/` + +Inside `addons/`, each Gramps minor gets its own subdirectory: + +``` +addons/ +├── gramps42/ +├── gramps50/ +├── gramps51/ +├── gramps52/ +├── gramps60/ +│ ├── download/ # .addon.tgz files +│ └── listings/ # JSON catalogues fetched by the addon manager +└── gramps61/ + ├── download/ + └── listings/ +``` + +The same addon can ship to multiple minors, each with its own `.addon.tgz` — that's why the addon manager reads only the listing for the running Gramps version. + +## Initial clone + +```bash +mkdir gramps-addons && cd gramps-addons + +git clone https://github.com/gramps-project/gramps.git +git clone https://github.com/gramps-project/addons-source.git +git clone https://github.com/gramps-project/addons.git + +cd addons-source +git checkout -b gramps60 origin/maintenance/gramps60 # for 6.0 +# or for master/6.1: +# git checkout -b gramps61 origin/master +``` + +The branch you check out in `addons-source` determines which Gramps minor your built addons target — `make.py` reads the branch name to pick the output directory inside `addons/`. + +See [14-compatibility](14-compatibility.md) for branch-targeting guidance per Gramps minor. + +## Build prerequisites + +`make.py` calls out to two environment things and one OS tool: + +- **`GRAMPSPATH`** — absolute path to your `gramps/` clone. +- **`LANGUAGE`** — must be set to `en_US.UTF-8` for the build to run. +- **`intltool`** — `sudo apt-get install intltool` on Debian/Ubuntu. + +The standard invocation: + +```bash +GRAMPSPATH=/path/to/gramps LANGUAGE='en_US.UTF-8' python3 make.py gramps60 +``` + +Cumbersome to type each time. Set the env vars in your shell startup once; only the `make.py` line varies per command. + +In the examples below, `gramps60` is the maintenance/gramps60 target; substitute `gramps61` when you're working on the master branch. + +## `make.py` cheat sheet + +`make.py` lives at the top of `addons-source` and runs against one addon at a time, or `all` for everything. The commands you'll use most: + +| Command | What it does | +|------------------------------------------|-------------------------------------------------------------------------------| +| `make.py gramps60 init ` | Create `/po/template.pot` from extracted strings | +| `make.py gramps60 init ` | Create `/po/-local.po` from the template | +| `make.py gramps60 update ` | Merge new strings from Gramps + the addon into `-local.po` | +| `make.py gramps60 compile ` | Compile every `-local.po` into `.mo` files | +| `make.py gramps60 build ` | Compile translations *and* produce `.addon.tgz` in `addons/gramps60/download/` | +| `make.py gramps60 listing ` | Refresh `addons/gramps60/listings/*.json` so the addon manager sees it | +| `make.py gramps60 clean ` | Delete generated files (`locale/`, `*.mo`) — run before `git add` | +| `make.py gramps60 build all` | Build every addon | + +`build` includes `compile`, so the standard release cycle is `clean` → edit → `build` → `listing` → commit + push to `addons/`. + +### What `build` packages + +By default `build` includes: + +- every `*.py` in the addon folder, +- every `*.glade`, `*.xml`, `*.txt`, +- every `locale/*/LC_MESSAGES/*.mo`. + +Anything else — README images, extra data files, help HTML — needs an explicit `MANIFEST` file in the addon's root listing them, **with the addon folder name prefixed on each line**: + +``` +/README.md +/help/index.html +/data/* +``` + +The `MANIFEST` mechanism was added in Gramps 5.0 and is the way to ship anything beyond the default file types. + +## The localisation flow + +Per-addon translations live under `/po/`. They are independent of Gramps' core catalogues — Gramps' plugin loader binds `_()` to the addon's own catalog when the implementation module declares: + +```python +from gramps.gen.const import GRAMPS_LOCALE as glocale + +_ = glocale.get_addon_translator(__file__).gettext +``` + +See [04-fundamentals → Translation](04-fundamentals.md#translation) for the full opt-in. Without that line, strings fall back to the core catalog regardless of where translators put them. + +### Adding a new language + +```bash +# 1. Generate (or refresh) the template from extracted strings. +make.py gramps60 init + +# 2. Initialise a fresh language file from the template. +make.py gramps60 init fr + +# 3. Translator edits /po/fr-local.po manually. + +# 4. Recompile so the .mo file lands under /locale/fr/LC_MESSAGES/. +make.py gramps60 compile + +# 5. Commit the .po (NOT the .mo or the generated locale/ tree). +git add /po/fr-local.po +git commit -m "Add French translation for " +``` + +### Refreshing an existing language + +When you've changed user-visible strings in the addon, every existing `-local.po` needs new entries merged in: + +```bash +make.py gramps60 update fr +``` + +This preserves existing translations and marks new or changed entries as `fuzzy` for the translator to review. + +### Editing `template.pot`'s header once + +The header of `/po/template.pot` carries author, maintainer, and team metadata. Edit it after the first `init` — the values propagate into every per-language file the next time you `init `. + +### What to commit, what to skip + +Commit: + +- `/po/template.pot` +- `/po/-local.po` (one per language) + +Don't commit: + +- `/locale/**` — generated by `compile` and `build`. Always run `make.py gramps60 clean ` (or `rm -rf /locale`) before `git add`. +- `/*.mo` outside `locale/` — same reason. + +## Publishing to `addons/` + +`build` writes one `.addon.tgz` per addon; `listing` rebuilds the JSON catalogues the addon manager fetches. Both go into the `addons/` repository and need committing **there**, not in `addons-source/`: + +```bash +# In addons-source/, build the package. +make.py gramps60 build +make.py gramps60 listing + +# Switch to the addons/ checkout. +cd ../addons + +# Stage the new .tgz and the refreshed listings. +git add gramps60/download/.addon.tgz +git add gramps60/listings/* +git commit -m "Add for Gramps 6.0" + +# Push when you have write access — see [16-guidelines] for the review gate. +``` + +The in-app addon manager hits the `addons` repo over HTTPS and shows the addon to users on their next "Check for updates" cycle. + +## Editing `addons-source/`, not the live plugin directory + +During development it's tempting to edit directly in `~/.local/share/gramps/gramps60/plugins//` so a Gramps restart picks up the change without copying. **Don't** — that directory is the auto-sync *target*, and Gramps writes back through it on every save. Edits made there are silently overwritten on the next source save. + +Edit in `addons-source//`. The dev loop is one of: + +- **Gramps 6.0** — copy / `rsync` the folder into the user plugin directory on each save. Plugin discovery doesn't follow symlinks. +- **Gramps 6.1+ on Linux/macOS** — symlink the working tree into the user plugin directory once, then edit in place. Plugin discovery follows symlinks with realpath-based loop dedup (commit `9443dcbb30`). +- **Windows, any version** — physical copy. The 6.1 symlink test is skipped on Windows because the platform's symlink behaviour is inconsistent without elevated privileges. + +See [01-overview → Where addons live](01-overview.md#where-addons-live) for the user plugin directory paths. + +## Submitting a new addon + +The first time you add an addon to `addons-source/`: + +```bash +cd addons-source + +# 1. Create the folder and the two required files. +mkdir +$EDITOR /.gpr.py +$EDITOR /.py + +# 2. (Optional, recommended) Translation scaffold + tests. +mkdir /po /tests +$EDITOR /tests/__init__.py # empty marker — see 07-testing +$EDITOR /tests/test_.py + +# 3. Clean any build artefacts before committing. +make.py gramps60 clean + +# 4. Commit and push to your fork, then open a PR. +git add +git commit -m "Add : " +``` + +Open the PR against the **correct branch** — for an addon targeting Gramps 6.0, that's `maintenance/gramps60`, which the maintainer cherry-picks forward to `gramps61` (see [16-guidelines → Contributor workflow](16-guidelines.md#contributor-workflow) for the full submission shape). + +## Update an existing addon + +After editing source: + +```bash +cd addons-source + +# 1. Iterate: edit, restart Gramps, verify behaviour. +# 2. Refresh translations if user-visible strings changed. +make.py gramps60 update all +# 3. Compile-check. +make.py gramps60 compile +# 4. Clean and commit. +make.py gramps60 clean +git add +git commit -m ": " +``` + +The `version` field in `.gpr.py` **stays untouched** in PRs — the maintainer manages versions centrally. (See [16-guidelines](16-guidelines.md#contributor-workflow); this came up on PR 911 where a bump was rejected.) + +## See also + +- [01-overview](01-overview.md) — what an addon is. +- [04-fundamentals → Translation](04-fundamentals.md#translation) — the `_()` injection that makes per-addon `.po` files load. +- [07-testing](07-testing.md) — what to put in `tests/` before packaging. +- [14-compatibility](14-compatibility.md) — picking the right Gramps minor and branch for your addon. +- [16-guidelines](16-guidelines.md) — the normative rules a PR must satisfy. +- [Addons development](https://gramps-project.org/wiki/index.php/Addons_development) — the standalone wiki page; primary scraped source for this chapter. +- [addons-source CONTRIBUTING.md](https://github.com/gramps-project/addons-source/blob/maintenance/gramps60/CONTRIBUTING.md) — the addons-source-side contributor guide. diff --git a/docs/addon-development/13-community.md b/docs/addon-development/13-community.md new file mode 100644 index 000000000..9478c8265 --- /dev/null +++ b/docs/addon-development/13-community.md @@ -0,0 +1,84 @@ +# Community + +[← Previous](12-packaging.md) · [Index](01-overview.md) · [Next →](14-compatibility.md) + + + +## Overview + +When the PR is merged and the package published ([Packaging](12-packaging.md)), the addon *exists* — but nobody can find it, read about it, or reach you about it. This page covers the four steps that make a merged addon part of the ecosystem: the addon-list entry, the addon's own wiki page, the announcement, and the ongoing support duty. None of them touch code; all of them decide whether the addon gets used. + +## List your addon + +Add a row for your addon to the release's addon list — [6.0 Addons](https://gramps-project.org/wiki/index.php/6.0_Addons) for the current release, or the next release's list (e.g. [6.1 Addons](https://gramps-project.org/wiki/index.php/6.1_Addons)) if the addon targets an unreleased minor. Copy an existing row and fill in the columns; the [Addon list legend](https://gramps-project.org/wiki/index.php/Addon_list_legend) explains what each column means (type, audience, rating, contact, download). + +The row skeleton, as it appears in the list page's wiki source: + +``` +|- +| +| +| +| +| +| +| +| +|- +``` + +This listing is what users browse; the Plugin Manager's download listing ([Packaging](12-packaging.md) → the `make.py listing` step) is what Gramps itself reads. An addon needs both. + +## Document your addon + +Give the addon its own wiki support page — the page the addon list's first column links to. Examine other addons' pages for the format; the conventional skeleton: + +``` +{{Third-party plugin}} + +== Usage == + +=== Configure Options === + +== Features == + +== Prerequisites == + +== Issues == + +[[Category:Addons]] +[[Category:Plugins]] +[[Category:Developers/General]] +``` + +Only add the sections the addon needs — a Gramplet with no options doesn't need *Configure Options*. The `{{Third-party plugin}}` template expands to the standard notice that the addon is third-party and where to report problems; every addon page carries it. + +## Announce the addon + +Join the [Gramps forum](https://gramps.discourse.group/) and announce the addon to users: what it does, why you built it, and how to use it. This is the step authors skip most — and an unannounced addon is invisible to the users who would have wanted it. + +## Support it through the issue tracker + +Register on the [Gramps MantisBT tracker](https://gramps-project.org/bugs/) and check it regularly. **There is no automated notification** that routes issues against your addon to you — reports sit unseen unless you look. (For fix workflow, the tracker conventions, and the commit-message trailers that close Mantis issues, see [Rules](16-guidelines.md) → Commit messages.) + +Users don't read code and they make assumptions; reports will be ambiguous or wrong about the cause. Be kind and guiding — a curt reply from an addon's own author is the fastest way to lose the users the announcement won. + +## Why addons exist + +Worth keeping in mind across the maintenance years that follow ([Compatibility](14-compatibility.md), [What's New](15-whats-new.md)): the addon channel is deliberately low-barrier. It provides: + +- a quick way for anyone to share their work — the project has never refused an addon; +- a place for a component to evolve continuously, often before core acceptance; +- a home for plugins that will never be accepted into core but are loved by many users; +- a place for experimental components to live. + +## See also + +- [Packaging](12-packaging.md) — the build/listing mechanics that precede these steps. +- [Compatibility](14-compatibility.md) — keeping the published addon working across Gramps versions. +- [Addons development](https://gramps-project.org/wiki/index.php/Addons_development) — the upstream page these steps derive from. +- [6.0 Addons](https://gramps-project.org/wiki/index.php/6.0_Addons) — the addon list itself. diff --git a/docs/addon-development/14-compatibility.md b/docs/addon-development/14-compatibility.md new file mode 100644 index 000000000..1d5a413fe --- /dev/null +++ b/docs/addon-development/14-compatibility.md @@ -0,0 +1,114 @@ +# Compatibility + +[← Previous](13-community.md) · [Index](01-overview.md) · [Next →](15-whats-new.md) + + + +## Overview + +How an addon survives — or fails — across Gramps versions. Two things to understand: the `gramps_target_version` contract (Gramps' minor matters; majors aren't even discussed), and the concrete deltas between adjacent maintenance branches that bite ports in practice. + +A working addon for Gramps 6.0 is usually a working addon for 6.1 with **zero** code changes. The exceptions are documented here; when in doubt, the safest move is to maintain one addon folder per Gramps minor in parallel `maintenance/gramps*` branches of `addons-source`. + +## `gramps_target_version` semantics + +The `.gpr.py` registration declares which Gramps minor the addon targets: + +```python +register( + GRAMPLET, + id="MyAddon", + gramps_target_version="6.0", # major.minor + ... +) +``` + +Gramps matches this **on the major.minor pair** at plugin discovery. A `6.0` addon will not load in 6.1, and a `6.1` addon will not load in 6.0 — the plugin manager silently skips the registration entry. + +### Supporting multiple minors + +The one-addon-per-minor convention is enforced by the branch directory split in the [`addons/`](https://github.com/gramps-project/addons) repo and the matching `maintenance/gramps*` branches in [`addons-source/`](https://github.com/gramps-project/addons-source). For an addon that supports 6.0 and 6.1: + +``` +addons-source @ maintenance/gramps60: MyAddon/MyAddon.gpr.py declares "6.0" +addons-source @ maintenance/gramps61: MyAddon/MyAddon.gpr.py declares "6.1" +``` + +A single `make.py gramps60 build MyAddon` on `maintenance/gramps60` produces the 6.0-targeted `.addon.tgz`; the same command with `gramps61` on `maintenance/gramps61` produces the 6.1-targeted one. See [12-packaging](12-packaging.md) for the workflow. + +When the **code is identical** between minors, the maintainer forward-merges the `maintenance/gramps60` branch into `maintenance/gramps61` and rebuilds — no per-minor source maintenance needed. + +When the code **isn't identical** (e.g. the GExiv2 version handling delta below), the two branches diverge intentionally, and you commit the minor-specific fix to each. + +## Branch targeting for fixes + +The rule that determines which branch a fix lands on differs between the two repos: + +- **`addons-source/`** → `maintenance/gramps60`. Gary cherry-picks forward to `gramps61`. (Gary Griffin, addons-source PR 915, 2026-05-24.) +- **`gramps/`** (core) → `maintenance/gramps61`. Fixes and cleanups go on the current production branch and forward-merge to `master`. Only genuinely new-feature work targets `master`. (jralls, gramps#2298.) + +A reviewer's instruction on a specific PR overrides the default (e.g. Nick-Hall asking for `master` on gramps#2299). See [16-guidelines → Contributor workflow](16-guidelines.md#contributor-workflow) for the normative form. + +## "Applies cleanly" is not "remains correct" + +A cherry-pick that `git` accepts without conflict can still be wrong on the target branch — the branches' *related* code may have changed even though the patch's hunks didn't. + +**Concrete example.** addons-source PR 829 rewrote GExiv2 version handling on `maintenance/gramps61` only. An addon that pins `requires_gi=[("GExiv2", "0.10")]` is fine on 6.0; the same pin on 6.1 may need adjustment because the code that reads the pin has changed shape. A cherry-pick of the addon would land cleanly and still be wrong. + +**The check.** Before treating a cross-branch port as done, diff the related code on the target branch — not just the file the patch touched. Read the surrounding functions; read the modules the declaration interacts with. + +## Notable 6.0 → 6.1 deltas + +The complete delta lives in the Gramps changelog; the entries below are the ones that have repeatedly affected addon authors. + +### Plugin discovery follows symlinks + +Gramps 6.0 plugin discovery **does not** follow symlinks; the addon folder must be physically present under the plugin path. Gramps 6.1 follows symlinks with realpath-based dedup against symlink loops (commit [`9443dcbb30`](https://github.com/gramps-project/gramps/commit/9443dcbb30) on `maintenance/gramps61`, with `_manager_symlinks_test.py` covering both the scan-via-symlink and loop-terminates cases). + +**Impact on dev loop.** On 6.0, copy or `rsync` the working tree into the user plugin directory on every save. On 6.1+ Linux/macOS, symlink once and edit in place. On Windows, the symlink test is skipped because the platform's symlink behaviour is inconsistent without elevated privileges; physical copy remains the safe approach on 6.1+ too. + +### Windows toolchain migrated to UCRT64 + +Gramps' Windows build migrated from MINGW64 to MSYS2 **UCRT64** in gramps PR [#2198](https://github.com/gramps-project/gramps/pull/2198) on `maintenance/gramps61` (merged 2026-04-19). MINGW64's Python target triple is rejected by orjson's `maturin` backend, so the change was forced. + +**Impact on addon Windows testing.** Addon tests run on UCRT64 on 6.1 and master only. Windows testing on 6.0 is unsupported by upstream's addons-source CI. See [07-testing](07-testing.md) for the filename-prefix convention that selects per-OS tests. + +### GExiv2 version handling rewritten + +addons-source PR [829](https://github.com/gramps-project/addons-source/pull/829) rewrote the GExiv2 version handling on `maintenance/gramps61`, in the EditExifMetadata addon. An addon that interacts with GExiv2 via `requires_gi` may need branch-specific declarations. + +**The check.** When the addon imports `GExiv2` or declares it in `requires_gi`, read the EditExifMetadata addon on the *target* branch before assuming a pin is correct. + +### BSDDB-on-Windows skip + +A test-skip rule for BSDDB on Windows landed on `maintenance/gramps61` only. Addons that exercise the BSDDB backend in tests need to account for the absence of BSDDB on Windows 6.1, not assume the 6.0 behaviour transfers. + +## Reading the deprecation signal + +When core deprecates an API, addon authors see two things in order: + +- A `DeprecationWarning` raised the first time the deprecated symbol is touched. Visible when Gramps runs with `python -W default`, or in the Gramps log window at `WARNING` level. +- A scheduled removal in the next major release. + +**Practical step.** Once a release, launch with `python -W default` against `example.gramps` and skim the log window. Every `DeprecationWarning` is a maintenance task for the next minor; deferring them until removal turns "the addon shows up but does nothing" bugs into the dominant porting failure mode. + +For the actual deprecated surface in the running Gramps, the authoritative reference is the source — search `gramps/gen/**/*.py` for `DeprecationWarning` on the target branch. + +## Sanity checks before a port + +1. **Read the new branch's relevant code.** Not the patch — the surrounding code. The patch lands; the assumption around it may have shifted. +2. **Run the addon's tests on the new branch.** The whole point of the per-OS prefix convention in [07-testing](07-testing.md) is to catch this exact case. +3. **Reproduce against `example.gramps` on both branches.** The canonical fixture is identical across minors, so an output difference is an actionable signal. +4. **Check the open PRs against `gramps` and `addons-source` for anything affecting your addon.** A fix may be in flight upstream; verifying that PR is usually better than writing your own. + +## See also + +- [01-overview → Where addons live](01-overview.md#where-addons-live) — the 6.0 vs 6.1 symlink discovery rule, with the dev-loop consequence. +- [04-fundamentals → The `.gpr.py` registration file](04-fundamentals.md#the-gprpy-registration-file) — `gramps_target_version` declaration in context. +- [12-packaging](12-packaging.md) — how the per-minor build flow uses `gramps_target_version`. +- [15-whats-new](15-whats-new.md) — scheduled per-release changes affecting addon authors. +- [16-guidelines → Contributor workflow](16-guidelines.md#contributor-workflow) — normative branch-targeting rules. diff --git a/docs/addon-development/15-whats-new.md b/docs/addon-development/15-whats-new.md new file mode 100644 index 000000000..fec2379da --- /dev/null +++ b/docs/addon-development/15-whats-new.md @@ -0,0 +1,79 @@ +# What's New + +[← Previous](14-compatibility.md) · [Index](01-overview.md) · [Next →](16-guidelines.md) + +## Overview + +API and convention changes that affect addon authors, per Gramps minor release. The audience is someone with a working addon on the previous version asking *"what do I need to know before I bump `gramps_target_version`?"* + +This page is the **addon-author slice** of the change log. It's not the full release notes — those live on the wiki proper. Entries here are filtered for things that affect: + +- the `gramps.gen.*` import surface, +- the plugin-registration surface (`_pluginreg.py`), +- the docgen and report APIs, +- per-addon translation / locale plumbing, +- the addon discovery and loading mechanism. + +For the practical *how to port* guidance — what to check on a cross-version port, when to maintain parallel branches — see [14-compatibility](14-compatibility.md). This page is the inventory; 14-compatibility is the procedure. + +## Gramps 6.1 + +Targeted from `maintenance/gramps61`; `master` until the 6.1.0 release. + +### Added + +- **Plugin discovery follows symlinks.** Symlinking a working-tree addon folder into the user plugin directory now works, with realpath-based dedup so cycles terminate. Commit [`9443dcbb30`](https://github.com/gramps-project/gramps/commit/9443dcbb30), with `_manager_symlinks_test.py` covering both the scan-via-symlink case and loop termination. The dev loop on Linux/macOS becomes *symlink once, edit in place*. (See [01-overview → Where addons live](01-overview.md#where-addons-live).) + +### Changed + +- **Windows toolchain migrated from MINGW64 to MSYS2 UCRT64.** Gramps' Windows build moved in PR [#2198](https://github.com/gramps-project/gramps/pull/2198) (merged 2026-04-19). MINGW64's Python target triple is rejected by orjson's `maturin` backend; the migration was forced. + - **Impact on addon authors:** Windows addon testing targets `maintenance/gramps61` and `master` only — Windows on 6.0 is not upstream-tested. See [14-compatibility → Windows toolchain migrated to UCRT64](14-compatibility.md#windows-toolchain-migrated-to-ucrt64). +- **GExiv2 version handling rewritten.** addons-source PR [829](https://github.com/gramps-project/addons-source/pull/829) rewrote how GExiv2's version is read and pinned. An addon's `requires_gi=[("GExiv2", "0.10")]` declaration may need adjustment; read the EditExifMetadata addon's GExiv2 code on the target branch before assuming a 6.0 pin transfers. See [14-compatibility → GExiv2 version handling rewritten](14-compatibility.md#gexiv2-version-handling-rewritten). +- **BSDDB-on-Windows test skip.** A skip rule for BSDDB on Windows landed on `maintenance/gramps61`. Addons exercising the BSDDB backend in tests need to account for its absence on Windows 6.1 (use `@unittest.skipUnless(...)`; see [07-testing → Skip cleanly](07-testing.md#skip-cleanly)). + +### Deprecated + +*None tracked here yet.* The authoritative reference for runtime deprecations is the source — search `gramps/gen/**/*.py` on the target branch for `DeprecationWarning`. See [14-compatibility → Reading the deprecation signal](14-compatibility.md#reading-the-deprecation-signal) for the recipe. + +### Removed + +*None tracked here yet.* + +## Gramps 6.0 + +The manual's baseline target. Addons declaring `gramps_target_version="6.0"` run on 6.0.x and are not loaded by 6.1 or later (and vice versa); see [14-compatibility → `gramps_target_version` semantics](14-compatibility.md#gramps_target_version-semantics). + +### Added + +- **SQLite became the default database backend.** New trees are SQLite-backed unless the user explicitly chooses BSDDB. Addons that do straight `gramps.gen.db.*` reads keep working unchanged — the abstraction holds — but addons that bypassed the abstraction (e.g. reaching into BSDDB-specific cursor APIs) need to migrate to the portable interface. + +### Changed + +- **Python 3.10+ minimum.** Older Pythons no longer run Gramps 6.0, which means addons can use modern type-hint syntax — `X | None` instead of `Optional[X]`, `list[X]` instead of `typing.List[X]` — without a compatibility shim. See [16-guidelines → Coding style](16-guidelines.md#coding-style). + +### Deprecated + +*Verify against the source.* `DeprecationWarning`s on `maintenance/gramps60` are the authoritative list. + +### Removed + +*None tracked here yet.* + +## Earlier releases + +The 5.x → 6.0 transition was a major release; many APIs changed and the maintenance window for addons targeting earlier minors is closing. The authoritative reference for cross-major changes is the [Gramps wiki's release-notes pages](https://www.gramps-project.org/wiki/index.php/Portal:Using_Gramps#Release_notes). + +Practical guidance: addons still targeting 5.x should pin `gramps_target_version="5.2"` (the last 5.x minor) and live on the matching `addons-source` branch; the cross-major port is a separate exercise from the per-minor deltas this page tracks. + +## How to read this page + +- Each release section is **incremental** — entries describe what changed *from the previous minor*, not the cumulative API surface. +- Where an entry has an upstream commit, PR, or addon-side fix, it's cited inline so the change is auditable. Entries without a citation reflect conventions that emerged rather than discrete commits. +- The *current* surface (what's available right now) lives in [06-api-reference](06-api-reference.md), not here. + +## See also + +- [14-compatibility](14-compatibility.md) — porting an addon across these releases; the practical companion to this inventory. +- [06-api-reference](06-api-reference.md) — the current `gramps.gen.*` surface. +- [Portal:Using Gramps → Release notes](https://www.gramps-project.org/wiki/index.php/Portal:Using_Gramps#Release_notes) — upstream release notes (full, not addon-filtered). +- [`gramps/NEWS`](https://github.com/gramps-project/gramps/blob/maintenance/gramps61/NEWS) — the in-tree change log on the target branch. diff --git a/docs/addon-development/16-guidelines.md b/docs/addon-development/16-guidelines.md new file mode 100644 index 000000000..fa427ad14 --- /dev/null +++ b/docs/addon-development/16-guidelines.md @@ -0,0 +1,183 @@ +# Rules + +[← Previous](15-whats-new.md) · [Index](01-overview.md) · [Next →](17-roadmap.md) + +## Overview + +Normative reference for addon authors. Conceptual / how-to material lives in the other section pages; this page enumerates the guidelines and is the one to cite in code review. + +## Repository scope + +- **This page applies to the addon repository — [`gramps-project/addons-source`](https://github.com/gramps-project/addons-source).** It does **not** govern Gramps core. +- Core contributions (`gramps-project/gramps`) follow the separate [Core Development — Rules](https://gramps-project.org/wiki/index.php/Gramps_6.1_Wiki_Manual_-_Core_Development_-_Rules) page. The two repositories diverge on branch target, test layout, translation tooling, and which static checks are enforced — do not transfer a rule across without checking it here. +- The full Python coding standard is inherited from core's `../gramps/AGENTS.md`; this page restates the parts addon code review enforces and adds the addon-specific structure, packaging, and translation rules that live outside that file. +- **When in doubt, the authoritative source wins and is what to check.** These pages are a convenience restatement. On coding style, core's `../gramps/AGENTS.md` is the source of truth; on addon-specific rules, the authority is upstream `addons-source` (its `CONTRIBUTING.md` and a maintainer's ruling on the PR). Where this page is silent, ambiguous, or disagrees with the authoritative source on the *target branch*, that source wins — verify against it rather than relying on this page from memory. +- **Core stands in where this page doesn't — one way only.** Where this page is not specific or prescriptive on a point, the [Core Development — Rules](https://gramps-project.org/wiki/index.php/Gramps_6.1_Wiki_Manual_-_Core_Development_-_Rules) page (and core's `AGENTS.md`) is the default that fills the gap — addons inherit from core. The fallback runs in this direction only: where this page *is* prescriptive on an addon-specific concern (structure, packaging, branch target, test layout — `tests/` + `test_*.py`, `maintenance/gramps60`), it governs and core does not override it; and the addon guidelines never fill a gap in the core page. + +## Conventions + +RFC 2119 keywords, with our short forms: + +| Keyword | Meaning | +|---------|---------| +| **MUST** / **MUST NOT** | Required; a violation is a defect | +| **SHOULD** / **SHOULD NOT** | Strongly recommended; deviate only with a stated reason | +| **MAY** | Allowed | + +Where a rule has a known origin — an upstream PR, a maintainer ruling, a Mantis bug — it's cited inline so the rule is auditable. + +## Structure + +- **MUST**: the addon's folder name is a valid Python import name (an importable identifier — no spaces). Gramps puts each addon's directory on `sys.path` and addons share code via `import ` (see [the upstream Addons development page](https://gramps-project.org/wiki/index.php/Addons_development) → "name your addons with a name appropriate for Python imports"). The folder name need **not** match the `id` in `.gpr.py`: the registration `id` is an independent plugin key and routinely differs (e.g. folder `DeepConnectionsGramplet` ↔ id `Deep Connections Gramplet`), and one folder may register several plugins with unrelated ids. +- **MUST**: `.gpr.py` declares `gramps_target_version` matching the Gramps minor the addon targets. +- **MUST**: `fname` points to an implementation module shipped in the same folder. +- **MUST**: the addon is physically present under the plugin path — a physical copy works on every Gramps version and OS. (Gramps 6.1+ also discovers an addon reached via a symlink, but a physical copy is the portable default.) +- **MUST NOT**: import `register`, `GRAMPLET`, `STABLE`, `_`, or any other name Gramps injects into the `.gpr.py` namespace. +- **MUST NOT**: add `__init__.py` to the addon directory itself. The plugin loader puts the addon dir on `sys.path` and imports `.py` by name; making the addon dir a regular package disturbs that resolution and can trigger the [Mantis 12691](https://gramps-project.org/bugs/view.php?id=12691) submodule-binding trap. (See [07-testing → Why `tests/__init__.py` exists](07-testing.md#why-tests__init__py-exists).) +- **MUST** (`TOOL` kind): register an `optionclass` even when the tool takes no options. Gramps refuses to load a `TOOL` without one; an empty `tool.ToolOptions` subclass is sufficient. +- **SHOULD**: ship a `po/` directory with at least `template.pot` if any user-visible string exists. Generate it with `make.py init ` (see [12-packaging](12-packaging.md)); if it's missing the maintainer creates it on initial check-in. +- **MAY**: ship a `tests/` package with an `__init__.py` marker and at least one test — most existing addons predate addon unit tests. When tests are shipped, the `__init__.py` marker keeps dotted-path loading deterministic and the layout rules under *Testing* apply; a bug fix still **SHOULD** ship a regression test. +- **MAY**: ship multiple plugin kinds from a single addon — multiple `register(...)` calls in one `.gpr.py`, and/or multiple `.gpr.py` files in the addon folder (the loader scans every `*.gpr.py`). + +## Source location + +- **MUST**: edit addon source in `addons-source/`, never in the live plugin directory. The auto-sync runs source → installed plugin one-way; edits in the live dir are silently overwritten on the next source save. + +## Translation + +The full how-to (registration setup, `make.py` lifecycle, Glade runtime-override pattern, function reference) lives in [11-internationalization](11-internationalization.md). The rules below are what code review enforces. + +- **MUST**: wrap every user-visible string with `_()`. +- **MUST NOT**: `import _` in `.gpr.py` — Gramps' plugin loader injects it. Implementation modules **MUST** bind it explicitly via `_ = glocale.get_addon_translator(__file__).gettext`. +- **MUST** (multi-file packages): when the addon's code is split across a nested package, bind `_` **once at the addon root** — the directory that holds `locale/`, in a root-level module (e.g. `_i18n.py`) — and import it everywhere else by **bare name** (`from _i18n import _`), **not** a `.`-prefixed path: the addon dir is on `sys.path` and its root is **not** a package (see *Structure* → MUST NOT `__init__.py`), so a root-level module imports directly, whereas `from ..i18n import _` raises `'' is not a package` at import time. `get_addon_translator(filename)` derives the catalog dir as `dirname(abspath(filename)) + "/locale"` (`gramps.gen.utils.grampslocale`), so a `get_addon_translator(__file__)` call from a nested module (e.g. `myaddon/views/tab.py`) resolves `myaddon/views/locale/`, which doesn't exist, and a non-English user silently gets the untranslated string. The flat `_ = glocale.get_addon_translator(__file__).gettext` form above is correct only because that module sits at the addon root; from a nested module, anchor the path at the root (e.g. `get_addon_translator(os.path.join(ADDON_ROOT, "_"))` — only `dirname(...)` is read, so the basename is an unused placeholder) instead of passing `__file__`. (NameSuite i18n-anchor fix, 2026-06-25.) +- **SHOULD**: verify an addon translation against an **addon-owned** msgid — one that appears only in the addon's `template.pot`, never a string that also exists in core (e.g. `"Given name"`). `get_addon_translator` returns the **core** translator with the addon catalog only as a *fallback*, so a core string renders translated whether or not the addon binding resolves — it cannot prove the fix. (Same fix: the original check used a core string and demonstrated nothing.) +- **MUST NOT**: wrap an f-string or `.format()` result in a translation function. `xgettext` cannot extract dynamically built strings. + - **Bad:** `_(f"User {name}")`, `_("User {}".format(name))` + - **Good:** `_("User %s") % name` +- **MUST** (Glade): translatable strings in `.glade` / `.ui` files are **not** picked up by the addon translation tooling — the extractor only sees Python. For each translatable Glade string, give the widget a meaningful `id`, mark the string with `translatable="yes"` (optionally with a `"context|"` prefix), and override the label at runtime in Python: `self.get_widget("place_name_label").set_label(_("place|Name:"))`. +- **SHOULD**: use `ngettext(singular, plural, n)` for plural forms. +- **SHOULD**: use the pipe-prefix form `_("Context|String")` whenever a word could carry multiple senses (e.g. `_("book|Title")` vs `_("person|Title")`). This is the convention used throughout `addons-source` and is what translators see in the `.po` file. The two-arg form `_(msg, context)` works equivalently. **MUST NOT** call `pgettext` or `sgettext` directly — go through `_`. +- **SHOULD**: use `N_("…")` to mark a string for extraction without translating it at call time (e.g. for module-level constants that are translated later when displayed). + +> Addons have no `POTFILES.in` to maintain by hand — the per-addon `po/template.pot` is regenerated by `make.py init ` (see [12-packaging](12-packaging.md)). Maintaining `po/POTFILES.in` / `POTFILES.skip` is a **core** rule; see the [Core Development — Rules](https://gramps-project.org/wiki/index.php/Gramps_6.1_Wiki_Manual_-_Core_Development_-_Rules) page. + +## Runtime + +- **MUST**: perform every database write inside a `DbTxn`: + ```python + with DbTxn(_("Adding example"), db) as trans: + db.add_person(person, trans) + ``` +- **MUST**: declare runtime imports in `requires_mod` using the *importable* module name (`PIL`), not the PyPI distribution name (`Pillow`). +- **MUST**: verify each `requires_mod` entry with `importlib.util.find_spec("")` on a system with the package installed before publishing. +- **MUST**: use `requires_gi` for GObject-Introspection bindings, with version strings. The version pin **must match what the code actually imports** at runtime — pins can drift between Gramps minors (e.g. GExiv2 handling was rewritten on `maintenance/gramps61` per addons-source PR 829), so verify the pin against the target branch's related code, not just the previous branch's working declaration. +- **MUST NOT**: mutate process-global state that Gramps' startup owns — run or quit the GTK main loop (`Gtk.main()` / `Gtk.main_quit()`), install screen-wide CSS / retheme the icon theme / change `Gtk.Settings`, replace `sys.excepthook`, call `locale.setlocale` or `gettext.install`, configure the root logger, leave permanent `sys.path` entries, or set `os.environ` keys. An addon is a guest in Gramps' process; the full startup surface with per-item alternatives is [04-fundamentals → The provided environment](04-fundamentals.md#the-provided-environment). +- **SHOULD**: use handles (`PersonHandle`, etc.) for internal traversal; reserve Gramps IDs (`I0001`, …) for user-facing display. Handles are internal and stable; Gramps IDs are user-editable and rewritten in bulk by the Reorder Gramps IDs tool. +- **SHOULD**: import only from `gramps.gen.*`. `gramps.gui.*` and `gramps.plugins.*` are internal to the shipped distribution and break across Gramps versions. +- **SHOULD**: use a module-level logger (`LOG = logging.getLogger(__name__)`); **MUST NOT** use `print()` for diagnostic output. +- **SHOULD**: raise existing exceptions from `gramps.gen.errors` and `gramps.gen.db.exceptions` before inventing a new class. +- **SHOULD**: raise `HandleError` for invalid or missing handles. +- **SHOULD**: compare backlink class names by string. `db.find_backlink_handles(handle)` yields `(class_name, handle)` tuples where `class_name` is `"Person"` / `"Family"` / … as a `str`, not the Python class — `if cls is Person:` always evaluates `False`. +- **MAY**: introduce a new exception class only when none of the existing ones accurately represent the error condition. + +## Testing + +- **MUST**: use stdlib `unittest` — never `pytest`. Gramps itself standardises on `unittest`, which keeps addon tests contributable upstream. +- **MUST**: name test files `test_*.py` and place them in a `tests/` package alongside the addon module. +- **MUST**: scope platform-specific tests with the correct prefix: + + | Prefix | Where it runs | + |--------|---------------| + | `test_*.py` | All platforms | + | `test_linux_*.py` | Linux only | + | `test_windows_*.py` | Windows only | + | `test_integration_*.py` | Linux only — full-pipeline / DB-backed | + +- **MUST**: tests run cleanly without the addon's `requires_mod` dependencies installed in the Python that runs them — mock at the import boundary, or skip cleanly with `@unittest.skipUnless(...)`. Mac contributors can't easily install addon deps into the Gramps Python, and there's no Gramps debug-mode on Mac. (Gary Griffin, 2026-05-16.) +- **MUST**: never call `gi.require_version` in addon modules or test files. At runtime Gramps pins Gtk/Gdk before any plugin loads (`gramps/grampsapp.py`, `gramps/gen/constfunc.py`); under test, the pins live once in addons-source's **repo-root** `tests/__init__.py` (addons-source PR 950) — the per-addon `tests/__init__.py` stays empty, and tests run from the repository root so the pinned environment holds. Redundant pins MAY be removed from files already being touched. A module-level pin passes unit tests but breaks inside Gramps as soon as the hardcoded pin and the running version diverge — see [07-testing → The GTK-pin contract](07-testing.md#the-gtk-pin-contract). +- **SHOULD**: ship a regression test with every bug fix that **fails pre-fix and passes post-fix**. Doc-only PRs are the only exception. (At PR level this hardens to a MUST-with-escape — the test, or an explicit "no test because X" rationale; see *Contributor workflow*.) +- **SHOULD**: prefer `example.gramps`-backed tests over mocked DBs for DB-traversal logic — real data has cross-typed backlinks and ID-normalisation shapes that mocks don't reproduce. +- **MAY**: ship mocked unit tests alongside real-DB tests as complementary coverage. + +## Coding style + +**The coding standard is core's `../gramps/AGENTS.md`, in full — this section lists only the addon deltas.** Black, Python 3.10+ type hints (`X | None`, `list[X]`), Sphinx docstrings, import grouping with comment headers, class-header navigation comments, the `cb_` callback prefix, handle/ID types from `gramps.gen.types` — all are specified there and apply to addon Python unchanged. They are **not** restated below; anything this section is silent on follows core. The deltas are only these: + +- **Enforcement is advisory, so the core standard's coding MUSTs read as SHOULDs here.** addons-source runs no `black` / `mypy` / pylint gate — the reviewer weighs the standard; CI does not block on it. You **SHOULD** still run `black --check` before pushing, so the maintainer's cherry-pick forward to gramps61 stays clean. +- **Two rules are not softened — they stay MUST despite the lighter gate:** every new `.py` file carries a GPL-2.0-or-later license header with copyright, and every user-visible string is wrapped with `_()` (§Translation). +- **`gen`-self-containment, reframed.** Core's MUST that `gramps.gen.*` import no other submodule has no direct addon analog, but addon code **SHOULD** uphold the same discipline against itself: factor pure logic into modules that don't import `gramps.gui.*`, so it stays unit-testable without a display. + +## Contributor workflow + +- **MUST**: one logical fix per PR. Bundling hides mistakes. +- **MUST**: target the right branch — addon changes (`addons-source`) → `maintenance/gramps60`. The maintainer cherry-picks forward to `gramps61`. (Gary Griffin on addons-source PR 915, 2026-05-24.) A reviewer's instruction on a specific PR wins over the default targeting. (e.g. Nick-Hall on gramps#2299.) Core changes target a different branch — see the [Core Development — Rules](https://gramps-project.org/wiki/index.php/Gramps_6.1_Wiki_Manual_-_Core_Development_-_Rules) page. +- **MUST**: branch from `upstream/`, not the fork's tracking copy — fork bases drift (e.g. PRs 2315/2316 carried a stray `AGENTS.md` from the fork). +- **MUST NOT**: bump the addon's `version` field in an addons-source PR. The maintainer manages versions centrally. (Caught on PR 911, bug 12572.) +- **MUST**: a bug-fix PR includes a regression test, or an explicit "no test because X" rationale plus a manual repro. "Add the test later" is not an option. +- **MUST**: open the PR body with a **`**User impact:**`** line (before Root cause), then structure it **Summary / What to look at / Root cause / Fix / Verification**, citing `path:lines` on the branch the PR targets in the Verification "Checked" line (the #106 format). +- **MUST**: when the PR modifies an addon, **call out its current maintainer** — add an `## Affected addon` section to the PR body that **@-mentions the addon's current maintainer**, a heads-up so they are *aware* of the change and don't miss it. This is awareness, not attribution. "Current maintainer" = the addon's `.gpr.py` `maintainers` field when declared, otherwise its `authors` (an addon with no separate maintainer is maintained by its original author — Doug's "original developer (or contributors)" and Nick's "current maintainer" are the same role). The `.gpr.py` records names/emails, not GitHub handles — resolve a handle best-effort from the declared email so the mention notifies, and name the person when no handle resolves. (Raised on addons-source PR #946 — Doug Blank: *"otherwise I could miss fixes to my addons"*; Nick Hall: *"mention the current maintainer if one exists."*) +- **OPTIONAL**: reference the Mantis bug in the PR body **when one exists** — a Mantis reference is optional for addons-source, since many addon fixes have no Mantis ticket (they're tracked as fork GitHub issues, or are ticketless). addons-source also does not use the `Fixes #NNNN` commit-message trailer at all — that is the core convention; see the [Core Development — Rules](https://gramps-project.org/wiki/index.php/Gramps_6.1_Wiki_Manual_-_Core_Development_-_Rules) page. +- **MUST**: keep upstream-repo cross-references out of PR text and fork issues — reference *other* upstream PRs/issues in **plain text** ("upstream PR 949"), never a GitHub URL or `owner/repo#NNN` cross-ref (it back-links/notifies that thread). The `#nnnn` Mantis reference and the PR's own target are exempt. Authoritative: `docs/INTEGRATION.md` §"No upstream-repo links". +- **MUST NOT**: merge across branches. Rebase rather than merge — PRs with merge commits are rejected upstream. +- **MUST NOT**: cosmetically update in-flight upstream PRs. Parity, "rebase is clean," and "branch is behind" are not reasons to force-push. Push only when a specific correctness issue needs fixing. +- **SHOULD**: before writing any fix, check upstream isn't ahead — merged history on the target branch AND `master`, *plus* closed and rejected PRs on the *affected file* (not just the bug number). Closed PRs are signal: a closed-unmerged PR with the same fix shape is the maintainer's "no." +- **SHOULD**: if a PR already exists for the bug, verify it instead of duplicating. Merged → confirm-and-close; open → review and defer to the maintainer; closed → treat as the maintainer's "no." +- **SHOULD**: reproduce against `example.gramps` first — it's the canonical fixture and "couldn't reproduce" is the most common reason a fix stalls in triage. +- **MAY**: open as a draft PR for early review or to publish work-in-progress; mark ready when the change is complete and the author has re-read the diff with fresh eyes. + +## Verification before commit + +- **MUST**: find a test procedure before committing — local run, dry-run, snippet check. Never commit untested changes. +- **MUST**: treat a green mechanical check (lint, `git cherry-pick` applies, build green, `py_compile` exits 0) as evidence of *that narrow check*, not of correctness. Name what the check verified and what it left unverified. +- **MUST**: after pushing a PR branch, watch the PR's CI checks until they finish (e.g. `gh pr checks --watch`). Local pre-commit catches static checks only; test failures surface in CI's actual unit-test run. + +## Commit messages + +Commit messages are parsed by scripts that update Mantis BT and generate the ChangeLog / News files for releases. Formatting must be followed precisely. + +- **MUST**: the first line is a short summary, **≤ 70 characters**. +- **MUST**: the description is separated from the summary by a single blank line, and wrapped at **80 characters**. +- **MUST**: describe the change from the user's perspective. Don't recap the diff — `git diff` exists. +- **SHOULD**: use complete sentences in the description. +- **MUST**: reference another commit by its **full hash**, not a short hash. GitHub auto-hyperlinks full hashes; short hashes in brackets do not link. +- **MUST**: the Mantis trailer is on the **last** line of the commit message, separated from the description by a single blank line. + +### Mantis trailer keywords + +To **resolve** a bug (closes it on commit): + +``` +Fixes #12345 +Fixed #12345 +Resolves #12345 +Resolved #12345 +Fixes #12345, #67890 +``` + +To **link** to a bug (cross-reference without closing): + +``` +Bug #12345 +Issue #12345 +Report #12345 +Bugs #12345, #67890 +``` + +Bare numbers (no `#`) and URLs both miss the auto-link — use the `#NNNN` form. Note this is the opposite of the convention *inside* MantisBT itself, where `#NNNN` auto-links to another Mantis issue and bare numbers are preferred; here, inside Git commit messages and GitHub PR bodies, `#NNNN` is what hooks the MantisBT scripts. + +For the trailer to wire up on Mantis, the Git **author** or **committer** has to be a developer on the Mantis bug tracker. The Git name must match the Mantis username or real name, or the Git email must match the Mantis email. + +### addons-source: bug reference in PR body + +addons-source PRs don't use `Fixes #NNNN` in the commit message — that trailer is the core convention. A Mantis bug reference in the PR body is **optional**: include it when the fix has a Mantis ticket, but many addon fixes have none (fork GitHub issue, or ticketless), and those need no reference. A present-but-malformed reference is still wrong. + +## See also + +- [Overview](01-overview.md) +- [Fundamentals](04-fundamentals.md) +- [Testing](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Addon_Testing) +- [Code analysis](10-code-analysis.md) +- [Packaging](12-packaging.md) +- `../gramps/AGENTS.md` — the full Python coding standard inherited here. +- [addons-source CONTRIBUTING.md](https://github.com/gramps-project/addons-source/blob/maintenance/gramps60/CONTRIBUTING.md) +- [Committing policies](https://www.gramps-project.org/wiki/index.php/Committing_policies) — upstream's commit-message + Mantis-trailer rules. diff --git a/docs/addon-development/17-roadmap.md b/docs/addon-development/17-roadmap.md new file mode 100644 index 000000000..bec8e65ff --- /dev/null +++ b/docs/addon-development/17-roadmap.md @@ -0,0 +1,106 @@ +# Roadmap + +[← Previous](16-guidelines.md) · [Index](01-overview.md) + +## Overview + +Forward-looking view of the addon-development surface — what's planned, what's in flight, what's slated for deprecation, and what open questions will eventually become rules. The audience is an addon author asking "what do I need to plan around?" + +This page is the **prospective** counterpart to [What's new](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Addon_Development_-_Whats_New), which is retrospective. An item moves from this page to *What's new* once it ships in a release. + +## How to read this page + +Each entry should answer four things: + +| Field | Meaning | +|-------|---------| +| **Status** | proposed / accepted / in-flight / shipped / deferred / rejected | +| **Target** | Gramps version (`6.1`, `6.2`, ...) or "unscheduled" | +| **Impact** | what addon authors need to do (rewrite / opt-in / nothing) | +| **Tracking** | PR / Mantis bug / wiki RFC / mailing-list thread | + +A roadmap entry without a tracking link is a wish, not a plan; either add the link or move the entry to a separate "ideas" section. + +## In flight + + + +- _none recorded yet_ + +## Accepted but not yet implemented + + + +- _none recorded yet_ + +## Deprecations and removals + + + +- _none recorded yet_ + +## Open questions + + + +- _none recorded yet_ + +## Deferred / rejected + + + +- _none recorded yet_ + +## Documentation roadmap + +The doc set itself is in flight. Pages with `managed: false` front-matter are draft stubs and will not appear in published output until promoted. Current draft state — flip to `managed: true` page by page as content lands: + +### Publishing-pipeline conventions (now supported) + +What `md2wiki.py` and `md2pdf.py` handle as of 2026-05-30 — pages authored with these conventions render correctly in both wikitext and PDF output. Verified by running both pipelines on [Fundamentals](04-fundamentals.md) (which contains an SVG embed + Obsidian-internal links). + +| Convention | Where converted | Notes | +|------------|-----------------|-------| +| `![[_media/foo.svg\|cap]]` Obsidian embed | `mdcommon.convert_obsidian_embeds` | Becomes `![cap](_media/foo.svg)` before pandoc | +| `[[Page]]` / `[[Page\|label]]` Obsidian-internal link | `mdcommon.convert_obsidian_internal_links` | Resolved via `mdcommon.build_title_map` (filename-stem → wiki title); unresolved targets error loudly | +| Markdown image with SVG src in PDF | `_preconvert_svgs` (md2pdf) | Pre-converted to PDF via `rsvg-convert` or `inkscape`; embeds natively in xelatex | +| Markdown image with relative path in PDF | `--resource-path` to pandoc | Resolved against the source file's directory | +| `[[File:_media/foo.svg]]` post-pandoc wikitext | `mdcommon.basenameify_file_refs` | Becomes `[[File:foo.svg]]` (MediaWiki's File: namespace is flat) | +| Media files alongside pages | `wikitransport.upload_if_changed` + `publish.upload_media_for` | SHA-1 dedup; uploaded BEFORE the page edit so refs never render red | +| HTML comments | `mdcommon.stash_html_comments` | Stashed around Obsidian preprocessors so syntax inside comments is not rewritten | + +What the pipeline already handled before these additions: +- `[label](wiki:Page_Name)` → wikitext `[[Page|label]]` / PDF anchor or external URL. +- `` template shims → raw `{{...}}` wikitext / dropped from PDF. +- YAML front-matter → `title`, `categories`, `managed`. +- Fenced code with language tags, tables. + +### Page-by-page state + +The section is substantive across all seventeen pages. Open deepening work: + +- [Tutorials](02-tutorials.md) — the screenshots for each tutorial's "Try it" closer are pending capture. +- [Data access](05-data-access.md) — worked examples for some API touch-points are still thin. +- [API Reference](06-api-reference.md) — needs periodic re-synchronisation against `gramps/gen/__init__.py` on the maintenance branch this manual targets. + +## See also + +- [What's new](https://gramps-project.org/wiki/index.php/Gramps_6.0_Wiki_Manual_-_Addon_Development_-_Whats_New) — retrospective counterpart. +- [Compatibility](14-compatibility.md) — porting guidance once an item ships. +- [Mantis bug tracker](https://gramps-project.org/bugs) — feature requests and design discussions originate here. +- [Gramps mailing lists](https://gramps-project.org/contact/) — where larger design questions get hashed out. diff --git a/docs/addon-development/README.md b/docs/addon-development/README.md new file mode 100644 index 000000000..95686ba8a --- /dev/null +++ b/docs/addon-development/README.md @@ -0,0 +1,24 @@ +# Gramps Addon Development manual + +The addon authors' manual for Gramps. Start at +[01-overview.md](01-overview.md) — the overview and section map. + +## Pages + +- [Addon Development](01-overview.md) +- [Tutorials](02-tutorials.md) +- [Addon Kinds](03-addon-kinds.md) +- [Fundamentals](04-fundamentals.md) +- [Data access](05-data-access.md) +- [API Reference](06-api-reference.md) +- [Testing](07-testing.md) +- [Debug](08-debug.md) +- [Troubleshoot](09-troubleshoot.md) +- [Code Analysis](10-code-analysis.md) +- [Internationalization](11-internationalization.md) +- [Packaging](12-packaging.md) +- [Community](13-community.md) +- [Compatibility](14-compatibility.md) +- [What's New](15-whats-new.md) +- [Rules](16-guidelines.md) +- [Roadmap](17-roadmap.md) diff --git a/docs/addon-development/_media/addon-kinds-ui-map.svg b/docs/addon-development/_media/addon-kinds-ui-map.svg new file mode 100644 index 000000000..f5507e356 --- /dev/null +++ b/docs/addon-development/_media/addon-kinds-ui-map.svg @@ -0,0 +1,172 @@ + + + + + + + + + + + + Gramps main window — where each addon kind appears + + + + + + + + + + File + + IMPORT / EXPORT + + + Edit + + + View + + + Reports + + REPORT + + + Tools + + TOOL + + + Windows + Help + + + + + [ toolbar ] + + + + + Navigator + ▸ Dashboard + ▸ People + ▸ Relationships + ▸ Families + ▸ Events + ▸ Places + ▸ Geography + ▸ Sources + ▸ Citations + ▸ Repositories + ▸ Media + ▸ Notes + + + + Main view area + (content varies by selected Navigator category) + + + + I0001 John Doe 1850– + + I0002 Jane Smith 1853– + + I0003 ... + + + + + Sidebar + + [ gramplet ] + + [ gramplet ] + + [ gramplet ] + + + + Bottombar + + [ gramplet ] + + [ gramplet ] + + [ gramplet ] + + [ gramplet ] + + + + + + SIDEBAR + — one per Navigator category + + + + + VIEW + — alternative way to browse a category + + + + + QUICKVIEW + — right-click context menu on a row + + + + + RULE + — Edit ▸ Person Filter Editor ▸ Add Rule + + + + + MAPSERVICE + — Geography view tile source + + + + + GRAMPLET + — Dashboard, sidebar, or bottombar widget + + + + + + Kinds with no direct UI surface + + DOCGEN — output format used by reports + DATABASE — backend selected at tree creation + RELCALC — used by Relationships view (per locale) + THUMBNAILER — media thumbnail generator + CITE — citation formatter style + GENERAL — shared library / pluggable category + + + Schematic — relative positions match Gramps 6.0's default layout; not pixel-accurate. See chapter 04-addon-kinds for the registration constants and base classes per kind. + diff --git a/docs/addon-development/_media/data-model.dot b/docs/addon-development/_media/data-model.dot new file mode 100644 index 000000000..dae803fc4 --- /dev/null +++ b/docs/addon-development/_media/data-model.dot @@ -0,0 +1,80 @@ +// Gramps primary objects and the most-traversed relationships. +// Regenerate the SVG with: +// dot -Tsvg data-model.dot -o data-model.svg +// +// Convention: +// - Solid arrow with no label = direct handle list +// - Solid arrow labelled "Ref" = goes through a ref object +// carrying metadata (Role, child +// relation, etc.) +// - Dashed arrow = reverse direction reached via +// db.find_backlink_handles() +// +// Notes and Tags can be attached to any primary object; omitted from +// the diagram to keep arrows readable, called out in the caption. + +digraph data_model { + rankdir=LR + bgcolor="transparent" + pad=0.25 + nodesep=0.5 + ranksep=0.9 + splines=true + + node [ + shape=box, + style="rounded,filled", + fillcolor="#eef2f7", + color="#4a5b6e", + fontname="Helvetica", + fontsize=11, + margin="0.18,0.10" + ] + edge [ + fontname="Helvetica", + fontsize=9, + color="#4a5b6e", + fontcolor="#4a5b6e" + ] + + // ---- Primary objects ---- + person [label="Person"] + family [label="Family"] + event [label="Event"] + place [label="Place"] + citation [label="Citation"] + source [label="Source"] + repository [label="Repository"] + media [label="Media"] + + // ---- Person <-> Family (two role-distinct relationships) ---- + person -> family [label=" parent_of\n family_handle_list "] + family -> person [label=" ChildRef ", style=solid] + + // ---- Events via EventRef (carries Role) ---- + person -> event [label=" EventRef\n (Role) "] + family -> event [label=" EventRef\n (Role) "] + + // ---- Places ---- + event -> place [label=" place_handle "] + place -> place [tailport="s", headport="s", label=" enclosed_by "] + + // ---- Sourcing chain ---- + person -> citation [label=" CitationRef "] + family -> citation [label=" CitationRef "] + event -> citation [label=" CitationRef "] + place -> citation [label=" CitationRef "] + citation -> source [label=" source_handle "] + source -> repository [label=" RepoRef "] + + // ---- Media ---- + person -> media [label=" MediaRef "] + event -> media [label=" MediaRef "] + source -> media [label=" MediaRef "] + + // ---- Layout hints to control column order ---- + { rank=same; person; family } + { rank=same; citation; media } + { rank=same; source } + { rank=same; repository } +} diff --git a/docs/addon-development/_media/data-model.svg b/docs/addon-development/_media/data-model.svg new file mode 100644 index 000000000..119275175 --- /dev/null +++ b/docs/addon-development/_media/data-model.svg @@ -0,0 +1,168 @@ + + + + + + +data_model + + +person + +Person + + + +family + +Family + + + +person->family + + +  parent_of +  family_handle_list   + + + +event + +Event + + + +person->event + + +  EventRef +  (Role)   + + + +citation + +Citation + + + +person->citation + + +  CitationRef   + + + +media + +Media + + + +person->media + + +  MediaRef   + + + +family->person + + +  ChildRef   + + + +family->event + + +  EventRef +  (Role)   + + + +family->citation + + +  CitationRef   + + + +place + +Place + + + +event->place + + +  place_handle   + + + +event->citation + + +  CitationRef   + + + +event->media + + +  MediaRef   + + + +place:s->place:s + + +  enclosed_by   + + + +place->citation + + +  CitationRef   + + + +source + +Source + + + +citation->source + + +  source_handle   + + + +repository + +Repository + + + +source->repository + + +  RepoRef   + + + +source->media + + +  MediaRef   + + + diff --git a/docs/addon-development/_media/packaging-pipeline.dot b/docs/addon-development/_media/packaging-pipeline.dot new file mode 100644 index 000000000..666835231 --- /dev/null +++ b/docs/addon-development/_media/packaging-pipeline.dot @@ -0,0 +1,85 @@ +// Three-repo packaging pipeline: addons-source -> make.py -> addons -> user. +// Regenerate the SVG with: +// dot -Tsvg packaging-pipeline.dot -o packaging-pipeline.svg + +digraph packaging_pipeline { + rankdir=LR + bgcolor="transparent" + pad=0.25 + nodesep=0.4 + ranksep=0.55 + + node [ + shape=box, + style="rounded,filled", + fillcolor="#eef2f7", + color="#4a5b6e", + fontname="Helvetica", + fontsize=11, + margin="0.18,0.10" + ] + edge [ + fontname="Helvetica", + fontsize=9, + color="#4a5b6e", + fontcolor="#4a5b6e" + ] + + // Repos and stages. + subgraph cluster_source { + label="addons-source repository" + labelloc="b" + fontname="Helvetica" + fontsize=10 + fontcolor="#4a5b6e" + color="#9aacc0" + style="rounded,dashed" + margin=10 + + source [label="MyAddon/\nMyAddon.gpr.py\nMyAddon.py\npo/\ntests/"] + } + + make [ + label="make.py gramps60 build MyAddon\n(compile po, package files)", + shape=box, + style="filled", + fillcolor="#dfe7f1" + ] + listing [ + label="make.py gramps60 listing MyAddon\n(refresh listings JSON)", + shape=box, + style="filled", + fillcolor="#dfe7f1" + ] + + subgraph cluster_addons { + label="addons repository" + labelloc="b" + fontname="Helvetica" + fontsize=10 + fontcolor="#4a5b6e" + color="#9aacc0" + style="rounded,dashed" + margin=10 + + tgz [label="gramps60/download/\nMyAddon.addon.tgz"] + listings [label="gramps60/listings/\n*.json"] + } + + manager [label="Gramps in-app\naddon manager"] + user [ + label="User plugin dir\n~/.local/share/gramps/\ngramps60/plugins/MyAddon/", + fillcolor="#e6efe2", + color="#5a7251" + ] + + // Edges. + source -> make [label=" author edits "] + source -> listing [style=invis] // keep alignment + make -> tgz [label=" build "] + make -> listing [style=dotted, arrowhead=none] + listing -> listings [label=" listing "] + tgz -> manager [label=" HTTPS fetch "] + listings -> manager [label=" HTTPS fetch "] + manager -> user [label=" install /\n update "] +} diff --git a/docs/addon-development/_media/packaging-pipeline.svg b/docs/addon-development/_media/packaging-pipeline.svg new file mode 100644 index 000000000..837a972a5 --- /dev/null +++ b/docs/addon-development/_media/packaging-pipeline.svg @@ -0,0 +1,124 @@ + + + + + + +packaging_pipeline + +cluster_source + +addons-source repository + + +cluster_addons + +addons repository + + + +source + +MyAddon/ +MyAddon.gpr.py +MyAddon.py +po/ +tests/ + + + +make + +make.py gramps60 build MyAddon +(compile po, package files) + + + +source->make + + +  author edits   + + + +listing + +make.py gramps60 listing MyAddon +(refresh listings JSON) + + + + +make->listing + + + + +tgz + +gramps60/download/ +MyAddon.addon.tgz + + + +make->tgz + + +  build   + + + +listings + +gramps60/listings/ +*.json + + + +listing->listings + + +  listing   + + + +manager + +Gramps in-app +addon manager + + + +tgz->manager + + +  HTTPS fetch   + + + +listings->manager + + +  HTTPS fetch   + + + +user + +User plugin dir +~/.local/share/gramps/ +gramps60/plugins/MyAddon/ + + + +manager->user + + +  install / +  update   + + + diff --git a/docs/addon-development/_media/plugin-discovery.dot b/docs/addon-development/_media/plugin-discovery.dot new file mode 100644 index 000000000..bb0179072 --- /dev/null +++ b/docs/addon-development/_media/plugin-discovery.dot @@ -0,0 +1,40 @@ +// Plugin discovery and load sequence. +// Regenerate the SVG with: +// dot -Tsvg plugin-discovery.dot -o plugin-discovery.svg + +digraph plugin_discovery { + rankdir=TB + bgcolor="transparent" + pad=0.2 + nodesep=0.4 + ranksep=0.5 + + node [ + shape=box, + style="rounded,filled", + fillcolor="#eef2f7", + color="#4a5b6e", + fontname="Helvetica", + fontsize=11, + margin="0.18,0.10" + ] + edge [ + fontname="Helvetica", + fontsize=9, + color="#4a5b6e", + fontcolor="#4a5b6e" + ] + + startup [label="Gramps startup"] + reg [label="reg_plugins(plugin_dir)\nrecursive scan\n(follows symlinks since 6.1)"] + gpr [label="Load each *.gpr.py\n(top-level register() calls execute)"] + catalog [label="Plugin catalog\nname, id, kind, target version\n(implementation module NOT loaded yet)"] + invoke [label="User invokes the addon\n(menu, sidebar, restart, ...)"] + load [label="Load fname module\nInstantiate the registered class"] + + startup -> reg + reg -> gpr [label=" for each\n addon folder"] + gpr -> catalog [label=" register(...)"] + catalog -> invoke [style=dashed, label=" later,\n on demand", constraint=false] + invoke -> load +} diff --git a/docs/addon-development/_media/plugin-discovery.svg b/docs/addon-development/_media/plugin-discovery.svg new file mode 100644 index 000000000..185d4468c --- /dev/null +++ b/docs/addon-development/_media/plugin-discovery.svg @@ -0,0 +1,90 @@ + + + + + + +plugin_discovery + + +startup + +Gramps startup + + + +reg + +reg_plugins(plugin_dir) +recursive scan +(follows symlinks since 6.1) + + + +startup->reg + + + + + +gpr + +Load each *.gpr.py +(top-level register() calls execute) + + + +reg->gpr + + +  for each +  addon folder + + + +catalog + +Plugin catalog +name, id, kind, target version +(implementation module NOT loaded yet) + + + +gpr->catalog + + +  register(...) + + + +invoke + +User invokes the addon +(menu, sidebar, restart, ...) + + + +catalog->invoke + + +  later, +  on demand + + + +load + +Load fname module +Instantiate the registered class + + + +invoke->load + + + + +