Koreader plugin fixes

.planning/PROJECT.md

@@ -0,0 +1,106 @@
+# KOmpanion - Highlights Sync
+
+## What This Is
+
+A feature to synchronize book highlights and notes from KOReader to Kompanion.
+Users can highlight text and add notes while reading in KOReader, and these annotations are automatically synced to Kompanion for storage and viewing.
+
+## Core Value
+
+Users can capture and review their book highlights in one place - Kompanion serves as the central repository for all reading annotations.
+
+## Requirements
+
+### Validated (Pre-existing)
+
+- ✓ Book library management — existing
+- ✓ KOReader progress sync — existing
+- ✓ WebDAV statistics sync — existing
+- ✓ OPDS catalog — existing
+- ✓ User authentication — existing
+- ✓ Device management — existing
+
+### Validated (Phase 1: API & Storage — 2026-03-21)
+
+- ✓ KOReader highlights sync via HTTP API — POST /syncs/highlights
+- ✓ Store highlights in PostgreSQL database — highlight_annotations table
+- ✓ Deduplication via content hash — ON CONFLICT DO NOTHING
+- ✓ Device authentication — MD5 hash, matches progress sync pattern
+
+### Validated (Phase 2: Web UI — 2026-03-21)
+
+- ✓ Display highlights on book detail page — read-only section with text, note, page/chapter
+- ✓ Highlight dependency wired through router to books handler
+- ✓ CSS styling matching existing design system
+
+### Validated (Phase 5: Standalone KOReader Plugin — 2026-03-22)
+
+- ✓ Standalone KOReader plugin — `koreader/kompanion.koplugin/`
+- ✓ WidgetContainer base class — Tools menu integration (not Export submenu)
+- ✓ Setup dialog for URL/device credentials — persisted in G_reader_settings
+- ✓ Dual format highlight extraction — annotations (new) + highlight/bookmarks (legacy)
+- ✓ HTTP sync with Basic Auth — POST /syncs/highlights
+- ✓ Success/error toasts — synced count or failure message
+
+### Active
+
+- None — all milestone requirements complete
+
+### Out of Scope
+
+- Two-way sync (Kompanion → KOReader) — deferred, KOReader exporter plugin doesn't support it
+- Highlight editing in web UI — read-only display for now
+- Highlight export from web UI — use KOReader's own export instead
+- Image highlights (text only) — complexity, deferred
+
+## Context
+
+### Existing System
+
+KOmpanion already has:
+- Progress sync at `/syncs/progress` using MD5-hashed device credentials
+- Statistics sync via WebDAV at `/webdav/statistics.sqlite3`
+- Book storage with PostgreSQL backend
+- Clean architecture with layered separation (entity, service, repository, controller)
+
+### KOReader Exporter Plugin
+
+KOReader has a built-in exporter plugin (`plugins/exporter.koplugin/`) that can export highlights to various targets:
+- JSON file export
+- Readwise API
+- Joplin notes
+- Nextcloud
+- Markdown/text
+
+The plugin uses a base exporter class with `makeJsonRequest` method for HTTP APIs.
+Highlights are stored in document sidecar files and parsed via `MyClipping` class.
+
+### Data Structure
+
+From KOReader, each highlight has:
+- `text`: the highlighted text
+- `note`: optional user note
+- `page`: page number or location
+- `chapter`: optional chapter title
+- `time`: Unix timestamp
+- `drawer`: highlight style
+- `color`: highlight color
+
+## Constraints
+
+- **Protocol**: Must use HTTP POST to match existing `/syncs/progress` pattern
+- **Authentication**: Use existing device credential system (MD5 hash)
+- **Database**: Store in PostgreSQL alongside existing data
+- **UI**: Integrate into existing book detail page template
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| One-way sync only | KOReader exporter doesn't support fetching, user confirmed | ✓ Implemented |
+| HTTP API over WebDAV | Simpler implementation, matches progress sync pattern | ✓ Implemented |
+| Read-only UI | Editing adds complexity, can be added later | ✓ Implemented |
+| WidgetContainer over Provider | Provider system unreliable, WidgetContainer stable like kosync | ✓ Implemented (Phase 5) |
+
+---
+*Last updated: 2026-03-22 after Phase 5 completion*

.planning/REQUIREMENTS.md

@@ -0,0 +1,135 @@
+# Requirements: KOmpanion Highlights Sync
+
+**Defined:** 2026-03-21
+**Core Value:** Users can capture and review their book highlights in one place
+
+## v1 Requirements
+
+Requirements for highlights sync feature. Each maps to roadmap phases.
+
+### API
+
+- [x] **API-01**: KOReader can sync highlights via HTTP POST to `/syncs/highlights`
+- [x] **API-02**: API accepts array of highlights in single request
+- [x] **API-03**: API uses device authentication (MD5 hash, existing pattern)
+- [x] **API-04**: API returns synced count and total count
+
+### Data Storage
+
+- [x] **DATA-01**: Highlights stored in PostgreSQL `highlight_annotations` table
+- [x] **DATA-02**: Highlight text is stored (required)
+- [x] **DATA-03**: User note is stored (optional)
+- [x] **DATA-04**: Page/location is stored
+- [x] **DATA-05**: Chapter is stored (optional)
+- [x] **DATA-06**: Timestamp from KOReader is stored
+- [x] **DATA-07**: Highlight style (drawer) and color are stored
+- [x] **DATA-08**: Device name is stored
+- [x] **DATA-09**: Document MD5 hash is stored for book matching
+- [x] **DATA-10**: Content hash for deduplication is stored
+
+### Sync Behavior
+
+- [x] **SYNC-01**: Re-syncing same highlights does not create duplicates
+- [x] **SYNC-02**: Highlights for books not in library are stored (orphan handling)
+- [x] **SYNC-03**: Both KOReader data models supported (annotations + legacy)
+
+### Web UI
+
+- [x] **UI-01**: Highlights displayed on book detail page
+- [x] **UI-02**: Highlights shown with text, page, chapter (when available)
+- [x] **UI-03**: User notes displayed alongside highlight text
+- [x] **UI-04**: Highlights ordered chronologically or by page
+- [x] **UI-05**: Read-only display (no editing in web UI)
+
+### Nextcloud Notes API
+
+- [x] **NC-01**: GET /notes returns notes filtered by authenticated device
+- [x] **NC-02**: POST /notes creates note with highlights formatted as markdown
+- [x] **NC-03**: PUT /notes/{id} updates existing note by document hash
+- [x] **NC-04**: One note per book containing all highlights
+- [x] **NC-05**: Basic Auth with device credentials
+- [x] **NC-06**: CRC32 IEEE hash for stable integer IDs
+
+### KOReader Lua Plugin
+
+- [x] **LUA-01**: Plugin appears in KOReader Export highlights menu
+- [x] **LUA-02**: User can configure server URL and device credentials via Setup dialog
+- [x] **LUA-03**: Export sends highlights to Kompanion /syncs/highlights endpoint with Basic Auth
+- [x] **LUA-04**: Success/failure shows as toast notification in KOReader
+
+## v2 Requirements
+
+Deferred to future release.
+
+### Export
+
+- **EXPR-01**: Export highlights to JSON
+- **EXPR-02**: Bulk export all highlights
+
+### Enhancement
+
+- **ENH-01**: Filter highlights by color/style
+- **ENH-02**: Highlight count badge on book cards
+- **ENH-03**: Dedicated highlights page across all books
+
+## Out of Scope
+
+Explicitly excluded. Documented to prevent scope creep.
+
+| Feature | Reason |
+|---------|--------|
+| Two-way sync (Kompanion → KOReader) | KOReader exporter plugin is push-only |
+| Highlight editing in web UI | Read-only for MVP, edit in KOReader |
+| Image highlights | Complexity, deferred per PROJECT.md |
+| Real-time sync | Over-engineering for reading use case |
+| Highlight sharing | Out of scope for self-hosted tool |
+| Auto-sync on book close | Deferred - could be future enhancement |
+| Plugin auto-update | KOReader doesn't support this natively |
+
+## Traceability
+
+Which phases cover which requirements. Updated during roadmap creation.
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| API-01 | Phase 1 | Complete |
+| API-02 | Phase 1 | Complete |
+| API-03 | Phase 1 | Complete |
+| API-04 | Phase 1 | Complete |
+| DATA-01 | Phase 1 | Complete |
+| DATA-02 | Phase 1 | Complete |
+| DATA-03 | Phase 1 | Complete |
+| DATA-04 | Phase 1 | Complete |
+| DATA-05 | Phase 1 | Complete |
+| DATA-06 | Phase 1 | Complete |
+| DATA-07 | Phase 1 | Complete |
+| DATA-08 | Phase 1 | Complete |
+| DATA-09 | Phase 1 | Complete |
+| DATA-10 | Phase 1 | Complete |
+| SYNC-01 | Phase 1 | Complete |
+| SYNC-02 | Phase 1 | Complete |
+| SYNC-03 | Phase 1 | Complete |
+| UI-01 | Phase 2 | Complete |
+| UI-02 | Phase 2 | Complete |
+| UI-03 | Phase 2 | Complete |
+| UI-04 | Phase 2 | Complete |
+| UI-05 | Phase 2 | Complete |
+| NC-01 | Phase 3 | Complete |
+| NC-02 | Phase 3 | Complete |
+| NC-03 | Phase 3 | Complete |
+| NC-04 | Phase 3 | Complete |
+| NC-05 | Phase 3 | Complete |
+| NC-06 | Phase 3 | Complete |
+| LUA-01 | Phase 4 | Not started |
+| LUA-02 | Phase 4 | Not started |
+| LUA-03 | Phase 4 | Not started |
+| LUA-04 | Phase 4 | Not started |
+
+**Coverage:**
+- v1 requirements: 31 total
+- Mapped to phases: 31
+- Unmapped: 0
+
+---
+*Requirements defined: 2026-03-21*
+*Last updated: 2026-03-22 after Phase 4 planning*

.planning/ROADMAP.md

@@ -0,0 +1,147 @@
+# Roadmap: KOmpanion Highlights Sync
+
+**Project:** KOmpanion - Highlights Sync
+**Core Value:** Users can capture and review their book highlights in one place
+**Created:** 2026-03-21
+
+---
+
+## Overview
+
+This roadmap delivers the highlights sync feature in 5 phases. Each phase delivers a complete, verifiable capability.
+
+**Total v1 Requirements:** 31 (including Phase 4 Lua plugin)
+**Granularity:** Coarse
+
+---
+
+## Phases
+
+- [x] **Phase 1: API & Storage** - KOReader can sync highlights via HTTP API, stored in PostgreSQL
+- [x] **Phase 2: Web UI** - Users can view their synced highlights on the book detail page
+- [x] **Phase 3: Nextcloud Notes API** - KOReader Nextcloud Notes exporter compatibility
+- [~] **Phase 4: KOReader Lua Plugin** - DEPRECATED: Provider-based approach proved unreliable
+- [x] **Phase 5: Standalone KOReader Plugin** - WidgetContainer-based plugin with native highlight extraction
+
+---
+
+## Phase Details
+
+### Phase 1: API & Storage
+**Goal:** KOReader devices can sync highlights to Kompanion via HTTP API, with persistent storage
+**Depends on:** Nothing (first phase)
+**Requirements:** API-01, API-02, API-03, API-04, DATA-01, DATA-02, DATA-03, DATA-04, DATA-05, DATA-06, DATA-07, DATA-08, DATA-09, DATA-10, SYNC-01, SYNC-02, SYNC-03
+**Success Criteria** (what must be TRUE):
+  1. KOReader can POST highlights to `/syncs/highlights` and receive synced count
+  2. Device authentication works (MD5 hash, matches existing progress sync pattern)
+  3. Highlights are stored in PostgreSQL with all metadata (text, note, page, chapter, timestamp, drawer, color, device, document hash)
+  4. Re-syncing same highlights does not create duplicates (idempotent via content hash)
+  5. Highlights for unknown books are stored without errors (orphan handling)
+
+Plans:
+- [x] 01-01-PLAN.md - Create Highlight entity and database migration
+- [x] 01-02-PLAN.md - Create highlight package interfaces
+- [x] 01-03-PLAN.md - Implement use case and PostgreSQL repository
+- [x] 01-04-PLAN.md - Implement HTTP handler and wire dependencies
+
+### Phase 2: Web UI
+**Goal:** Users can view their synced highlights on the book detail page
+**Depends on:** Phase 1 (API & Storage)
+**Requirements:** UI-01, UI-02, UI-03, UI-04, UI-05
+**Success Criteria** (what must be TRUE):
+  1. Book detail page displays all highlights for that book
+  2. Each highlight shows text, page, and chapter (when available)
+  3. User notes appear alongside highlight text
+  4. Highlights are ordered chronologically or by page number
+  5. Display is read-only (no editing controls shown)
+**Plans:** 3 plans
+
+Plans:
+- [x] 02-01-PLAN.md - Wire highlight dependency through router and fetch highlights in viewBook handler
+- [x] 02-02-PLAN.md - Add highlights template section to book.html
+- [x] 02-03-PLAN.md - Add CSS styling for highlights
+
+### Phase 3: Nextcloud Notes API Endpoint
+**Goal:** Implement Nextcloud Notes API-compatible endpoint for KOReader exporter (replaces XMNote due to auth security)
+**Depends on:** Phase 2
+**Requirements:** NC-01, NC-02, NC-03, NC-04, NC-05, NC-06
+
+**Success Criteria** (what must be TRUE):
+  1. KOReader can connect using Nextcloud Notes exporter with Basic Auth
+  2. GET /notes returns notes filtered by authenticated device
+  3. POST /notes creates note with highlights formatted as markdown
+  4. PUT /notes/{id} updates existing note by document hash
+  5. One note per book containing all highlights
+
+**Plans:** 2 plans in 2 waves
+
+Plans:
+- [x] 03-01-PLAN.md - Create notes package with markdown formatter (Wave 1)
+- [x] 03-02-PLAN.md - Implement Notes API handlers and wire routes (Wave 2, depends on 03-01)
+
+### Phase 4: KOReader Lua Plugin for Highlights Sync
+**Goal:** Create a KOReader Lua plugin that exports highlights to Kompanion's existing `/syncs/highlights` API. The built-in exporter plugin is marked deprecated, so this provides a dedicated integration path.
+**Depends on:** Phase 3
+**Requirements:** LUA-01, LUA-02, LUA-03, LUA-04
+
+**Success Criteria** (what must be TRUE):
+  1. KOReader user sees Kompanion option in Export highlights menu
+  2. User can configure server URL and device credentials via Setup dialog
+  3. Export sends highlights to Kompanion /syncs/highlights endpoint
+  4. Success/failure shows as toast notification in KOReader
+
+**Plans:** 1 plan in 1 wave
+
+Plans:
+- [x] 04-01-PLAN.md - Create KOReader Lua plugin with Provider registration (Wave 1)
+
+### Phase 5: Standalone KOReader Plugin with Native Highlight Extraction
+**Goal:** Create a standalone KOReader Lua plugin that directly extracts highlights from document sidecar files and syncs to Kompanion. Does NOT depend on the exporter.koplugin framework - reads DocSettings directly using clip.lua patterns.
+**Depends on:** Phase 4 (replaces unreliable Provider-based approach)
+**Requirements:** D-01 through D-16 (implementation decisions from CONTEXT.md)
+
+**Success Criteria** (what must be TRUE):
+  1. Plugin appears in KOReader Tools menu (not Export highlights submenu)
+  2. User can configure server URL, device name, and password via Setup dialog
+  3. Sync highlights menu item sends highlights to Kompanion /syncs/highlights endpoint
+  4. Plugin extracts highlights from both new `annotations` and legacy `highlight`+`bookmarks` formats
+  5. Success/failure shows as toast notification with synced count
+
+**Plans:** 1 plan in 1 wave
+
+Plans:
+- [x] 05-01-PLAN.md - Create WidgetContainer-based KOReader plugin with native highlight extraction (Wave 1)
+
+---
+
+## Progress
+
+| Phase | Plans Complete | Status | Completed |
+|-------|-----------------|--------|-----------|
+| 1. API & Storage | 4/4 | Complete | 2026-03-21 |
+| 2. Web UI | 3/3 | Complete | 2026-03-21 |
+| 3. Nextcloud Notes API | 2/2 | Complete | 2026-03-21 |
+| 4. KOReader Lua Plugin | N/A | Deprecated | - |
+| 5. Standalone KOReader Plugin | 1/1 | Complete | 2026-03-22 |
+
+---
+
+## Coverage Map
+
+| Category | Requirements | Phase |
+|----------|--------------|-------|
+| API | API-01, API-02, API-03, API-04 | 1 |
+| Data Storage | DATA-01 through DATA-10 | 1 |
+| Sync Behavior | SYNC-01, SYNC-02, SYNC-03 | 1 |
+| Web UI | UI-01 through UI-05 | 2 |
+| Notes API | NC-01, NC-02, NC-03, NC-04, NC-05, NC-06 | 3 |
+| Lua Plugin | LUA-01, LUA-02, LUA-03, LUA-04 | 4 |
+| Standalone Plugin | D-01 through D-16 | 5 |
+
+**Total:** 31 requirements mapped to 4 phases + Phase 5 implementation decisions
+**Orphaned:** 0
+
+---
+
+*Roadmap created: 2026-03-21*
+*Last updated: 2026-03-22 after Phase 5 planning*