plans + specs

docs/superpowers/specs/2026-08-15-mywhoosh-garmin-sync-design.md

@@ -0,0 +1,620 @@
+# MyWhoosh -> Garmin Sync Service — Design
+
+Date: 2026-08-15
+Status: Draft for user review
+
+## 1. Goal
+
+Build a self-hosted Docker service that periodically checks MyWhoosh for new cycling activities for multiple configured users, downloads each new FIT file, rewrites the device metadata to a Garmin Edge 1030 Plus, and imports the activity into Garmin Connect without forwarding it to Strava.
+
+The service is administered through a local-only web interface protected by a single admin password.
+
+## 2. Scope
+
+### In scope for v1
+
+- Multiple independent sync users.
+- Local-only admin UI.
+- Admin password supplied via environment variable.
+- SQLite persistence.
+- Encrypted MyWhoosh and Garmin credentials at rest.
+- Persistent per-user MyWhoosh and Garmin token stores.
+- Periodic background sync at a configurable interval.
+- Manual "sync now" actions.
+- MyWhoosh activity discovery and FIT download.
+- Binary FIT metadata patching to Garmin Edge 1030 Plus.
+- FIT CRC validation and repair.
+- Garmin Connect import via `import_activity()`.
+- No intentional forwarding to Strava.
+- Per-user error isolation, status and retry handling.
+- Garmin MFA handling through the admin UI when required.
+- Activity history and sync-run history.
+
+### Out of scope for v1
+
+- Public Internet exposure.
+- Multi-admin or per-user web logins.
+- OAuth/OIDC for the admin UI.
+- Editing deployment configuration from the web UI.
+- Strava integration.
+- Mobile app.
+- Distributed workers or external queues.
+- Automatic CAPTCHA solving or browser automation for MyWhoosh login.
+
+## 3. High-level architecture
+
+The application runs as one Docker container with a persistent `/data` volume.
+
+Components:
+
+1. FastAPI application.
+2. Server-rendered Jinja2 admin UI with HTMX for small interactive actions.
+3. SQLite database.
+4. Scheduler.
+5. Sync manager.
+6. Per-user MyWhoosh client.
+7. FIT rewriter.
+8. Per-user Garmin client/uploader.
+9. Credential encryption service.
+
+Data flow:
+
+MyWhoosh -> download FIT -> validate FIT -> patch device metadata -> rewrite CRC -> Garmin `import_activity()` -> persist result.
+
+## 4. Deployment configuration
+
+Configuration is supplied through environment variables, for example:
+
+- `ADMIN_PASSWORD`
+- `SECRET_KEY`
+- `CREDENTIAL_ENCRYPTION_KEY`
+- `SYNC_INTERVAL_MINUTES`
+- `DATABASE_URL=sqlite:////data/app.db`
+- `DATA_DIR=/data`
+
+The web server binds inside the container and is published only to the trusted local network by Docker configuration.
+
+The web UI does not modify these deployment-level settings.
+
+## 5. Admin authentication
+
+The application has one admin login with no username.
+
+- The password is read from `ADMIN_PASSWORD`.
+- The password is never persisted in SQLite.
+- Successful login establishes a signed session using `SECRET_KEY`.
+- Authentication failures reveal no account details.
+- Session cookies should be `HttpOnly` and `SameSite=Lax`.
+- If TLS is later placed in front of the service, `Secure` should be enabled for the cookie.
+
+Because the service is intended for LAN-only use, v1 does not introduce a separate identity provider.
+
+## 6. User model
+
+Each sync user is independent.
+
+A user contains:
+
+- id
+- display name
+- enabled flag
+- health state
+- encrypted MyWhoosh email/password
+- encrypted Garmin email/password
+- created/updated timestamps
+
+Credentials are encrypted before being written to SQLite. The encryption key comes exclusively from `CREDENTIAL_ENCRYPTION_KEY`.
+
+Stored passwords are never returned to the browser. When editing a user, an empty password field means "keep the existing password".
+
+## 7. Token storage
+
+Authentication/session tokens are separated per user.
+
+Suggested filesystem layout:
+
+```text
+/data/
+  app.db
+  tokens/
+    <user-id>/
+      mywhoosh.json
+      garmin/
+  activities/
+    <user-id>/
+```
+
+The Garmin tokenstore mechanism from `python-garminconnect` should be reused rather than reimplemented.
+
+The MyWhoosh token cache stores the access token and, where usable, the refresh token and associated account metadata.
+
+No user may read or reuse another user's tokenstore.
+
+## 8. MyWhoosh authentication
+
+The service should follow the same direct API-login pattern used by `jdelrue/mywhoosh2garmin` rather than automating the MyWhoosh web login page.
+
+The intended flow is:
+
+1. Load cached per-user token.
+2. Attempt an authenticated activities request.
+3. If accepted, continue.
+4. If unauthorized, attempt API login using the configured MyWhoosh credentials and the Android-style login payload used by the reference project.
+5. Persist fresh token data.
+6. Retry the operation once.
+
+The implementation must not attempt to defeat or automate CAPTCHA/reCAPTCHA challenges.
+
+The MyWhoosh endpoints are not treated as a stable public API. Changes in these endpoints should surface as a clear `action_required`/authentication or integration failure rather than causing uncontrolled retries.
+
+## 9. MyWhoosh activity discovery
+
+For each enabled user, the MyWhoosh client retrieves recent activities using the authenticated bearer token.
+
+Each MyWhoosh activity must have a stable external activity identifier. The pair `(user_id, mywhoosh_activity_id)` is unique in SQLite.
+
+This makes discovery idempotent: the same activity may be returned on every scheduler run but is processed only once unless it previously failed at a retryable stage.
+
+The client is responsible only for:
+
+- authentication,
+- listing activities,
+- normalizing metadata,
+- downloading the original FIT file.
+
+It has no knowledge of Garmin or FIT rewriting.
+
+## 10. FIT rewriting
+
+The FIT rewriter follows the binary-patching approach from the existing working Python implementation rather than fully decoding and re-encoding the activity.
+
+### 10.1 Device identity
+
+Target device:
+
+- manufacturer: Garmin (`1`)
+- product: Edge 1030 Plus (`3570`)
+- product name: `Edge 1030 Plus`
+- serial number: optional; if absent, the original serial field is left unchanged unless a later compatibility requirement proves otherwise
+
+### 10.2 Patched messages
+
+`file_id` fields when present:
+
+- manufacturer
+- product
+- optional serial number
+- product name
+
+`device_info` fields are patched only for the creator device (`device_index == 0`) when a usable device index is present.
+
+Other sensor/device records should remain unchanged so a trainer, HR sensor or power meter does not become an Edge 1030 Plus accidentally.
+
+If the source FIT lacks enough information to identify creator-specific `device_info` safely, `file_id` remains mandatory and `device_info` patching should be conservative rather than rewriting all device messages.
+
+### 10.3 Binary preservation
+
+The rewriter must preserve all bytes not deliberately changed, except FIT CRC fields.
+
+It must support:
+
+- 12-byte and 14-byte FIT headers,
+- little- and big-endian definition architectures,
+- compressed timestamp records,
+- developer fields,
+- changing local message definitions.
+
+### 10.4 Validation
+
+Before patching:
+
+- validate `.FIT` signature,
+- validate declared length,
+- validate header CRC when present,
+- validate file CRC.
+
+After patching:
+
+- rewrite header CRC when present,
+- rewrite file CRC,
+- validate the output again,
+- verify expected target metadata is readable.
+
+Invalid FIT input is a non-retryable activity error unless the original file is later replaced/redownloaded.
+
+## 11. Garmin import
+
+The existing Garmin uploader pattern is reused with `python-garminconnect`.
+
+The final activity is sent using `import_activity()` rather than `upload_activity()` because the desired behavior is to import into Garmin Connect without intentional onward synchronization to Strava.
+
+Per user:
+
+1. Reuse Garmin tokenstore where possible.
+2. Login/refresh when required.
+3. Call `import_activity()` with the converted FIT path.
+4. Record the returned Garmin activity/import identifier if available.
+5. Treat known duplicate responses as completed `duplicate`, not as fatal failures.
+
+## 12. Garmin MFA
+
+MFA is modeled as an explicit user state.
+
+If Garmin requires MFA and there is no one-time code available:
+
+- the user's health becomes `action_required`,
+- that user's Garmin import attempts pause,
+- other users continue syncing normally,
+- the dashboard shows that MFA is required.
+
+The admin can submit the one-time MFA code through the local UI.
+
+The code:
+
+- is used only for that login attempt,
+- is never written to SQLite,
+- is never written to logs,
+- is discarded immediately after use.
+
+On successful authentication the Garmin tokenstore is persisted and the user returns to normal sync behavior.
+
+## 13. Activity state machine
+
+An activity progresses through durable stages:
+
+- `discovered`
+- `downloaded`
+- `converted`
+- `imported`
+- `duplicate`
+- `failed`
+
+Persisted activity fields include:
+
+- internal id
+- user id
+- MyWhoosh activity id
+- activity date/time
+- activity name
+- original FIT path
+- converted FIT path
+- current status
+- Garmin activity/import id when known
+- last error
+- created/updated timestamps
+
+Completed terminal states are `imported` and `duplicate`.
+
+A failure must retain the latest successfully completed stage so a retry can resume without repeating unnecessary work.
+
+## 14. Sync-run model
+
+Each user sync invocation creates a sync-run record containing:
+
+- id
+- user id
+- start time
+- finish time
+- status (`running`, `success`, `partial`, `failed`)
+- discovered count
+- imported count
+- skipped count
+- failed count
+- summary error when relevant
+
+Detailed application logs remain on stdout; SQLite stores only UI-relevant summaries.
+
+## 15. Scheduler and concurrency
+
+A central scheduler triggers every `SYNC_INTERVAL_MINUTES`.
+
+On each tick:
+
+1. Load enabled users.
+2. Schedule one sync job per user.
+3. Allow different users to run concurrently.
+4. Enforce at most one active sync per user with a per-user lock.
+
+Manual "sync now" uses exactly the same sync pipeline and the same lock.
+
+If a manual request arrives while that user is already syncing, the application should return a clear "already running" result rather than start another run.
+
+The scheduler must not block because one account is slow, broken or waiting for user action.
+
+## 16. Retry policy
+
+Retries depend on failure type.
+
+### Retryable automatically
+
+- transient network errors
+- timeouts
+- temporary MyWhoosh/Garmin server errors
+- expired session/token after one reauthentication attempt
+
+Within one sync run, use at most a small bounded retry (for example one retry). Further retry occurs on the next scheduler tick.
+
+### Action required
+
+- invalid MyWhoosh credentials
+- MyWhoosh login/API behavior changed in a way that prevents authentication
+- Garmin MFA required
+- invalid Garmin credentials
+
+### Non-retryable per activity
+
+- corrupt/invalid FIT file
+- unsupported FIT structure that cannot be safely patched
+
+The admin UI can expose an explicit "retry" action for failed activities after the underlying issue is fixed.
+
+## 17. User health state
+
+Each user has a concise operational state:
+
+- `healthy`
+- `syncing`
+- `degraded`
+- `action_required`
+- `disabled`
+
+This state is derived from configuration and recent sync/authentication outcomes and is shown prominently on the dashboard.
+
+## 18. Admin UI
+
+### 18.1 Login
+
+Single password field and submit action.
+
+### 18.2 Dashboard
+
+Shows all users with:
+
+- name
+- health state
+- MyWhoosh connection state
+- Garmin connection state
+- last sync
+- last imported activity
+- primary error/action if any
+- "sync now"
+- "details"
+- MFA action when needed
+
+Includes "add account".
+
+### 18.3 User create/edit
+
+Fields:
+
+- display name
+- MyWhoosh email
+- MyWhoosh password
+- Garmin email
+- Garmin password
+- enabled flag
+
+Actions:
+
+- save
+- test connection
+
+Existing passwords are never rendered back to the browser.
+
+### 18.4 User details
+
+Shows:
+
+- current connection and health states
+- most recent sync-run summary
+- recent activities and status
+- latest errors
+
+Actions:
+
+- sync now
+- retry failed activity
+- enter Garmin MFA when required
+
+### 18.5 System page
+
+Read-only operational information:
+
+- application version
+- configured sync interval
+- last scheduler tick
+- next expected tick
+- account count
+- activity count
+
+Action:
+
+- sync all now
+
+## 19. UI technology
+
+Use:
+
+- FastAPI
+- Jinja2
+- HTMX
+- small application-specific CSS
+
+Do not introduce Angular, React, Tailwind or Bootstrap for v1 unless requirements change.
+
+HTMX is used for bounded actions such as:
+
+- sync now
+- test connection
+- submit MFA
+- retry activity
+
+The application remains server-rendered and easy to operate as one container.
+
+## 20. Security requirements
+
+- Never log passwords, bearer tokens, session tokens, encryption keys or MFA codes.
+- Encrypt stored MyWhoosh and Garmin credentials.
+- Keep tokenstores under the persistent data directory with restrictive filesystem permissions where possible.
+- Escape all user-visible data rendered into HTML.
+- Protect state-changing web requests against CSRF.
+- Validate all IDs against the authenticated admin session rather than trusting client-provided paths blindly.
+- Use prepared/ORM parameterized database access.
+- Do not expose decrypted credentials through API responses or templates.
+
+## 21. Cleanup and retention
+
+For v1, original and converted FIT files are retained because they are valuable for debugging failed imports.
+
+Automated retention/cleanup can be added later after operating behavior is known.
+
+## 22. Error isolation
+
+Failure of one user must never prevent other users from syncing.
+
+Examples:
+
+- User A imports successfully while User B requires Garmin MFA.
+- User C may have invalid MyWhoosh credentials without affecting scheduler execution for A or B.
+- A corrupt activity file affects only that activity and user.
+
+## 23. Testing strategy
+
+### FIT rewriter tests
+
+- valid 12-byte header FIT
+- valid 14-byte header FIT
+- invalid header CRC
+- invalid file CRC
+- malformed/truncated definitions
+- developer fields preserved
+- compressed timestamp records handled
+- Edge 1030 Plus manufacturer/product patched correctly
+- non-creator `device_info` unchanged
+- output CRC valid
+- bytes outside expected metadata and CRC locations unchanged
+
+### MyWhoosh client tests
+
+Use mocked HTTP responses for:
+
+- valid cached token
+- expired token followed by successful login
+- invalid credentials
+- transient server error
+- activity listing
+- FIT download
+
+Do not make live MyWhoosh requests in the normal unit test suite.
+
+### Garmin uploader tests
+
+Use a fake/protocol-compatible Garmin client for:
+
+- tokenstore login
+- successful import
+- duplicate
+- MFA required
+- invalid login
+- transient import failure
+
+### Sync manager tests
+
+- new activity full happy path
+- discovered activity is not duplicated
+- resume from downloaded
+- resume from converted
+- retry after transient Garmin failure
+- one user's failure does not affect another
+- per-user lock prevents concurrent duplicate sync
+
+### Web tests
+
+- admin login success/failure
+- unauthenticated routes redirect/reject
+- create/edit/disable user
+- password never returned
+- manual sync action
+- MFA submission lifecycle
+- CSRF on mutating requests
+
+## 24. Suggested module boundaries
+
+```text
+app/
+  main.py
+  config.py
+
+  auth/
+    admin.py
+
+  db/
+    models.py
+    session.py
+    repositories.py
+
+  security/
+    credentials.py
+
+  mywhoosh/
+    client.py
+    models.py
+    tokenstore.py
+
+  fit/
+    rewriter.py
+    crc.py
+    models.py
+
+  garmin/
+    uploader.py
+
+  sync/
+    manager.py
+    scheduler.py
+    states.py
+
+  web/
+    routes.py
+    forms.py
+    templates/
+    static/
+
+tests/
+```
+
+Each unit should depend on explicit interfaces/protocols where external services are involved so tests do not require live accounts.
+
+## 25. Primary design decisions
+
+1. One local admin instead of user-facing authentication.
+2. Multiple independent sync accounts.
+3. SQLite for durable application state.
+4. Environment variables for deployment secrets/configuration.
+5. Encrypted service credentials at rest.
+6. Per-user tokenstores.
+7. Direct MyWhoosh API login pattern; no CAPTCHA automation.
+8. Binary FIT patching rather than decode/re-encode.
+9. Garmin Edge 1030 Plus product ID `3570`.
+10. Conservative creator-device patching.
+11. Garmin `import_activity()` for Garmin-only import behavior.
+12. FastAPI + Jinja2 + HTMX for a small single-container admin UI.
+13. Parallel sync across users, serialized sync within each user.
+14. Durable activity stages for resumable/idempotent sync.
+
+## 26. Acceptance criteria for v1
+
+The system is ready for v1 when:
+
+1. It runs from Docker with persistent `/data` storage.
+2. The admin can log in locally using the environment-configured password.
+3. The admin can add at least two independent users.
+4. Each user can authenticate independently to MyWhoosh and Garmin.
+5. New MyWhoosh activities are discovered automatically on schedule.
+6. FIT files are downloaded and patched to Garmin Edge 1030 Plus metadata with valid CRCs.
+7. Converted activities are imported into the corresponding Garmin Connect account using `import_activity()`.
+8. Already processed activities are not imported again.
+9. Garmin MFA for one user can be resolved through the UI and does not block other users.
+10. A failure for one user or one activity does not stop the scheduler.
+11. The dashboard shows current state, recent syncs and actionable errors.
+12. Secrets and MFA codes do not appear in logs or browser responses.