plans + specs

This commit is contained in:
Bastian Wagner
2026-08-15 09:00:41 +02:00
commit f6da346e18
5 changed files with 3804 additions and 0 deletions

View File

@@ -0,0 +1,577 @@
# FIT Edge 1030 Plus Rewriter Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement a binary-preserving FIT metadata rewriter that changes only Garmin creator-device metadata plus required CRC bytes, targeting Garmin Edge 1030 Plus product ID `3570`.
**Architecture:** Parse FIT definition/data records only far enough to locate `file_id` and creator `device_info` fields, patch bytes in place, recalculate header/file CRC, and validate the output. Do not fully decode/re-encode the activity, so unknown/developer fields and all non-target bytes remain untouched.
**Tech Stack:** Python 3.12 standard library (`struct`, `dataclasses`, `pathlib`), pytest.
## Global Constraints
- Garmin manufacturer ID is `1`.
- Garmin Edge 1030 Plus product ID is `3570`.
- Product name is `Edge 1030 Plus`.
- Support FIT header sizes 12 and 14.
- Validate declared data size and `.FIT` signature.
- Validate header CRC when a 14-byte header is present.
- Validate file CRC before and after modification.
- Support little- and big-endian definition architectures, compressed timestamp records, developer fields, and changing local message definitions.
- Patch `file_id` device fields when present.
- Patch `device_info` only when it is safely identified as the creator (`device_index == 0`).
- If creator-specific `device_info` cannot be identified safely, leave it unchanged rather than rewriting all device records.
- Preserve every non-target byte except CRC fields.
---
## File Structure
```text
app/fit/
__init__.py
crc.py
models.py
rewriter.py
tests/fit/
builders.py
test_crc.py
test_rewriter_validation.py
test_rewriter_patching.py
test_rewriter_preservation.py
```
## Task 1: Implement Garmin FIT CRC
**Files:**
- Create: `app/fit/crc.py`
- Create: `tests/fit/test_crc.py`
**Interfaces:**
- Produces: `fit_crc(data: bytes | bytearray | memoryview) -> int`.
- [ ] **Step 1: Write failing CRC vector tests**
```python
# tests/fit/test_crc.py
from app.fit.crc import fit_crc
def test_empty_crc_is_zero() -> None:
assert fit_crc(b"") == 0
def test_crc_is_incremental_equivalent() -> None:
payload = b".FIT-device-metadata"
assert fit_crc(payload) == fit_crc(memoryview(payload))
assert 0 <= fit_crc(payload) <= 0xFFFF
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/fit/test_crc.py -v`
Expected: import failure.
- [ ] **Step 3: Implement the FIT nibble-table CRC algorithm**
```python
# app/fit/crc.py
CRC_TABLE = (
0x0000, 0xCC01, 0xD801, 0x1400,
0xF001, 0x3C00, 0x2800, 0xE401,
0xA001, 0x6C00, 0x7800, 0xB401,
0x5000, 0x9C01, 0x8801, 0x4400,
)
def fit_crc(data: bytes | bytearray | memoryview) -> int:
crc = 0
for byte in data:
tmp = CRC_TABLE[crc & 0xF]
crc = (crc >> 4) & 0x0FFF
crc ^= tmp ^ CRC_TABLE[byte & 0xF]
tmp = CRC_TABLE[crc & 0xF]
crc = (crc >> 4) & 0x0FFF
crc ^= tmp ^ CRC_TABLE[(byte >> 4) & 0xF]
return crc & 0xFFFF
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/fit/test_crc.py -v`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add app/fit/crc.py tests/fit/test_crc.py
git commit -m "feat: add FIT CRC calculation"
```
## Task 2: Parse and validate the FIT container safely
**Files:**
- Create: `app/fit/models.py`
- Create: `app/fit/rewriter.py`
- Create: `tests/fit/builders.py`
- Create: `tests/fit/test_rewriter_validation.py`
**Interfaces:**
- Produces: `FitFormatError`, `FieldDefinition`, `LocalDefinition`, `_validate_fit_container()`, `_iter_data_fields()`.
- `_iter_data_fields(data)` returns data records with exact field offsets without decoding unrelated values.
- [ ] **Step 1: Create a deterministic minimal FIT fixture builder**
```python
# tests/fit/builders.py
import struct
from app.fit.crc import fit_crc
def make_fit(data_records: bytes, *, header_size: int = 14) -> bytes:
if header_size not in {12, 14}:
raise ValueError(header_size)
header = bytearray(header_size)
header[0] = header_size
header[1] = 0x20
struct.pack_into("<H", header, 2, 0x0100)
struct.pack_into("<I", header, 4, len(data_records))
header[8:12] = b".FIT"
if header_size == 14:
struct.pack_into("<H", header, 12, fit_crc(header[:12]))
body = header + data_records
return bytes(body + struct.pack("<H", fit_crc(body)))
def definition(local: int, global_num: int, fields: list[tuple[int, int, int]], *, endian: str = "<") -> bytes:
architecture = 1 if endian == ">" else 0
payload = bytearray([0, architecture])
payload.extend(struct.pack(f"{endian}H", global_num))
payload.append(len(fields))
for num, size, base_type in fields:
payload.extend(bytes([num, size, base_type]))
return bytes([0x40 | local]) + bytes(payload)
def data(local: int, payload: bytes) -> bytes:
return bytes([local]) + payload
```
- [ ] **Step 2: Write validation tests**
```python
# tests/fit/test_rewriter_validation.py
from pathlib import Path
import pytest
from app.fit.rewriter import FitFormatError, is_fit_file
from tests.fit.builders import make_fit
def test_valid_12_and_14_byte_headers(tmp_path: Path) -> None:
for size in (12, 14):
path = tmp_path / f"valid-{size}.fit"
path.write_bytes(make_fit(b"", header_size=size))
assert is_fit_file(path) is True
def test_bad_file_crc_is_rejected(tmp_path: Path) -> None:
payload = bytearray(make_fit(b""))
payload[-1] ^= 0xFF
path = tmp_path / "bad.fit"
path.write_bytes(payload)
assert is_fit_file(path) is False
```
- [ ] **Step 3: Run and verify failure**
Run: `pytest tests/fit/test_rewriter_validation.py -v`
Expected: import failure.
- [ ] **Step 4: Implement the parser structures and validation**
```python
# app/fit/models.py
from dataclasses import dataclass
@dataclass(frozen=True)
class FieldDefinition:
num: int
size: int
base_type: int
@dataclass(frozen=True)
class LocalDefinition:
global_message_num: int
endian: str
fields: tuple[FieldDefinition, ...]
developer_field_size: int
```
Implement in `app/fit/rewriter.py` the validated logic from the known working implementation:
```python
class FitFormatError(ValueError):
pass
def _validate_fit_container(data: bytearray) -> None:
if len(data) < 14:
raise FitFormatError("FIT file is too small")
header_size = data[0]
if header_size not in {12, 14}:
raise FitFormatError(f"Unsupported FIT header size: {header_size}")
if len(data) < header_size + 2:
raise FitFormatError("FIT file is shorter than its header")
if bytes(data[8:12]) != b".FIT":
raise FitFormatError("Missing .FIT signature")
data_size = struct.unpack_from("<I", data, 4)[0]
expected_size = header_size + data_size + 2
if len(data) != expected_size:
raise FitFormatError(f"FIT size mismatch: header says {expected_size} bytes, file has {len(data)}")
if header_size == 14:
expected_header_crc = struct.unpack_from("<H", data, 12)[0]
if expected_header_crc != fit_crc(data[:12]):
raise FitFormatError("FIT header CRC check failed")
expected_file_crc = struct.unpack_from("<H", data, len(data) - 2)[0]
if expected_file_crc != fit_crc(data[:-2]):
raise FitFormatError("FIT file CRC check failed")
```
Implement the parser helpers explicitly:
```python
def _read_definition(
data: bytearray,
offset: int,
has_developer_fields: bool,
end_offset: int,
) -> tuple[LocalDefinition, int]:
if offset + 5 > end_offset:
raise FitFormatError("Truncated FIT definition message")
offset += 1 # reserved byte
architecture = data[offset]
offset += 1
if architecture not in {0, 1}:
raise FitFormatError(f"Unsupported FIT architecture: {architecture}")
endian = ">" if architecture == 1 else "<"
global_message_num = struct.unpack_from(f"{endian}H", data, offset)[0]
offset += 2
field_count = data[offset]
offset += 1
fields: list[FieldDefinition] = []
for _ in range(field_count):
if offset + 3 > end_offset:
raise FitFormatError("Truncated FIT field definition")
fields.append(FieldDefinition(data[offset], data[offset + 1], data[offset + 2]))
offset += 3
developer_field_size = 0
if has_developer_fields:
if offset >= end_offset:
raise FitFormatError("Truncated FIT developer field count")
developer_count = data[offset]
offset += 1
for _ in range(developer_count):
if offset + 3 > end_offset:
raise FitFormatError("Truncated FIT developer field definition")
developer_field_size += data[offset + 1]
offset += 3
return LocalDefinition(global_message_num, endian, tuple(fields), developer_field_size), offset
def _collect_field_offsets(
definition: LocalDefinition,
offset: int,
end_offset: int,
) -> tuple[list[tuple[FieldDefinition, int]], int]:
result: list[tuple[FieldDefinition, int]] = []
current = offset
for field in definition.fields:
if current + field.size > end_offset:
raise FitFormatError("Truncated FIT data record")
result.append((field, current))
current += field.size
if current + definition.developer_field_size > end_offset:
raise FitFormatError("Truncated FIT developer field payload")
current += definition.developer_field_size
return result, current
def _iter_data_fields(
data: bytearray,
) -> list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]]:
header_size = data[0]
data_size = struct.unpack_from("<I", data, 4)[0]
offset = header_size
end_offset = header_size + data_size
definitions: dict[int, LocalDefinition] = {}
records: list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]] = []
while offset < end_offset:
record_header = data[offset]
offset += 1
if record_header & 0x80:
local = (record_header >> 5) & 0x03
definition = definitions.get(local)
if definition is None:
raise FitFormatError(f"Compressed timestamp record used unknown local definition {local}")
field_offsets, offset = _collect_field_offsets(definition, offset, end_offset)
records.append((definition, field_offsets))
continue
local = record_header & 0x0F
is_definition = bool(record_header & 0x40)
has_developer_fields = bool(record_header & 0x20)
if is_definition:
definition, offset = _read_definition(data, offset, has_developer_fields, end_offset)
definitions[local] = definition
continue
definition = definitions.get(local)
if definition is None:
raise FitFormatError(f"Data record used unknown local definition {local}")
field_offsets, offset = _collect_field_offsets(definition, offset, end_offset)
records.append((definition, field_offsets))
if offset != end_offset:
raise FitFormatError("FIT parser did not end on data boundary")
return records
```
- [ ] **Step 5: Expose `is_fit_file` and verify parser boundary failures**
```python
def is_fit_file(path: Path) -> bool:
try:
data = bytearray(path.read_bytes())
_validate_fit_container(data)
_iter_data_fields(data)
except (OSError, FitFormatError):
return False
return True
```
- [ ] **Step 6: Run validation tests**
Run: `pytest tests/fit/test_rewriter_validation.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/fit/models.py app/fit/rewriter.py tests/fit/builders.py tests/fit/test_rewriter_validation.py
git commit -m "feat: parse and validate FIT containers"
```
## Task 3: Patch `file_id` and only creator `device_info`
**Files:**
- Modify: `app/fit/models.py`
- Modify: `app/fit/rewriter.py`
- Create: `tests/fit/test_rewriter_patching.py`
**Interfaces:**
- Produces: `GarminDevice`, `FitConversionResult`, `DeviceFieldValue`, `convert_fit_device()`, `read_device_field_values()`.
- Default device is Garmin manufacturer `1`, Edge 1030 Plus product `3570`, name `Edge 1030 Plus`.
- [ ] **Step 1: Add model types**
```python
# app/fit/models.py additions
from pathlib import Path
GARMIN_MANUFACTURER_ID = 1
@dataclass(frozen=True)
class GarminDevice:
manufacturer_id: int = GARMIN_MANUFACTURER_ID
product_id: int = 3570
product_name: str = "Edge 1030 Plus"
serial_number: int | None = None
@dataclass(frozen=True)
class FitConversionResult:
source_path: Path
output_path: Path
patched_field_count: int
header_crc: int | None
file_crc: int
@dataclass(frozen=True)
class DeviceFieldValue:
global_message_num: int
field_num: int
value: int | str
```
- [ ] **Step 2: Write a fixture containing one creator and one sensor `device_info`**
Use these FIT field numbers in `tests/fit/test_rewriter_patching.py`:
```python
FILE_ID_MESG_NUM = 0
DEVICE_INFO_MESG_NUM = 23
# file_id: manufacturer(1/u16), product(2/u16)
file_def = definition(0, FILE_ID_MESG_NUM, [(1, 2, 0x84), (2, 2, 0x84)])
file_data = data(0, struct.pack("<HH", 255, 999))
# device_info: device_index(0/u8), manufacturer(2/u16), product(4/u16)
device_def = definition(1, DEVICE_INFO_MESG_NUM, [(0, 1, 0x02), (2, 2, 0x84), (4, 2, 0x84)])
creator = data(1, struct.pack("<BHH", 0, 255, 999))
sensor = data(1, struct.pack("<BHH", 1, 32, 1234))
```
Write assertions that `file_id` and creator become `(1, 3570)` while sensor remains `(32, 1234)`.
- [ ] **Step 3: Run the test and verify failure**
Run: `pytest tests/fit/test_rewriter_patching.py -v`
Expected: missing conversion functions.
- [ ] **Step 4: Implement conservative creator detection**
In `_patch_device_metadata`, process each `device_info` record as a unit:
```python
field_map = {field.num: (field, offset) for field, offset in field_offsets}
if definition.global_message_num == DEVICE_INFO_MESG_NUM:
device_index_entry = field_map.get(0)
if device_index_entry is None:
continue
index_field, index_offset = device_index_entry
device_index = _read_field_value(data, index_offset, index_field, definition.endian)
if device_index != 0:
continue
```
Only after this check may fields `2`, `3`, `4`, and `27` be patched. Do not patch a `device_info` message that lacks field `0`.
Always patch eligible `file_id` fields `1`, `2`, optional `3`, and `8`.
- [ ] **Step 5: Implement numeric/string read-write helpers and conversion entry point**
Use the supplied working `_read_field_value()` and `_write_field_value()` behavior, then expose:
```python
def convert_fit_device(source_path: Path, output_path: Path, device: GarminDevice | None = None) -> FitConversionResult:
resolved = device or GarminDevice()
data = bytearray(source_path.read_bytes())
_validate_fit_container(data)
patched_count = _patch_device_metadata(data, resolved)
header_crc = _rewrite_header_crc(data)
file_crc = _rewrite_file_crc(data)
_validate_fit_container(data)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_bytes(data)
return FitConversionResult(source_path, output_path, patched_count, header_crc, file_crc)
```
- [ ] **Step 6: Run patching tests**
Run: `pytest tests/fit/test_rewriter_patching.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/fit tests/fit/test_rewriter_patching.py
git commit -m "feat: patch FIT creator as Edge 1030 Plus"
```
## Task 4: Prove binary preservation and advanced record support
**Files:**
- Create: `tests/fit/test_rewriter_preservation.py`
- Modify: `tests/fit/builders.py`
- Modify: `app/fit/rewriter.py` only if a test exposes an actual parser defect
**Interfaces:**
- No new public interface; this task hardens `convert_fit_device()`.
- [ ] **Step 1: Write a preservation test that records changed byte positions**
```python
# tests/fit/test_rewriter_preservation.py
from pathlib import Path
from app.fit.rewriter import convert_fit_device
def test_only_target_fields_and_crcs_change(tmp_path: Path, complex_fit_bytes: bytes) -> None:
source = tmp_path / "source.fit"
output = tmp_path / "output.fit"
source.write_bytes(complex_fit_bytes)
convert_fit_device(source, output)
before = source.read_bytes()
after = output.read_bytes()
assert len(before) == len(after)
changed = {index for index, (a, b) in enumerate(zip(before, after)) if a != b}
expected_metadata_offsets = set(find_expected_device_metadata_offsets(before))
crc_offsets = {12, 13, len(before) - 2, len(before) - 1}
assert changed <= expected_metadata_offsets | crc_offsets
```
The fixture helper `find_expected_device_metadata_offsets()` must use the test fixture's known construction offsets, not production parser code, so the test is independent.
- [ ] **Step 2: Add fixtures for compressed timestamps, developer fields, and local-definition replacement**
Construct one synthetic FIT data section containing:
1. a normal `file_id` definition/data pair;
2. a `device_info` definition with creator record;
3. a record definition with one developer field and one data record;
4. a compressed-timestamp data header referring to a known local definition;
5. a later replacement definition for the same local message number.
The test passes if conversion completes, output CRC validates, and non-target payload bytes remain identical.
- [ ] **Step 3: Run the focused advanced tests**
Run: `pytest tests/fit/test_rewriter_preservation.py -v`
Expected: PASS. If a parser guard fails, fix only the smallest parser defect required by the test.
- [ ] **Step 4: Add rejection tests for malformed/truncated definitions**
```python
def test_truncated_definition_is_non_recoverable(tmp_path: Path) -> None:
path = tmp_path / "truncated.fit"
path.write_bytes(make_fit(bytes([0x40, 0x00, 0x00])))
assert is_fit_file(path) is False
```
Also assert that `convert_fit_device()` raises `FitFormatError` for the same input.
- [ ] **Step 5: Run the entire FIT suite**
Run: `pytest tests/fit -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add tests/fit app/fit/rewriter.py
git commit -m "test: verify FIT binary preservation"
```

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,716 @@
# MyWhoosh and Garmin Service Clients Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement isolated per-user MyWhoosh and Garmin clients with persistent tokenstores, mockable network boundaries, MyWhoosh direct API login/activity download, Garmin `import_activity()`, duplicate handling, and MFA signaling.
**Architecture:** Keep external integrations behind narrow protocols so the sync engine never depends directly on `httpx` or `garminconnect`. MyWhoosh uses an injected `httpx.AsyncClient` and a per-user JSON tokenstore. Garmin uses an injected client factory and per-user `python-garminconnect` token directory; blocking Garmin calls are later run through `asyncio.to_thread` by the sync layer.
**Tech Stack:** Python 3.12, httpx, python-garminconnect, pytest, pytest-asyncio.
## Global Constraints
- Do not automate or defeat MyWhoosh CAPTCHA/reCAPTCHA.
- Follow the direct Android-style API login flow used by the reference `jdelrue/mywhoosh2garmin` project.
- Cache MyWhoosh tokens per user under `/data/tokens/<user-id>/mywhoosh.json`.
- Cache Garmin tokens per user under `/data/tokens/<user-id>/garmin/` using the library's tokenstore behavior.
- Authentication/API changes must become explicit integration/authentication errors, not uncontrolled retries.
- Garmin activity transfer must use `import_activity()`, not `upload_activity()`.
- Known duplicate Garmin responses are successful terminal outcomes.
- Garmin MFA must raise a dedicated exception so the UI can collect a one-time code.
- Never log passwords, bearer tokens, Garmin tokens, or MFA codes.
---
## File Structure
```text
app/mywhoosh/
__init__.py
models.py
tokenstore.py
client.py
app/garmin/
__init__.py
uploader.py
tests/mywhoosh/
test_tokenstore.py
test_client_auth.py
test_client_activities.py
tests/garmin/
test_uploader.py
```
## Task 1: Implement MyWhoosh models and tokenstore
**Files:**
- Create: `app/mywhoosh/models.py`
- Create: `app/mywhoosh/tokenstore.py`
- Create: `tests/mywhoosh/test_tokenstore.py`
**Interfaces:**
- Produces `MyWhooshToken`, `MyWhooshActivity`, `MyWhooshTokenStore.load()`, `save()`, `clear()`.
- [ ] **Step 1: Write tokenstore tests**
```python
# tests/mywhoosh/test_tokenstore.py
from pathlib import Path
from app.mywhoosh.models import MyWhooshToken
from app.mywhoosh.tokenstore import MyWhooshTokenStore
def test_tokenstore_round_trip_and_permissions(tmp_path: Path) -> None:
store = MyWhooshTokenStore(tmp_path / "tokens" / "7" / "mywhoosh.json")
token = MyWhooshToken(access_token="access", refresh_token="refresh", whoosh_id="whoosh-7")
store.save(token)
assert store.load() == token
assert oct(store.path.stat().st_mode & 0o777) == "0o600"
def test_missing_token_returns_none(tmp_path: Path) -> None:
store = MyWhooshTokenStore(tmp_path / "missing.json")
assert store.load() is None
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/mywhoosh/test_tokenstore.py -v`
Expected: import failure.
- [ ] **Step 3: Implement models**
```python
# app/mywhoosh/models.py
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class MyWhooshToken:
access_token: str
refresh_token: str | None
whoosh_id: str | None
@dataclass(frozen=True)
class MyWhooshActivity:
id: str
title: str
activity_file_id: str
started_at: datetime | None
```
- [ ] **Step 4: Implement atomic JSON token persistence**
```python
# app/mywhoosh/tokenstore.py
import json
import os
from pathlib import Path
from app.mywhoosh.models import MyWhooshToken
class MyWhooshTokenStore:
def __init__(self, path: Path) -> None:
self.path = path
def load(self) -> MyWhooshToken | None:
try:
raw = json.loads(self.path.read_text("utf-8"))
except FileNotFoundError:
return None
return MyWhooshToken(
access_token=raw["access_token"],
refresh_token=raw.get("refresh_token"),
whoosh_id=raw.get("whoosh_id"),
)
def save(self, token: MyWhooshToken) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
tmp = self.path.with_suffix(".tmp")
tmp.write_text(
json.dumps(
{
"access_token": token.access_token,
"refresh_token": token.refresh_token,
"whoosh_id": token.whoosh_id,
},
indent=2,
),
"utf-8",
)
os.chmod(tmp, 0o600)
tmp.replace(self.path)
os.chmod(self.path, 0o600)
def clear(self) -> None:
self.path.unlink(missing_ok=True)
```
- [ ] **Step 5: Run tests**
Run: `pytest tests/mywhoosh/test_tokenstore.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/mywhoosh/models.py app/mywhoosh/tokenstore.py tests/mywhoosh/test_tokenstore.py
git commit -m "feat: add MyWhoosh token persistence"
```
## Task 2: Implement MyWhoosh login and cached-session recovery
**Files:**
- Create: `app/mywhoosh/client.py`
- Create: `tests/mywhoosh/test_client_auth.py`
- Modify: `pyproject.toml`
**Interfaces:**
- Produces exceptions `MyWhooshAuthError`, `MyWhooshTransientError`, `MyWhooshIntegrationError`.
- Produces `MyWhooshClient.ensure_authenticated(email: str, password: str) -> None`.
- Login endpoint: `https://services.mywhoosh.com/http-service/api/login`.
- Login payload fields: `Username`, `Password`, `Platform="Android"`, `Action=1001`, random `CorrelationId`, random `DeviceId`, `Authorization=""`.
- [ ] **Step 1: Add test dependencies**
Add to `pyproject.toml` runtime dependencies:
```toml
"httpx>=0.27,<1",
```
and test dependencies:
```toml
"pytest-asyncio>=0.24,<1",
```
- [ ] **Step 2: Write authentication tests using `httpx.MockTransport`**
```python
# tests/mywhoosh/test_client_auth.py
import httpx
import pytest
from app.mywhoosh.client import MyWhooshClient, MyWhooshAuthError
from app.mywhoosh.models import MyWhooshToken
from app.mywhoosh.tokenstore import MyWhooshTokenStore
@pytest.mark.asyncio
async def test_login_saves_access_refresh_and_whoosh_id(tmp_path) -> None:
async def handler(request: httpx.Request) -> httpx.Response:
assert request.url.path == "/http-service/api/login"
return httpx.Response(
200,
json={
"Success": True,
"AccessToken": "new-access",
"RefreshToken": "new-refresh",
"WhooshId": "w-1",
},
)
store = MyWhooshTokenStore(tmp_path / "mywhoosh.json")
client = MyWhooshClient(store, http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)))
await client.login("rider@example.com", "secret")
assert store.load() == MyWhooshToken("new-access", "new-refresh", "w-1")
@pytest.mark.asyncio
async def test_invalid_credentials_raise_auth_error(tmp_path) -> None:
async def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(200, json={"Success": False, "Message": "Invalid credentials"})
client = MyWhooshClient(
MyWhooshTokenStore(tmp_path / "mywhoosh.json"),
http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)),
)
with pytest.raises(MyWhooshAuthError):
await client.login("rider@example.com", "bad")
```
- [ ] **Step 3: Run and verify failure**
Run: `pytest tests/mywhoosh/test_client_auth.py -v`
Expected: missing client implementation.
- [ ] **Step 4: Implement exception taxonomy and login**
```python
# app/mywhoosh/client.py
from __future__ import annotations
import uuid
import httpx
from app.mywhoosh.models import MyWhooshToken
from app.mywhoosh.tokenstore import MyWhooshTokenStore
LOGIN_URL = "https://services.mywhoosh.com/http-service/api/login"
ACTIVITIES_BASE = "https://service14.mywhoosh.com/v2/"
class MyWhooshError(RuntimeError):
pass
class MyWhooshAuthError(MyWhooshError):
pass
class MyWhooshTransientError(MyWhooshError):
pass
class MyWhooshIntegrationError(MyWhooshError):
pass
class MyWhooshClient:
def __init__(self, token_store: MyWhooshTokenStore, http_client: httpx.AsyncClient | None = None) -> None:
self.token_store = token_store
self.http = http_client or httpx.AsyncClient(timeout=30.0)
self.token = token_store.load()
async def login(self, email: str, password: str) -> None:
payload = {
"Username": email,
"Password": password,
"Platform": "Android",
"Action": 1001,
"CorrelationId": str(uuid.uuid4()),
"DeviceId": str(uuid.uuid4()),
"Authorization": "",
}
try:
response = await self.http.post(LOGIN_URL, json=payload)
except httpx.TransportError as exc:
raise MyWhooshTransientError("MyWhoosh login request failed") from exc
if response.status_code >= 500:
raise MyWhooshTransientError(f"MyWhoosh login returned HTTP {response.status_code}")
if response.status_code >= 400:
raise MyWhooshAuthError(f"MyWhoosh login returned HTTP {response.status_code}")
try:
body = response.json()
except ValueError as exc:
raise MyWhooshIntegrationError("MyWhoosh login returned invalid JSON") from exc
if body.get("Success") is not True or not body.get("AccessToken"):
raise MyWhooshAuthError(str(body.get("Message") or "MyWhoosh login failed"))
self.token = MyWhooshToken(
access_token=str(body["AccessToken"]),
refresh_token=str(body["RefreshToken"]) if body.get("RefreshToken") else None,
whoosh_id=str(body["WhooshId"]) if body.get("WhooshId") else None,
)
self.token_store.save(self.token)
```
- [ ] **Step 5: Implement `ensure_authenticated` as cache-first validation**
Do not invent a refresh endpoint. Validate cached tokens using the normal activities request; on `401/403`, clear the cache, login once, and continue. The later `list_activities()` task provides the request method. Expose the intended behavior now:
```python
async def ensure_authenticated(self, email: str, password: str) -> None:
if self.token is None:
await self.login(email, password)
```
Task 3 extends this with one retry after an unauthorized activities response.
- [ ] **Step 6: Run authentication tests**
Run: `pytest tests/mywhoosh/test_client_auth.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add pyproject.toml app/mywhoosh/client.py tests/mywhoosh/test_client_auth.py
git commit -m "feat: add MyWhoosh API login"
```
## Task 3: Implement MyWhoosh activity listing and FIT download
**Files:**
- Modify: `app/mywhoosh/client.py`
- Create: `tests/mywhoosh/test_client_activities.py`
**Interfaces:**
- Produces `list_activities(email: str, password: str) -> list[MyWhooshActivity]`.
- Produces `download_fit(activity_file_id: str, email: str, password: str) -> bytes`.
- Activities endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/activities` with `{"sortDate":"DESC","page":N}`.
- Download endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/download-activity-file` with `{"fileId": activity_file_id}`; response `data` is a pre-signed URL which is fetched with GET.
- [ ] **Step 1: Write paginated activity-list test**
```python
@pytest.mark.asyncio
async def test_list_activities_paginates_and_normalizes(tmp_path) -> None:
calls = []
async def handler(request: httpx.Request) -> httpx.Response:
calls.append(str(request.url))
if request.url.path.endswith("/activities"):
payload = json.loads(request.content)
page = payload["page"]
result = {
"data": {
"totalPages": 2,
"results": [{
"id": f"a-{page}",
"title": f"Ride {page}",
"activityFileId": f"f-{page}",
"startDatetime": "2026-08-15T06:00:00.000Z",
}],
}
}
return httpx.Response(200, json=result)
raise AssertionError(request.url)
```
Preload the tokenstore with `access_token="cached"`; assert two normalized `MyWhooshActivity` values are returned.
- [ ] **Step 2: Write expired-token reauthentication test**
The mock transport sequence must return `401` for the first activities request, a successful login response, then `200` for the retried activities request. Assert login is attempted exactly once and the tokenstore contains the new access token.
- [ ] **Step 3: Write FIT download test**
Mock the download-activity-file endpoint to return `{"data":"https://signed.example/activity.fit"}`, then mock that URL to return bytes beginning with a valid FIT header. Assert `download_fit()` returns those exact bytes.
- [ ] **Step 4: Implement one authenticated-request retry helper**
```python
async def _authenticated_post(self, url: str, payload: dict, email: str, password: str) -> httpx.Response:
await self.ensure_authenticated(email, password)
for attempt in range(2):
assert self.token is not None
try:
response = await self.http.post(
url,
json=payload,
headers={"Authorization": f"Bearer {self.token.access_token}"},
)
except httpx.TransportError as exc:
raise MyWhooshTransientError("MyWhoosh request failed") from exc
if response.status_code not in {401, 403}:
if response.status_code >= 500:
raise MyWhooshTransientError(f"MyWhoosh returned HTTP {response.status_code}")
return response
if attempt == 0:
self.token_store.clear()
self.token = None
await self.login(email, password)
continue
raise MyWhooshAuthError("MyWhoosh session rejected after reauthentication")
raise AssertionError("unreachable")
```
- [ ] **Step 5: Implement list normalization and download**
Parse `startDatetime` as UTC when present. Skip malformed activity rows only if they lack no stable `id` or `activityFileId`; otherwise surface JSON/schema failures as `MyWhooshIntegrationError` so API changes are visible.
```python
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
response = await self._authenticated_post(
ACTIVITIES_BASE + "rider/profile/download-activity-file",
{"fileId": activity_file_id},
email,
password,
)
if response.status_code >= 400:
raise MyWhooshIntegrationError(f"download metadata returned HTTP {response.status_code}")
url = response.json().get("data")
if not isinstance(url, str) or not url:
raise MyWhooshIntegrationError("MyWhoosh download response has no URL")
try:
fit_response = await self.http.get(url)
except httpx.TransportError as exc:
raise MyWhooshTransientError("FIT download failed") from exc
if fit_response.status_code >= 500:
raise MyWhooshTransientError(f"FIT host returned HTTP {fit_response.status_code}")
if fit_response.status_code >= 400:
raise MyWhooshIntegrationError(f"FIT host returned HTTP {fit_response.status_code}")
return fit_response.content
```
- [ ] **Step 6: Run MyWhoosh tests**
Run: `pytest tests/mywhoosh -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/mywhoosh/client.py tests/mywhoosh/test_client_activities.py
git commit -m "feat: fetch MyWhoosh activities and FIT files"
```
## Task 4: Implement Garmin import adapter with duplicate and MFA handling
**Files:**
- Create: `app/garmin/uploader.py`
- Create: `tests/garmin/test_uploader.py`
- Modify: `pyproject.toml`
**Interfaces:**
- Produces `GarminUploader.import_fit(fit_path: Path, mfa_code: str | None = None) -> UploadResult`.
- Produces exceptions `GarminUploadBlocked`, `GarminAuthError`, `GarminTransientError`.
- Uses `client.login(tokenstore_path)` and `client.import_activity(activity_path)`.
- [ ] **Step 1: Add Garmin dependency**
Add to runtime dependencies:
```toml
"garminconnect>=0.2,<1",
```
- [ ] **Step 2: Write fake-client tests based on the previously working uploader pattern**
```python
# tests/garmin/test_uploader.py
from pathlib import Path
import pytest
from app.garmin.uploader import GarminUploadBlocked, GarminUploader
class FakeGarmin:
def __init__(self, *args, prompt_mfa=None, import_result=None, login_error=None, import_error=None, **kwargs):
self.prompt_mfa = prompt_mfa
self.import_result = import_result or {"activityId": 42}
self.login_error = login_error
self.import_error = import_error
self.login_path = None
def login(self, tokenstore=None):
self.login_path = tokenstore
if self.login_error:
raise self.login_error
def import_activity(self, activity_path: str):
if self.import_error:
raise self.import_error
return self.import_result
```
Add these concrete tests below the fake client:
```python
def test_successful_import_returns_activity_id(tmp_path: Path) -> None:
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_result={"activityId": 42}),
)
result = uploader.import_fit(tmp_path / "ride.fit")
assert result.status == "imported"
assert result.garmin_activity_id == "42"
def test_duplicate_is_terminal_success(tmp_path: Path) -> None:
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_error=RuntimeError("409 duplicate")),
)
result = uploader.import_fit(tmp_path / "ride.fit")
assert result.duplicate is True
assert result.status == "duplicate"
def test_mfa_without_code_is_blocked(tmp_path: Path) -> None:
class MfaGarmin(FakeGarmin):
def login(self, tokenstore=None):
self.prompt_mfa()
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=MfaGarmin,
)
with pytest.raises(GarminUploadBlocked):
uploader.import_fit(tmp_path / "ride.fit")
def test_mfa_code_is_returned_only_to_prompt(tmp_path: Path) -> None:
seen = []
class MfaGarmin(FakeGarmin):
def login(self, tokenstore=None):
seen.append(self.prompt_mfa())
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=MfaGarmin,
)
uploader.import_fit(tmp_path / "ride.fit", mfa_code="123456")
assert seen == ["123456"]
```
- [ ] **Step 3: Run and verify failure**
Run: `pytest tests/garmin/test_uploader.py -v`
Expected: import failure.
- [ ] **Step 4: Implement the adapter**
```python
# app/garmin/uploader.py
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Protocol
class GarminClientProtocol(Protocol):
def login(self, tokenstore: str | None = None) -> Any: ...
def import_activity(self, activity_path: str) -> Any: ...
@dataclass(frozen=True)
class UploadResult:
status: str
duplicate: bool
garmin_activity_id: str | None
raw_response: Any
class GarminUploadBlocked(RuntimeError):
pass
class GarminAuthError(RuntimeError):
pass
class GarminTransientError(RuntimeError):
pass
```
Implement the uploader fully:
```python
class GarminUploader:
def __init__(
self,
*,
email: str,
password: str,
tokenstore: Path,
client_factory: Callable[..., GarminClientProtocol] | None = None,
) -> None:
self.email = email
self.password = password
self.tokenstore = tokenstore
self.client_factory = client_factory
self._client: GarminClientProtocol | None = None
self._mfa_code: str | None = None
def import_fit(self, fit_path: Path, mfa_code: str | None = None) -> UploadResult:
self._mfa_code = mfa_code
try:
client = self._ensure_client()
try:
response = client.import_activity(str(fit_path))
except Exception as exc:
if _looks_duplicate_error(exc):
return UploadResult("duplicate", True, None, str(exc))
text = str(exc).lower()
if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")):
raise GarminTransientError("Garmin import failed transiently") from exc
raise
return UploadResult("imported", False, _extract_activity_id(response), response)
finally:
self._mfa_code = None
def _ensure_client(self) -> GarminClientProtocol:
if self._client is not None:
return self._client
self.tokenstore.mkdir(parents=True, exist_ok=True)
factory = self.client_factory or _default_garmin_factory
client = factory(self.email, self.password, prompt_mfa=self._prompt_mfa)
try:
client.login(str(self.tokenstore))
except GarminUploadBlocked:
raise
except Exception as exc:
text = str(exc).lower()
if "mfa" in text:
raise GarminUploadBlocked("Garmin MFA is required") from exc
if any(token in text for token in ("password", "credential", "unauthorized", "401")):
raise GarminAuthError("Garmin authentication failed") from exc
if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")):
raise GarminTransientError("Garmin login failed transiently") from exc
raise GarminAuthError("Garmin login failed") from exc
self._client = client
return client
def _prompt_mfa(self) -> str:
if self._mfa_code:
return self._mfa_code
raise GarminUploadBlocked("Garmin requested MFA")
def _default_garmin_factory(*args: Any, **kwargs: Any) -> GarminClientProtocol:
from garminconnect import Garmin
return Garmin(*args, **kwargs)
```
- [ ] **Step 5: Keep duplicate and response-ID extraction deterministic**
Use these helpers:
```python
def _looks_duplicate_error(exc: Exception) -> bool:
text = str(exc).lower()
return any(token in text for token in ("duplicate", "already exists", "409"))
def _extract_activity_id(response: Any) -> str | None:
if not isinstance(response, dict):
return None
candidates = [response.get("activityId"), response.get("activity_id"), response.get("id")]
detailed = response.get("detailedImportResult")
if isinstance(detailed, dict):
candidates.extend([detailed.get("uploadId"), detailed.get("activityId")])
for key in ("successes", "success", "importedActivities"):
items = response.get(key)
if isinstance(items, list) and items and isinstance(items[0], dict):
candidates.extend([items[0].get("activityId"), items[0].get("id")])
return next((str(value) for value in candidates if value is not None), None)
```
- [ ] **Step 6: Run Garmin tests**
Run: `pytest tests/garmin/test_uploader.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add pyproject.toml app/garmin/uploader.py tests/garmin/test_uploader.py
git commit -m "feat: import FIT activities into Garmin"
```

View File

@@ -0,0 +1,804 @@
# Sync Engine, Scheduler, and Operational UI Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Integrate the database, MyWhoosh client, FIT rewriter, Garmin importer, scheduler, MFA workflow, retry behavior, and operational admin pages into a resilient multi-user sync service.
**Architecture:** A `SyncManager` owns per-user `asyncio.Lock` instances and executes a durable activity state machine. External clients are injected via factories for tests. A lightweight FastAPI lifespan scheduler triggers syncs at the configured interval; different users run concurrently, while each user's pipeline is serialized.
**Tech Stack:** Python 3.12, asyncio, FastAPI lifespan, SQLAlchemy, HTMX, Jinja2, pytest/pytest-asyncio.
## Global Constraints
- Multiple users sync independently and may run concurrently.
- At most one sync may run for a given user at a time.
- Manual sync and scheduled sync use the same pipeline and lock.
- Durable activity stages are `discovered`, `downloaded`, `converted`, `imported`, `duplicate`, `failed` with `last_completed_stage` retained on failure.
- `imported` and `duplicate` are terminal.
- Transient network/server failures retry at most once in a run; later attempts occur on future scheduler ticks.
- Invalid MyWhoosh/Garmin credentials and Garmin MFA set `action_required`.
- Corrupt/unsupported FIT is non-retryable per activity.
- Failure of one user or one activity must never stop other users.
- Original and converted FIT files remain on disk in v1.
- MFA codes are never persisted or logged.
---
## File Structure
```text
app/sync/
__init__.py
states.py
manager.py
scheduler.py
app/web/
operations.py
templates/
dashboard.html
users/detail.html
system.html
fragments/user_card.html
fragments/sync_result.html
fragments/mfa_form.html
app/db/
repositories.py
tests/sync/
fakes.py
test_manager.py
test_concurrency.py
test_scheduler.py
tests/web/
test_operations.py
test_mfa.py
```
## Task 1: Add durable activity/sync-run repository operations
**Files:**
- Modify: `app/db/repositories.py`
- Create: `tests/db/test_sync_state.py`
**Interfaces:**
- Produces methods to advance activity stages, mark failures without losing `last_completed_stage`, list pending activities, and create/finalize sync runs.
- [ ] **Step 1: Write failing state-transition tests**
```python
# tests/db/test_sync_state.py
from app.db.models import ActivityStatus
def test_failure_retains_last_completed_stage(activity_repository, seeded_activity) -> None:
activity_repository.mark_downloaded(seeded_activity.id, "/data/activities/1/a/source.fit")
activity_repository.mark_failed(seeded_activity.id, "Garmin timeout", retryable=True)
activity = activity_repository.get(seeded_activity.id)
assert activity.status == ActivityStatus.FAILED
assert activity.last_completed_stage == ActivityStatus.DOWNLOADED
assert activity.retryable is True
def test_converted_activity_is_pending_until_terminal(activity_repository, seeded_activity) -> None:
activity_repository.mark_converted(seeded_activity.id, "/data/activities/1/a/converted.fit")
ids = [item.id for item in activity_repository.list_pending_for_user(seeded_activity.user_id)]
assert seeded_activity.id in ids
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/db/test_sync_state.py -v`
Expected: missing repository methods.
- [ ] **Step 3: Implement explicit transition methods**
Add methods with these exact effects:
```python
def mark_downloaded(self, activity_id: int, path: str) -> Activity:
activity = self._require(activity_id)
activity.source_fit_path = path
activity.status = ActivityStatus.DOWNLOADED
activity.last_completed_stage = ActivityStatus.DOWNLOADED
activity.last_error = None
activity.retryable = True
self.session.commit()
return activity
def mark_converted(self, activity_id: int, path: str) -> Activity:
activity = self._require(activity_id)
activity.converted_fit_path = path
activity.status = ActivityStatus.CONVERTED
activity.last_completed_stage = ActivityStatus.CONVERTED
activity.last_error = None
activity.retryable = True
self.session.commit()
return activity
def mark_imported(self, activity_id: int, garmin_activity_id: str | None) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.IMPORTED
activity.last_completed_stage = ActivityStatus.IMPORTED
activity.garmin_activity_id = garmin_activity_id
activity.last_error = None
activity.retryable = False
self.session.commit()
return activity
def mark_duplicate(self, activity_id: int) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.DUPLICATE
activity.last_completed_stage = ActivityStatus.DUPLICATE
activity.last_error = None
activity.retryable = False
self.session.commit()
return activity
def mark_failed(self, activity_id: int, error: str, *, retryable: bool) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.FAILED
activity.last_error = error[:2000]
activity.retryable = retryable
self.session.commit()
return activity
```
`list_pending_for_user()` must exclude terminal states and include failed rows only when `retryable=True`.
- [ ] **Step 4: Add `SyncRunRepository` create/finalize methods**
`start(user_id)` creates `RUNNING`; `finish(...)` sets counts, `finished_at`, status, and optional summary error. Do not store exception tracebacks in SQLite.
- [ ] **Step 5: Run DB state tests**
Run: `pytest tests/db/test_sync_state.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/db/repositories.py tests/db/test_sync_state.py
git commit -m "feat: add durable sync state transitions"
```
## Task 2: Implement the single-user sync state machine
**Files:**
- Create: `app/sync/states.py`
- Create: `app/sync/manager.py`
- Create: `tests/sync/fakes.py`
- Create: `tests/sync/test_manager.py`
**Interfaces:**
- Produces `SyncManager.sync_user(user_id: int, mfa_code: str | None = None) -> SyncOutcome`.
- Constructor receives `session_factory`, `credential_cipher`, `settings`, `mywhoosh_factory`, `garmin_factory`, and `fit_converter`.
- `mywhoosh_factory(token_store: MyWhooshTokenStore) -> MyWhooshClient`.
- `garmin_factory(email: str, password: str, tokenstore: Path) -> GarminUploader`.
- `fit_converter(source_path: Path, output_path: Path) -> FitConversionResult`.
- [ ] **Step 1: Define result models and fake integration factories**
```python
# app/sync/states.py
from dataclasses import dataclass
@dataclass(frozen=True)
class SyncOutcome:
user_id: int
status: str
discovered: int
imported: int
skipped: int
failed: int
message: str | None = None
```
Implement concrete fakes:
```python
# tests/sync/fakes.py
from app.garmin.uploader import UploadResult
class FakeMyWhooshClient:
def __init__(self, activities, fit_bytes: bytes) -> None:
self.activities = activities
self.fit_bytes = fit_bytes
self.list_calls = 0
self.download_calls = 0
async def list_activities(self, email: str, password: str):
self.list_calls += 1
return list(self.activities)
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
self.download_calls += 1
return self.fit_bytes
class FakeGarminUploader:
def __init__(self, result: UploadResult | None = None, error: Exception | None = None) -> None:
self.result = result or UploadResult("imported", False, "g-1", {"activityId": "g-1"})
self.error = error
self.calls = 0
def import_fit(self, fit_path, mfa_code=None):
self.calls += 1
if self.error is not None:
raise self.error
return self.result
```
- [ ] **Step 2: Write the happy-path test**
```python
@pytest.mark.asyncio
async def test_new_activity_downloads_converts_and_imports(manager, seeded_user, tmp_path) -> None:
outcome = await manager.sync_user(seeded_user.id)
assert outcome.discovered == 1
assert outcome.imported == 1
assert outcome.failed == 0
activity = load_only_activity(seeded_user.id)
assert activity.status == ActivityStatus.IMPORTED
assert Path(activity.source_fit_path).exists()
assert Path(activity.converted_fit_path).exists()
```
- [ ] **Step 3: Write resume tests before implementation**
```python
@pytest.mark.asyncio
@pytest.mark.parametrize(
("status", "last_stage", "expected_downloads", "expected_conversions", "expected_imports"),
[
(ActivityStatus.DOWNLOADED, ActivityStatus.DOWNLOADED, 0, 1, 1),
(ActivityStatus.CONVERTED, ActivityStatus.CONVERTED, 0, 0, 1),
(ActivityStatus.IMPORTED, ActivityStatus.IMPORTED, 0, 0, 0),
(ActivityStatus.FAILED, ActivityStatus.CONVERTED, 0, 0, 1),
],
)
async def test_resume_from_durable_stage(
manager_factory, seeded_activity_factory, status, last_stage,
expected_downloads, expected_conversions, expected_imports,
) -> None:
activity = seeded_activity_factory(status=status, last_completed_stage=last_stage, retryable=True)
manager, mywhoosh, converter, garmin = manager_factory(activity)
await manager.sync_user(activity.user_id)
assert mywhoosh.download_calls == expected_downloads
assert converter.calls == expected_conversions
assert garmin.calls == expected_imports
```
- [ ] **Step 4: Run and verify failure**
Run: `pytest tests/sync/test_manager.py -v`
Expected: missing manager.
- [ ] **Step 5: Implement per-activity filesystem layout and state machine**
Use paths:
```python
activity_dir = settings.activities_dir / str(user.id) / activity.mywhoosh_activity_id
source_path = activity_dir / "source.fit"
converted_path = activity_dir / "edge-1030-plus.fit"
```
Create per-user integration instances from decrypted credentials and isolated token paths:
```python
mw_email = self.credential_cipher.decrypt(user.mywhoosh_email_enc)
mw_password = self.credential_cipher.decrypt(user.mywhoosh_password_enc)
garmin_email = self.credential_cipher.decrypt(user.garmin_email_enc)
garmin_password = self.credential_cipher.decrypt(user.garmin_password_enc)
token_dir = self.settings.tokens_dir / str(user.id)
mywhoosh = self.mywhoosh_factory(MyWhooshTokenStore(token_dir / "mywhoosh.json"))
garmin = self.garmin_factory(garmin_email, garmin_password, token_dir / "garmin")
remote_activities = await mywhoosh.list_activities(mw_email, mw_password)
```
For each remote activity, call `get_or_create_discovered(...)`, then resume from `activity.last_completed_stage` when `activity.status == FAILED`; otherwise use `activity.status`.
Core sequence:
```python
if stage == ActivityStatus.DISCOVERED:
fit_bytes = await mywhoosh.download_fit(remote.activity_file_id, mw_email, mw_password)
activity_dir.mkdir(parents=True, exist_ok=True)
source_path.write_bytes(fit_bytes)
repo.mark_downloaded(activity.id, str(source_path))
if stage in {ActivityStatus.DOWNLOADED}:
fit_converter(source_path, converted_path)
repo.mark_converted(activity.id, str(converted_path))
if stage in {ActivityStatus.CONVERTED}:
upload = await asyncio.to_thread(garmin.import_fit, converted_path, mfa_code)
if upload.duplicate:
repo.mark_duplicate(activity.id)
else:
repo.mark_imported(activity.id, upload.garmin_activity_id)
```
After each repository transition, update the local `stage` variable from the returned record so resume behavior is deterministic.
- [ ] **Step 6: Implement exception mapping**
Map exceptions with explicit user connection-state updates:
```python
except MyWhooshTransientError as exc:
user.health_state = HealthState.DEGRADED
user.mywhoosh_state = "error"
repo.mark_failed(activity.id, str(exc), retryable=True)
except MyWhooshAuthError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.mywhoosh_state = "auth_required"
user.action_reason = "mywhoosh_auth_required"
stop_user_run = True
except MyWhooshIntegrationError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.mywhoosh_state = "integration_error"
user.action_reason = "mywhoosh_integration_changed"
stop_user_run = True
except GarminUploadBlocked:
user.health_state = HealthState.ACTION_REQUIRED
user.garmin_state = "mfa_required"
user.action_reason = "garmin_mfa_required"
stop_user_run = True
except GarminAuthError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.garmin_state = "auth_required"
user.action_reason = "garmin_auth_required"
stop_user_run = True
except GarminTransientError as exc:
user.health_state = HealthState.DEGRADED
user.garmin_state = "error"
repo.mark_failed(activity.id, str(exc), retryable=True)
except FitFormatError as exc:
repo.mark_failed(activity.id, str(exc), retryable=False)
```
On successful MyWhoosh listing set `mywhoosh_state="connected"`; on successful Garmin import set `garmin_state="connected"`. Persist the user after each connection-state change. Unexpected exceptions mark the run/user `degraded` and log only exception class plus sanitized message.
- [ ] **Step 7: Run manager tests**
Run: `pytest tests/sync/test_manager.py -v`
Expected: PASS.
- [ ] **Step 8: Commit**
```bash
git add app/sync/states.py app/sync/manager.py tests/sync
git commit -m "feat: add resumable per-user sync pipeline"
```
## Task 3: Add per-user locks and cross-user isolation
**Files:**
- Modify: `app/sync/manager.py`
- Create: `tests/sync/test_concurrency.py`
**Interfaces:**
- Produces `SyncAlreadyRunning` and ensures only one active `sync_user()` call per user.
- [ ] **Step 1: Write concurrency tests**
```python
@pytest.mark.asyncio
async def test_same_user_cannot_run_twice(manager, seeded_user) -> None:
first_started = asyncio.Event()
release_first = asyncio.Event()
manager.test_hooks = SyncTestHooks(first_started=first_started, release=release_first)
first = asyncio.create_task(manager.sync_user(seeded_user.id))
await first_started.wait()
with pytest.raises(SyncAlreadyRunning):
await manager.sync_user(seeded_user.id)
release_first.set()
await first
@pytest.mark.asyncio
async def test_different_users_can_run_concurrently(manager, user_a, user_b) -> None:
results = await asyncio.gather(manager.sync_user(user_a.id), manager.sync_user(user_b.id))
assert {result.user_id for result in results} == {user_a.id, user_b.id}
```
Do not leave production-only `test_hooks`; instead inject a fake MyWhoosh client whose `list_activities()` blocks on test events.
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/sync/test_concurrency.py -v`
Expected: same-user duplicate execution is not yet blocked.
- [ ] **Step 3: Implement lock registry**
```python
class SyncAlreadyRunning(RuntimeError):
pass
class SyncManager:
def __init__(...):
self._locks: dict[int, asyncio.Lock] = {}
self._locks_guard = asyncio.Lock()
async def _lock_for(self, user_id: int) -> asyncio.Lock:
async with self._locks_guard:
return self._locks.setdefault(user_id, asyncio.Lock())
async def sync_user(self, user_id: int, mfa_code: str | None = None) -> SyncOutcome:
lock = await self._lock_for(user_id)
if lock.locked():
raise SyncAlreadyRunning(f"sync already running for user {user_id}")
async with lock:
return await self._sync_user_locked(user_id, mfa_code)
```
- [ ] **Step 4: Add a `sync_all_enabled()` isolation method**
```python
async def sync_all_enabled(self) -> list[SyncOutcome | Exception]:
user_ids = self._load_enabled_user_ids()
return await asyncio.gather(
*(self.sync_user(user_id) for user_id in user_ids),
return_exceptions=True,
)
```
A failure for one user must appear as one list element and must not cancel sibling jobs.
- [ ] **Step 5: Run concurrency tests**
Run: `pytest tests/sync/test_concurrency.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/sync/manager.py tests/sync/test_concurrency.py
git commit -m "feat: isolate concurrent user syncs"
```
## Task 4: Add the periodic scheduler through FastAPI lifespan
**Files:**
- Create: `app/sync/scheduler.py`
- Modify: `app/main.py`
- Create: `tests/sync/test_scheduler.py`
**Interfaces:**
- Produces `SyncScheduler.start()`, `stop()`, `run_once()`, `last_tick`, `next_tick`.
- Scheduler interval is `Settings.sync_interval_minutes`.
- [ ] **Step 1: Write scheduler test with a short injected interval**
```python
@pytest.mark.asyncio
async def test_scheduler_calls_sync_all_and_survives_failure() -> None:
fake = FakeSyncManager(results=[RuntimeError("one user failed")])
scheduler = SyncScheduler(fake, interval_seconds=0.01)
await scheduler.start()
await asyncio.sleep(0.035)
await scheduler.stop()
assert fake.calls >= 2
assert scheduler.last_tick is not None
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/sync/test_scheduler.py -v`
Expected: missing scheduler.
- [ ] **Step 3: Implement scheduler loop**
```python
class SyncScheduler:
def __init__(self, manager, *, interval_seconds: float) -> None:
self.manager = manager
self.interval_seconds = interval_seconds
self._task: asyncio.Task | None = None
self._stop = asyncio.Event()
self.last_tick = None
self.next_tick = None
async def run_once(self) -> None:
self.last_tick = datetime.now(timezone.utc)
await self.manager.sync_all_enabled()
self.next_tick = datetime.now(timezone.utc) + timedelta(seconds=self.interval_seconds)
async def _run(self) -> None:
while not self._stop.is_set():
await self.run_once()
try:
await asyncio.wait_for(self._stop.wait(), timeout=self.interval_seconds)
except TimeoutError:
pass
```
`stop()` sets the event and awaits the task. Never allow one `sync_all_enabled()` exception to kill the loop; log it and continue.
- [ ] **Step 4: Wire into FastAPI lifespan**
Build the concrete `SyncManager` once during app startup, store it on `app.state.sync_manager`, create `SyncScheduler(... interval_minutes * 60)`, start it, and stop it during lifespan shutdown.
- [ ] **Step 5: Run scheduler tests**
Run: `pytest tests/sync/test_scheduler.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/sync/scheduler.py app/main.py tests/sync/test_scheduler.py
git commit -m "feat: schedule periodic user synchronization"
```
## Task 5: Add dashboard/manual sync/system operational routes
**Files:**
- Create: `app/web/operations.py`
- Modify: `app/web/routes.py`
- Modify: `app/web/templates/dashboard.html`
- Create: `app/web/templates/system.html`
- Create: `app/web/templates/fragments/sync_result.html`
- Create: `tests/web/test_operations.py`
**Interfaces:**
- Routes: `POST /users/{id}/sync`, `POST /sync-all`, `GET /system`.
- Manual actions use the same `SyncManager` instance and lock as the scheduler.
- [ ] **Step 1: Write manual-sync tests**
```python
def test_manual_sync_calls_shared_manager(authenticated_client, fake_sync_manager) -> None:
response = authenticated_client.post(
"/users/1/sync",
data={"csrf_token": authenticated_client.csrf_token},
)
assert response.status_code == 200
assert fake_sync_manager.user_calls == [1]
def test_manual_sync_reports_already_running(authenticated_client, fake_sync_manager) -> None:
fake_sync_manager.raise_already_running = True
response = authenticated_client.post(
"/users/1/sync",
data={"csrf_token": authenticated_client.csrf_token},
)
assert response.status_code == 409
assert "already running" in response.text.lower()
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/web/test_operations.py -v`
Expected: routes missing.
- [ ] **Step 3: Implement routes with admin and CSRF checks**
Each state-changing route must execute in this order:
```python
require_admin(request)
validate_csrf(request, csrf_token)
```
Then call `await request.app.state.sync_manager.sync_user(user_id)` or `sync_all_enabled()`.
- [ ] **Step 4: Expand dashboard data**
Add a repository projection that contains only safe display fields:
```python
@dataclass(frozen=True)
class UserDashboardRow:
id: int
name: str
enabled: bool
health_state: str
action_reason: str | None
last_sync_at: datetime | None
last_activity_name: str | None
last_activity_status: str | None
def dashboard_rows(self) -> list[UserDashboardRow]:
users = self.list_all()
rows = []
for user in users:
last_run = self.session.scalar(
select(SyncRun).where(SyncRun.user_id == user.id).order_by(SyncRun.started_at.desc()).limit(1)
)
last_activity = self.session.scalar(
select(Activity).where(Activity.user_id == user.id).order_by(Activity.created_at.desc()).limit(1)
)
rows.append(UserDashboardRow(
id=user.id,
name=user.name,
enabled=user.enabled,
health_state=user.health_state.value,
action_reason=user.action_reason,
last_sync_at=last_run.finished_at if last_run else None,
last_activity_name=last_activity.activity_name if last_activity else None,
last_activity_status=last_activity.status.value if last_activity else None,
))
return rows
```
Pass only these rows to `dashboard.html`. Render the MFA action only when `row.action_reason == "garmin_mfa_required"`. No decrypted credential is part of this projection.
- [ ] **Step 5: Implement read-only system page**
Expose application version, configured interval, scheduler `last_tick` and `next_tick`, user count, and activity count. The only action is a CSRF-protected `sync all now` POST.
- [ ] **Step 6: Run tests**
Run: `pytest tests/web/test_operations.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/web tests/web/test_operations.py
git commit -m "feat: add operational sync controls"
```
## Task 6: Add Garmin MFA lifecycle and failed-activity retry
**Files:**
- Modify: `app/web/routes.py`
- Modify: `app/web/templates/users/detail.html`
- Create: `app/web/templates/fragments/mfa_form.html`
- Create: `tests/web/test_mfa.py`
- Modify: `app/sync/manager.py`
**Interfaces:**
- Route: `POST /users/{id}/garmin-mfa` with one-time `code`.
- Route: `POST /activities/{id}/retry`.
- MFA code exists only in request memory and the immediate `sync_user(user_id, mfa_code=code)` call.
- [ ] **Step 1: Write MFA lifecycle test**
```python
def test_mfa_code_is_used_once_and_not_persisted(authenticated_client, fake_sync_manager, db_session) -> None:
response = authenticated_client.post(
"/users/1/garmin-mfa",
data={"csrf_token": authenticated_client.csrf_token, "code": "123456"},
)
assert response.status_code == 200
assert fake_sync_manager.mfa_calls == [(1, "123456")]
persisted_text = " ".join(str(row) for row in db_session.execute(text("select * from sync_runs")).all())
assert "123456" not in persisted_text
```
- [ ] **Step 2: Write retry test for non-terminal failed activity**
Assert the route changes a retryable failed activity back to `status=last_completed_stage`, clears `last_error`, then calls the user's normal sync. Reject retry for `retryable=False` with HTTP 409.
- [ ] **Step 3: Implement MFA route**
Validate code as a non-empty short string, never log it, and call:
```python
outcome = await request.app.state.sync_manager.sync_user(user_id, mfa_code=code.strip())
```
After a successful Garmin login/import, clear `action_reason` and restore health to `healthy` or `degraded` according to the resulting sync outcome.
- [ ] **Step 4: Implement failed-activity reset operation**
Repository method:
```python
def reset_retryable_failure(self, activity_id: int) -> Activity:
activity = self._require(activity_id)
if activity.status != ActivityStatus.FAILED or not activity.retryable:
raise ValueError("activity is not retryable")
activity.status = activity.last_completed_stage
activity.last_error = None
self.session.commit()
return activity
```
- [ ] **Step 5: Run MFA/retry tests**
Run: `pytest tests/web/test_mfa.py tests/web/test_operations.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/web app/sync/manager.py app/db/repositories.py tests/web/test_mfa.py
git commit -m "feat: handle Garmin MFA and activity retries"
```
## Task 7: End-to-end regression and Docker acceptance
**Files:**
- Modify: `docker-compose.example.yml` only if integration exposes a missing runtime configuration
- Create: `tests/test_acceptance.py`
**Interfaces:**
- No new interface; verifies the v1 acceptance criteria with fake external services.
- [ ] **Step 1: Add an application-level acceptance test with two users**
Build the app with temporary SQLite/data directories and injected fake MyWhoosh/Garmin factories. Seed two enabled users, give each one distinct remote activity IDs, run `sync_all_enabled()`, and assert:
```python
assert all(result.status == "success" for result in results)
assert count_terminal_activities(user_a.id) == 1
assert count_terminal_activities(user_b.id) == 1
assert user_a_source_path.parent != user_b_source_path.parent
assert user_a_garmin_factory.tokenstore != user_b_garmin_factory.tokenstore
```
- [ ] **Step 2: Add isolation acceptance test**
Configure User B to raise `GarminUploadBlocked`; assert User A still imports and User B ends `action_required` with no impact on User A.
- [ ] **Step 3: Run the full suite**
Run: `pytest -v`
Expected: PASS.
- [ ] **Step 4: Build Docker image again**
Run: `docker build -t mywhoosh-garmin-sync:test .`
Expected: successful build with the complete dependency set.
- [ ] **Step 5: Start local container and exercise smoke paths**
Start with a temporary bind-mounted `/data`, then verify:
```bash
curl -fsS http://127.0.0.1:18080/healthz
curl -I http://127.0.0.1:18080/
```
Expected: health JSON and dashboard redirect to `/login` when unauthenticated.
- [ ] **Step 6: Verify secrets are absent from captured test logs**
Run:
```bash
pytest -v 2>&1 | tee /tmp/mywhoosh-garmin-test.log
! grep -F "mw-secret" /tmp/mywhoosh-garmin-test.log
! grep -F "garmin-secret" /tmp/mywhoosh-garmin-test.log
! grep -F "123456" /tmp/mywhoosh-garmin-test.log
```
Expected: all three negated `grep` commands succeed.
- [ ] **Step 7: Commit**
```bash
git add tests/test_acceptance.py docker-compose.example.yml
git commit -m "test: cover multi-user sync acceptance"
```

View File

@@ -0,0 +1,620 @@
# MyWhoosh -> Garmin Sync Service — Design
Date: 2026-08-15
Status: Draft for user review
## 1. Goal
Build a self-hosted Docker service that periodically checks MyWhoosh for new cycling activities for multiple configured users, downloads each new FIT file, rewrites the device metadata to a Garmin Edge 1030 Plus, and imports the activity into Garmin Connect without forwarding it to Strava.
The service is administered through a local-only web interface protected by a single admin password.
## 2. Scope
### In scope for v1
- Multiple independent sync users.
- Local-only admin UI.
- Admin password supplied via environment variable.
- SQLite persistence.
- Encrypted MyWhoosh and Garmin credentials at rest.
- Persistent per-user MyWhoosh and Garmin token stores.
- Periodic background sync at a configurable interval.
- Manual "sync now" actions.
- MyWhoosh activity discovery and FIT download.
- Binary FIT metadata patching to Garmin Edge 1030 Plus.
- FIT CRC validation and repair.
- Garmin Connect import via `import_activity()`.
- No intentional forwarding to Strava.
- Per-user error isolation, status and retry handling.
- Garmin MFA handling through the admin UI when required.
- Activity history and sync-run history.
### Out of scope for v1
- Public Internet exposure.
- Multi-admin or per-user web logins.
- OAuth/OIDC for the admin UI.
- Editing deployment configuration from the web UI.
- Strava integration.
- Mobile app.
- Distributed workers or external queues.
- Automatic CAPTCHA solving or browser automation for MyWhoosh login.
## 3. High-level architecture
The application runs as one Docker container with a persistent `/data` volume.
Components:
1. FastAPI application.
2. Server-rendered Jinja2 admin UI with HTMX for small interactive actions.
3. SQLite database.
4. Scheduler.
5. Sync manager.
6. Per-user MyWhoosh client.
7. FIT rewriter.
8. Per-user Garmin client/uploader.
9. Credential encryption service.
Data flow:
MyWhoosh -> download FIT -> validate FIT -> patch device metadata -> rewrite CRC -> Garmin `import_activity()` -> persist result.
## 4. Deployment configuration
Configuration is supplied through environment variables, for example:
- `ADMIN_PASSWORD`
- `SECRET_KEY`
- `CREDENTIAL_ENCRYPTION_KEY`
- `SYNC_INTERVAL_MINUTES`
- `DATABASE_URL=sqlite:////data/app.db`
- `DATA_DIR=/data`
The web server binds inside the container and is published only to the trusted local network by Docker configuration.
The web UI does not modify these deployment-level settings.
## 5. Admin authentication
The application has one admin login with no username.
- The password is read from `ADMIN_PASSWORD`.
- The password is never persisted in SQLite.
- Successful login establishes a signed session using `SECRET_KEY`.
- Authentication failures reveal no account details.
- Session cookies should be `HttpOnly` and `SameSite=Lax`.
- If TLS is later placed in front of the service, `Secure` should be enabled for the cookie.
Because the service is intended for LAN-only use, v1 does not introduce a separate identity provider.
## 6. User model
Each sync user is independent.
A user contains:
- id
- display name
- enabled flag
- health state
- encrypted MyWhoosh email/password
- encrypted Garmin email/password
- created/updated timestamps
Credentials are encrypted before being written to SQLite. The encryption key comes exclusively from `CREDENTIAL_ENCRYPTION_KEY`.
Stored passwords are never returned to the browser. When editing a user, an empty password field means "keep the existing password".
## 7. Token storage
Authentication/session tokens are separated per user.
Suggested filesystem layout:
```text
/data/
app.db
tokens/
<user-id>/
mywhoosh.json
garmin/
activities/
<user-id>/
```
The Garmin tokenstore mechanism from `python-garminconnect` should be reused rather than reimplemented.
The MyWhoosh token cache stores the access token and, where usable, the refresh token and associated account metadata.
No user may read or reuse another user's tokenstore.
## 8. MyWhoosh authentication
The service should follow the same direct API-login pattern used by `jdelrue/mywhoosh2garmin` rather than automating the MyWhoosh web login page.
The intended flow is:
1. Load cached per-user token.
2. Attempt an authenticated activities request.
3. If accepted, continue.
4. If unauthorized, attempt API login using the configured MyWhoosh credentials and the Android-style login payload used by the reference project.
5. Persist fresh token data.
6. Retry the operation once.
The implementation must not attempt to defeat or automate CAPTCHA/reCAPTCHA challenges.
The MyWhoosh endpoints are not treated as a stable public API. Changes in these endpoints should surface as a clear `action_required`/authentication or integration failure rather than causing uncontrolled retries.
## 9. MyWhoosh activity discovery
For each enabled user, the MyWhoosh client retrieves recent activities using the authenticated bearer token.
Each MyWhoosh activity must have a stable external activity identifier. The pair `(user_id, mywhoosh_activity_id)` is unique in SQLite.
This makes discovery idempotent: the same activity may be returned on every scheduler run but is processed only once unless it previously failed at a retryable stage.
The client is responsible only for:
- authentication,
- listing activities,
- normalizing metadata,
- downloading the original FIT file.
It has no knowledge of Garmin or FIT rewriting.
## 10. FIT rewriting
The FIT rewriter follows the binary-patching approach from the existing working Python implementation rather than fully decoding and re-encoding the activity.
### 10.1 Device identity
Target device:
- manufacturer: Garmin (`1`)
- product: Edge 1030 Plus (`3570`)
- product name: `Edge 1030 Plus`
- serial number: optional; if absent, the original serial field is left unchanged unless a later compatibility requirement proves otherwise
### 10.2 Patched messages
`file_id` fields when present:
- manufacturer
- product
- optional serial number
- product name
`device_info` fields are patched only for the creator device (`device_index == 0`) when a usable device index is present.
Other sensor/device records should remain unchanged so a trainer, HR sensor or power meter does not become an Edge 1030 Plus accidentally.
If the source FIT lacks enough information to identify creator-specific `device_info` safely, `file_id` remains mandatory and `device_info` patching should be conservative rather than rewriting all device messages.
### 10.3 Binary preservation
The rewriter must preserve all bytes not deliberately changed, except FIT CRC fields.
It must support:
- 12-byte and 14-byte FIT headers,
- little- and big-endian definition architectures,
- compressed timestamp records,
- developer fields,
- changing local message definitions.
### 10.4 Validation
Before patching:
- validate `.FIT` signature,
- validate declared length,
- validate header CRC when present,
- validate file CRC.
After patching:
- rewrite header CRC when present,
- rewrite file CRC,
- validate the output again,
- verify expected target metadata is readable.
Invalid FIT input is a non-retryable activity error unless the original file is later replaced/redownloaded.
## 11. Garmin import
The existing Garmin uploader pattern is reused with `python-garminconnect`.
The final activity is sent using `import_activity()` rather than `upload_activity()` because the desired behavior is to import into Garmin Connect without intentional onward synchronization to Strava.
Per user:
1. Reuse Garmin tokenstore where possible.
2. Login/refresh when required.
3. Call `import_activity()` with the converted FIT path.
4. Record the returned Garmin activity/import identifier if available.
5. Treat known duplicate responses as completed `duplicate`, not as fatal failures.
## 12. Garmin MFA
MFA is modeled as an explicit user state.
If Garmin requires MFA and there is no one-time code available:
- the user's health becomes `action_required`,
- that user's Garmin import attempts pause,
- other users continue syncing normally,
- the dashboard shows that MFA is required.
The admin can submit the one-time MFA code through the local UI.
The code:
- is used only for that login attempt,
- is never written to SQLite,
- is never written to logs,
- is discarded immediately after use.
On successful authentication the Garmin tokenstore is persisted and the user returns to normal sync behavior.
## 13. Activity state machine
An activity progresses through durable stages:
- `discovered`
- `downloaded`
- `converted`
- `imported`
- `duplicate`
- `failed`
Persisted activity fields include:
- internal id
- user id
- MyWhoosh activity id
- activity date/time
- activity name
- original FIT path
- converted FIT path
- current status
- Garmin activity/import id when known
- last error
- created/updated timestamps
Completed terminal states are `imported` and `duplicate`.
A failure must retain the latest successfully completed stage so a retry can resume without repeating unnecessary work.
## 14. Sync-run model
Each user sync invocation creates a sync-run record containing:
- id
- user id
- start time
- finish time
- status (`running`, `success`, `partial`, `failed`)
- discovered count
- imported count
- skipped count
- failed count
- summary error when relevant
Detailed application logs remain on stdout; SQLite stores only UI-relevant summaries.
## 15. Scheduler and concurrency
A central scheduler triggers every `SYNC_INTERVAL_MINUTES`.
On each tick:
1. Load enabled users.
2. Schedule one sync job per user.
3. Allow different users to run concurrently.
4. Enforce at most one active sync per user with a per-user lock.
Manual "sync now" uses exactly the same sync pipeline and the same lock.
If a manual request arrives while that user is already syncing, the application should return a clear "already running" result rather than start another run.
The scheduler must not block because one account is slow, broken or waiting for user action.
## 16. Retry policy
Retries depend on failure type.
### Retryable automatically
- transient network errors
- timeouts
- temporary MyWhoosh/Garmin server errors
- expired session/token after one reauthentication attempt
Within one sync run, use at most a small bounded retry (for example one retry). Further retry occurs on the next scheduler tick.
### Action required
- invalid MyWhoosh credentials
- MyWhoosh login/API behavior changed in a way that prevents authentication
- Garmin MFA required
- invalid Garmin credentials
### Non-retryable per activity
- corrupt/invalid FIT file
- unsupported FIT structure that cannot be safely patched
The admin UI can expose an explicit "retry" action for failed activities after the underlying issue is fixed.
## 17. User health state
Each user has a concise operational state:
- `healthy`
- `syncing`
- `degraded`
- `action_required`
- `disabled`
This state is derived from configuration and recent sync/authentication outcomes and is shown prominently on the dashboard.
## 18. Admin UI
### 18.1 Login
Single password field and submit action.
### 18.2 Dashboard
Shows all users with:
- name
- health state
- MyWhoosh connection state
- Garmin connection state
- last sync
- last imported activity
- primary error/action if any
- "sync now"
- "details"
- MFA action when needed
Includes "add account".
### 18.3 User create/edit
Fields:
- display name
- MyWhoosh email
- MyWhoosh password
- Garmin email
- Garmin password
- enabled flag
Actions:
- save
- test connection
Existing passwords are never rendered back to the browser.
### 18.4 User details
Shows:
- current connection and health states
- most recent sync-run summary
- recent activities and status
- latest errors
Actions:
- sync now
- retry failed activity
- enter Garmin MFA when required
### 18.5 System page
Read-only operational information:
- application version
- configured sync interval
- last scheduler tick
- next expected tick
- account count
- activity count
Action:
- sync all now
## 19. UI technology
Use:
- FastAPI
- Jinja2
- HTMX
- small application-specific CSS
Do not introduce Angular, React, Tailwind or Bootstrap for v1 unless requirements change.
HTMX is used for bounded actions such as:
- sync now
- test connection
- submit MFA
- retry activity
The application remains server-rendered and easy to operate as one container.
## 20. Security requirements
- Never log passwords, bearer tokens, session tokens, encryption keys or MFA codes.
- Encrypt stored MyWhoosh and Garmin credentials.
- Keep tokenstores under the persistent data directory with restrictive filesystem permissions where possible.
- Escape all user-visible data rendered into HTML.
- Protect state-changing web requests against CSRF.
- Validate all IDs against the authenticated admin session rather than trusting client-provided paths blindly.
- Use prepared/ORM parameterized database access.
- Do not expose decrypted credentials through API responses or templates.
## 21. Cleanup and retention
For v1, original and converted FIT files are retained because they are valuable for debugging failed imports.
Automated retention/cleanup can be added later after operating behavior is known.
## 22. Error isolation
Failure of one user must never prevent other users from syncing.
Examples:
- User A imports successfully while User B requires Garmin MFA.
- User C may have invalid MyWhoosh credentials without affecting scheduler execution for A or B.
- A corrupt activity file affects only that activity and user.
## 23. Testing strategy
### FIT rewriter tests
- valid 12-byte header FIT
- valid 14-byte header FIT
- invalid header CRC
- invalid file CRC
- malformed/truncated definitions
- developer fields preserved
- compressed timestamp records handled
- Edge 1030 Plus manufacturer/product patched correctly
- non-creator `device_info` unchanged
- output CRC valid
- bytes outside expected metadata and CRC locations unchanged
### MyWhoosh client tests
Use mocked HTTP responses for:
- valid cached token
- expired token followed by successful login
- invalid credentials
- transient server error
- activity listing
- FIT download
Do not make live MyWhoosh requests in the normal unit test suite.
### Garmin uploader tests
Use a fake/protocol-compatible Garmin client for:
- tokenstore login
- successful import
- duplicate
- MFA required
- invalid login
- transient import failure
### Sync manager tests
- new activity full happy path
- discovered activity is not duplicated
- resume from downloaded
- resume from converted
- retry after transient Garmin failure
- one user's failure does not affect another
- per-user lock prevents concurrent duplicate sync
### Web tests
- admin login success/failure
- unauthenticated routes redirect/reject
- create/edit/disable user
- password never returned
- manual sync action
- MFA submission lifecycle
- CSRF on mutating requests
## 24. Suggested module boundaries
```text
app/
main.py
config.py
auth/
admin.py
db/
models.py
session.py
repositories.py
security/
credentials.py
mywhoosh/
client.py
models.py
tokenstore.py
fit/
rewriter.py
crc.py
models.py
garmin/
uploader.py
sync/
manager.py
scheduler.py
states.py
web/
routes.py
forms.py
templates/
static/
tests/
```
Each unit should depend on explicit interfaces/protocols where external services are involved so tests do not require live accounts.
## 25. Primary design decisions
1. One local admin instead of user-facing authentication.
2. Multiple independent sync accounts.
3. SQLite for durable application state.
4. Environment variables for deployment secrets/configuration.
5. Encrypted service credentials at rest.
6. Per-user tokenstores.
7. Direct MyWhoosh API login pattern; no CAPTCHA automation.
8. Binary FIT patching rather than decode/re-encode.
9. Garmin Edge 1030 Plus product ID `3570`.
10. Conservative creator-device patching.
11. Garmin `import_activity()` for Garmin-only import behavior.
12. FastAPI + Jinja2 + HTMX for a small single-container admin UI.
13. Parallel sync across users, serialized sync within each user.
14. Durable activity stages for resumable/idempotent sync.
## 26. Acceptance criteria for v1
The system is ready for v1 when:
1. It runs from Docker with persistent `/data` storage.
2. The admin can log in locally using the environment-configured password.
3. The admin can add at least two independent users.
4. Each user can authenticate independently to MyWhoosh and Garmin.
5. New MyWhoosh activities are discovered automatically on schedule.
6. FIT files are downloaded and patched to Garmin Edge 1030 Plus metadata with valid CRCs.
7. Converted activities are imported into the corresponding Garmin Connect account using `import_activity()`.
8. Already processed activities are not imported again.
9. Garmin MFA for one user can be resolved through the UI and does not block other users.
10. A failure for one user or one activity does not stop the scheduler.
11. The dashboard shows current state, recent syncs and actionable errors.
12. Secrets and MFA codes do not appear in logs or browser responses.