plans + specs
This commit is contained in:
577
docs/superpowers/plans/2026-08-15-fit-rewriter.md
Normal file
577
docs/superpowers/plans/2026-08-15-fit-rewriter.md
Normal file
@@ -0,0 +1,577 @@
|
||||
# FIT Edge 1030 Plus Rewriter Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Implement a binary-preserving FIT metadata rewriter that changes only Garmin creator-device metadata plus required CRC bytes, targeting Garmin Edge 1030 Plus product ID `3570`.
|
||||
|
||||
**Architecture:** Parse FIT definition/data records only far enough to locate `file_id` and creator `device_info` fields, patch bytes in place, recalculate header/file CRC, and validate the output. Do not fully decode/re-encode the activity, so unknown/developer fields and all non-target bytes remain untouched.
|
||||
|
||||
**Tech Stack:** Python 3.12 standard library (`struct`, `dataclasses`, `pathlib`), pytest.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Garmin manufacturer ID is `1`.
|
||||
- Garmin Edge 1030 Plus product ID is `3570`.
|
||||
- Product name is `Edge 1030 Plus`.
|
||||
- Support FIT header sizes 12 and 14.
|
||||
- Validate declared data size and `.FIT` signature.
|
||||
- Validate header CRC when a 14-byte header is present.
|
||||
- Validate file CRC before and after modification.
|
||||
- Support little- and big-endian definition architectures, compressed timestamp records, developer fields, and changing local message definitions.
|
||||
- Patch `file_id` device fields when present.
|
||||
- Patch `device_info` only when it is safely identified as the creator (`device_index == 0`).
|
||||
- If creator-specific `device_info` cannot be identified safely, leave it unchanged rather than rewriting all device records.
|
||||
- Preserve every non-target byte except CRC fields.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```text
|
||||
app/fit/
|
||||
__init__.py
|
||||
crc.py
|
||||
models.py
|
||||
rewriter.py
|
||||
tests/fit/
|
||||
builders.py
|
||||
test_crc.py
|
||||
test_rewriter_validation.py
|
||||
test_rewriter_patching.py
|
||||
test_rewriter_preservation.py
|
||||
```
|
||||
|
||||
## Task 1: Implement Garmin FIT CRC
|
||||
|
||||
**Files:**
|
||||
- Create: `app/fit/crc.py`
|
||||
- Create: `tests/fit/test_crc.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `fit_crc(data: bytes | bytearray | memoryview) -> int`.
|
||||
|
||||
- [ ] **Step 1: Write failing CRC vector tests**
|
||||
|
||||
```python
|
||||
# tests/fit/test_crc.py
|
||||
from app.fit.crc import fit_crc
|
||||
|
||||
|
||||
def test_empty_crc_is_zero() -> None:
|
||||
assert fit_crc(b"") == 0
|
||||
|
||||
|
||||
def test_crc_is_incremental_equivalent() -> None:
|
||||
payload = b".FIT-device-metadata"
|
||||
assert fit_crc(payload) == fit_crc(memoryview(payload))
|
||||
assert 0 <= fit_crc(payload) <= 0xFFFF
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/fit/test_crc.py -v`
|
||||
|
||||
Expected: import failure.
|
||||
|
||||
- [ ] **Step 3: Implement the FIT nibble-table CRC algorithm**
|
||||
|
||||
```python
|
||||
# app/fit/crc.py
|
||||
CRC_TABLE = (
|
||||
0x0000, 0xCC01, 0xD801, 0x1400,
|
||||
0xF001, 0x3C00, 0x2800, 0xE401,
|
||||
0xA001, 0x6C00, 0x7800, 0xB401,
|
||||
0x5000, 0x9C01, 0x8801, 0x4400,
|
||||
)
|
||||
|
||||
|
||||
def fit_crc(data: bytes | bytearray | memoryview) -> int:
|
||||
crc = 0
|
||||
for byte in data:
|
||||
tmp = CRC_TABLE[crc & 0xF]
|
||||
crc = (crc >> 4) & 0x0FFF
|
||||
crc ^= tmp ^ CRC_TABLE[byte & 0xF]
|
||||
tmp = CRC_TABLE[crc & 0xF]
|
||||
crc = (crc >> 4) & 0x0FFF
|
||||
crc ^= tmp ^ CRC_TABLE[(byte >> 4) & 0xF]
|
||||
return crc & 0xFFFF
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests**
|
||||
|
||||
Run: `pytest tests/fit/test_crc.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add app/fit/crc.py tests/fit/test_crc.py
|
||||
git commit -m "feat: add FIT CRC calculation"
|
||||
```
|
||||
|
||||
## Task 2: Parse and validate the FIT container safely
|
||||
|
||||
**Files:**
|
||||
- Create: `app/fit/models.py`
|
||||
- Create: `app/fit/rewriter.py`
|
||||
- Create: `tests/fit/builders.py`
|
||||
- Create: `tests/fit/test_rewriter_validation.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `FitFormatError`, `FieldDefinition`, `LocalDefinition`, `_validate_fit_container()`, `_iter_data_fields()`.
|
||||
- `_iter_data_fields(data)` returns data records with exact field offsets without decoding unrelated values.
|
||||
|
||||
- [ ] **Step 1: Create a deterministic minimal FIT fixture builder**
|
||||
|
||||
```python
|
||||
# tests/fit/builders.py
|
||||
import struct
|
||||
|
||||
from app.fit.crc import fit_crc
|
||||
|
||||
|
||||
def make_fit(data_records: bytes, *, header_size: int = 14) -> bytes:
|
||||
if header_size not in {12, 14}:
|
||||
raise ValueError(header_size)
|
||||
header = bytearray(header_size)
|
||||
header[0] = header_size
|
||||
header[1] = 0x20
|
||||
struct.pack_into("<H", header, 2, 0x0100)
|
||||
struct.pack_into("<I", header, 4, len(data_records))
|
||||
header[8:12] = b".FIT"
|
||||
if header_size == 14:
|
||||
struct.pack_into("<H", header, 12, fit_crc(header[:12]))
|
||||
body = header + data_records
|
||||
return bytes(body + struct.pack("<H", fit_crc(body)))
|
||||
|
||||
|
||||
def definition(local: int, global_num: int, fields: list[tuple[int, int, int]], *, endian: str = "<") -> bytes:
|
||||
architecture = 1 if endian == ">" else 0
|
||||
payload = bytearray([0, architecture])
|
||||
payload.extend(struct.pack(f"{endian}H", global_num))
|
||||
payload.append(len(fields))
|
||||
for num, size, base_type in fields:
|
||||
payload.extend(bytes([num, size, base_type]))
|
||||
return bytes([0x40 | local]) + bytes(payload)
|
||||
|
||||
|
||||
def data(local: int, payload: bytes) -> bytes:
|
||||
return bytes([local]) + payload
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write validation tests**
|
||||
|
||||
```python
|
||||
# tests/fit/test_rewriter_validation.py
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.fit.rewriter import FitFormatError, is_fit_file
|
||||
from tests.fit.builders import make_fit
|
||||
|
||||
|
||||
def test_valid_12_and_14_byte_headers(tmp_path: Path) -> None:
|
||||
for size in (12, 14):
|
||||
path = tmp_path / f"valid-{size}.fit"
|
||||
path.write_bytes(make_fit(b"", header_size=size))
|
||||
assert is_fit_file(path) is True
|
||||
|
||||
|
||||
def test_bad_file_crc_is_rejected(tmp_path: Path) -> None:
|
||||
payload = bytearray(make_fit(b""))
|
||||
payload[-1] ^= 0xFF
|
||||
path = tmp_path / "bad.fit"
|
||||
path.write_bytes(payload)
|
||||
assert is_fit_file(path) is False
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/fit/test_rewriter_validation.py -v`
|
||||
|
||||
Expected: import failure.
|
||||
|
||||
- [ ] **Step 4: Implement the parser structures and validation**
|
||||
|
||||
```python
|
||||
# app/fit/models.py
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FieldDefinition:
|
||||
num: int
|
||||
size: int
|
||||
base_type: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LocalDefinition:
|
||||
global_message_num: int
|
||||
endian: str
|
||||
fields: tuple[FieldDefinition, ...]
|
||||
developer_field_size: int
|
||||
```
|
||||
|
||||
Implement in `app/fit/rewriter.py` the validated logic from the known working implementation:
|
||||
|
||||
```python
|
||||
class FitFormatError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _validate_fit_container(data: bytearray) -> None:
|
||||
if len(data) < 14:
|
||||
raise FitFormatError("FIT file is too small")
|
||||
header_size = data[0]
|
||||
if header_size not in {12, 14}:
|
||||
raise FitFormatError(f"Unsupported FIT header size: {header_size}")
|
||||
if len(data) < header_size + 2:
|
||||
raise FitFormatError("FIT file is shorter than its header")
|
||||
if bytes(data[8:12]) != b".FIT":
|
||||
raise FitFormatError("Missing .FIT signature")
|
||||
data_size = struct.unpack_from("<I", data, 4)[0]
|
||||
expected_size = header_size + data_size + 2
|
||||
if len(data) != expected_size:
|
||||
raise FitFormatError(f"FIT size mismatch: header says {expected_size} bytes, file has {len(data)}")
|
||||
if header_size == 14:
|
||||
expected_header_crc = struct.unpack_from("<H", data, 12)[0]
|
||||
if expected_header_crc != fit_crc(data[:12]):
|
||||
raise FitFormatError("FIT header CRC check failed")
|
||||
expected_file_crc = struct.unpack_from("<H", data, len(data) - 2)[0]
|
||||
if expected_file_crc != fit_crc(data[:-2]):
|
||||
raise FitFormatError("FIT file CRC check failed")
|
||||
```
|
||||
|
||||
Implement the parser helpers explicitly:
|
||||
|
||||
```python
|
||||
def _read_definition(
|
||||
data: bytearray,
|
||||
offset: int,
|
||||
has_developer_fields: bool,
|
||||
end_offset: int,
|
||||
) -> tuple[LocalDefinition, int]:
|
||||
if offset + 5 > end_offset:
|
||||
raise FitFormatError("Truncated FIT definition message")
|
||||
offset += 1 # reserved byte
|
||||
architecture = data[offset]
|
||||
offset += 1
|
||||
if architecture not in {0, 1}:
|
||||
raise FitFormatError(f"Unsupported FIT architecture: {architecture}")
|
||||
endian = ">" if architecture == 1 else "<"
|
||||
global_message_num = struct.unpack_from(f"{endian}H", data, offset)[0]
|
||||
offset += 2
|
||||
field_count = data[offset]
|
||||
offset += 1
|
||||
|
||||
fields: list[FieldDefinition] = []
|
||||
for _ in range(field_count):
|
||||
if offset + 3 > end_offset:
|
||||
raise FitFormatError("Truncated FIT field definition")
|
||||
fields.append(FieldDefinition(data[offset], data[offset + 1], data[offset + 2]))
|
||||
offset += 3
|
||||
|
||||
developer_field_size = 0
|
||||
if has_developer_fields:
|
||||
if offset >= end_offset:
|
||||
raise FitFormatError("Truncated FIT developer field count")
|
||||
developer_count = data[offset]
|
||||
offset += 1
|
||||
for _ in range(developer_count):
|
||||
if offset + 3 > end_offset:
|
||||
raise FitFormatError("Truncated FIT developer field definition")
|
||||
developer_field_size += data[offset + 1]
|
||||
offset += 3
|
||||
|
||||
return LocalDefinition(global_message_num, endian, tuple(fields), developer_field_size), offset
|
||||
|
||||
|
||||
def _collect_field_offsets(
|
||||
definition: LocalDefinition,
|
||||
offset: int,
|
||||
end_offset: int,
|
||||
) -> tuple[list[tuple[FieldDefinition, int]], int]:
|
||||
result: list[tuple[FieldDefinition, int]] = []
|
||||
current = offset
|
||||
for field in definition.fields:
|
||||
if current + field.size > end_offset:
|
||||
raise FitFormatError("Truncated FIT data record")
|
||||
result.append((field, current))
|
||||
current += field.size
|
||||
if current + definition.developer_field_size > end_offset:
|
||||
raise FitFormatError("Truncated FIT developer field payload")
|
||||
current += definition.developer_field_size
|
||||
return result, current
|
||||
|
||||
|
||||
def _iter_data_fields(
|
||||
data: bytearray,
|
||||
) -> list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]]:
|
||||
header_size = data[0]
|
||||
data_size = struct.unpack_from("<I", data, 4)[0]
|
||||
offset = header_size
|
||||
end_offset = header_size + data_size
|
||||
definitions: dict[int, LocalDefinition] = {}
|
||||
records: list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]] = []
|
||||
|
||||
while offset < end_offset:
|
||||
record_header = data[offset]
|
||||
offset += 1
|
||||
|
||||
if record_header & 0x80:
|
||||
local = (record_header >> 5) & 0x03
|
||||
definition = definitions.get(local)
|
||||
if definition is None:
|
||||
raise FitFormatError(f"Compressed timestamp record used unknown local definition {local}")
|
||||
field_offsets, offset = _collect_field_offsets(definition, offset, end_offset)
|
||||
records.append((definition, field_offsets))
|
||||
continue
|
||||
|
||||
local = record_header & 0x0F
|
||||
is_definition = bool(record_header & 0x40)
|
||||
has_developer_fields = bool(record_header & 0x20)
|
||||
if is_definition:
|
||||
definition, offset = _read_definition(data, offset, has_developer_fields, end_offset)
|
||||
definitions[local] = definition
|
||||
continue
|
||||
|
||||
definition = definitions.get(local)
|
||||
if definition is None:
|
||||
raise FitFormatError(f"Data record used unknown local definition {local}")
|
||||
field_offsets, offset = _collect_field_offsets(definition, offset, end_offset)
|
||||
records.append((definition, field_offsets))
|
||||
|
||||
if offset != end_offset:
|
||||
raise FitFormatError("FIT parser did not end on data boundary")
|
||||
return records
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Expose `is_fit_file` and verify parser boundary failures**
|
||||
|
||||
```python
|
||||
def is_fit_file(path: Path) -> bool:
|
||||
try:
|
||||
data = bytearray(path.read_bytes())
|
||||
_validate_fit_container(data)
|
||||
_iter_data_fields(data)
|
||||
except (OSError, FitFormatError):
|
||||
return False
|
||||
return True
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run validation tests**
|
||||
|
||||
Run: `pytest tests/fit/test_rewriter_validation.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add app/fit/models.py app/fit/rewriter.py tests/fit/builders.py tests/fit/test_rewriter_validation.py
|
||||
git commit -m "feat: parse and validate FIT containers"
|
||||
```
|
||||
|
||||
## Task 3: Patch `file_id` and only creator `device_info`
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/fit/models.py`
|
||||
- Modify: `app/fit/rewriter.py`
|
||||
- Create: `tests/fit/test_rewriter_patching.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `GarminDevice`, `FitConversionResult`, `DeviceFieldValue`, `convert_fit_device()`, `read_device_field_values()`.
|
||||
- Default device is Garmin manufacturer `1`, Edge 1030 Plus product `3570`, name `Edge 1030 Plus`.
|
||||
|
||||
- [ ] **Step 1: Add model types**
|
||||
|
||||
```python
|
||||
# app/fit/models.py additions
|
||||
from pathlib import Path
|
||||
|
||||
GARMIN_MANUFACTURER_ID = 1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GarminDevice:
|
||||
manufacturer_id: int = GARMIN_MANUFACTURER_ID
|
||||
product_id: int = 3570
|
||||
product_name: str = "Edge 1030 Plus"
|
||||
serial_number: int | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FitConversionResult:
|
||||
source_path: Path
|
||||
output_path: Path
|
||||
patched_field_count: int
|
||||
header_crc: int | None
|
||||
file_crc: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DeviceFieldValue:
|
||||
global_message_num: int
|
||||
field_num: int
|
||||
value: int | str
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write a fixture containing one creator and one sensor `device_info`**
|
||||
|
||||
Use these FIT field numbers in `tests/fit/test_rewriter_patching.py`:
|
||||
|
||||
```python
|
||||
FILE_ID_MESG_NUM = 0
|
||||
DEVICE_INFO_MESG_NUM = 23
|
||||
|
||||
# file_id: manufacturer(1/u16), product(2/u16)
|
||||
file_def = definition(0, FILE_ID_MESG_NUM, [(1, 2, 0x84), (2, 2, 0x84)])
|
||||
file_data = data(0, struct.pack("<HH", 255, 999))
|
||||
|
||||
# device_info: device_index(0/u8), manufacturer(2/u16), product(4/u16)
|
||||
device_def = definition(1, DEVICE_INFO_MESG_NUM, [(0, 1, 0x02), (2, 2, 0x84), (4, 2, 0x84)])
|
||||
creator = data(1, struct.pack("<BHH", 0, 255, 999))
|
||||
sensor = data(1, struct.pack("<BHH", 1, 32, 1234))
|
||||
```
|
||||
|
||||
Write assertions that `file_id` and creator become `(1, 3570)` while sensor remains `(32, 1234)`.
|
||||
|
||||
- [ ] **Step 3: Run the test and verify failure**
|
||||
|
||||
Run: `pytest tests/fit/test_rewriter_patching.py -v`
|
||||
|
||||
Expected: missing conversion functions.
|
||||
|
||||
- [ ] **Step 4: Implement conservative creator detection**
|
||||
|
||||
In `_patch_device_metadata`, process each `device_info` record as a unit:
|
||||
|
||||
```python
|
||||
field_map = {field.num: (field, offset) for field, offset in field_offsets}
|
||||
if definition.global_message_num == DEVICE_INFO_MESG_NUM:
|
||||
device_index_entry = field_map.get(0)
|
||||
if device_index_entry is None:
|
||||
continue
|
||||
index_field, index_offset = device_index_entry
|
||||
device_index = _read_field_value(data, index_offset, index_field, definition.endian)
|
||||
if device_index != 0:
|
||||
continue
|
||||
```
|
||||
|
||||
Only after this check may fields `2`, `3`, `4`, and `27` be patched. Do not patch a `device_info` message that lacks field `0`.
|
||||
|
||||
Always patch eligible `file_id` fields `1`, `2`, optional `3`, and `8`.
|
||||
|
||||
- [ ] **Step 5: Implement numeric/string read-write helpers and conversion entry point**
|
||||
|
||||
Use the supplied working `_read_field_value()` and `_write_field_value()` behavior, then expose:
|
||||
|
||||
```python
|
||||
def convert_fit_device(source_path: Path, output_path: Path, device: GarminDevice | None = None) -> FitConversionResult:
|
||||
resolved = device or GarminDevice()
|
||||
data = bytearray(source_path.read_bytes())
|
||||
_validate_fit_container(data)
|
||||
patched_count = _patch_device_metadata(data, resolved)
|
||||
header_crc = _rewrite_header_crc(data)
|
||||
file_crc = _rewrite_file_crc(data)
|
||||
_validate_fit_container(data)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(data)
|
||||
return FitConversionResult(source_path, output_path, patched_count, header_crc, file_crc)
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run patching tests**
|
||||
|
||||
Run: `pytest tests/fit/test_rewriter_patching.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add app/fit tests/fit/test_rewriter_patching.py
|
||||
git commit -m "feat: patch FIT creator as Edge 1030 Plus"
|
||||
```
|
||||
|
||||
## Task 4: Prove binary preservation and advanced record support
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/fit/test_rewriter_preservation.py`
|
||||
- Modify: `tests/fit/builders.py`
|
||||
- Modify: `app/fit/rewriter.py` only if a test exposes an actual parser defect
|
||||
|
||||
**Interfaces:**
|
||||
- No new public interface; this task hardens `convert_fit_device()`.
|
||||
|
||||
- [ ] **Step 1: Write a preservation test that records changed byte positions**
|
||||
|
||||
```python
|
||||
# tests/fit/test_rewriter_preservation.py
|
||||
from pathlib import Path
|
||||
|
||||
from app.fit.rewriter import convert_fit_device
|
||||
|
||||
|
||||
def test_only_target_fields_and_crcs_change(tmp_path: Path, complex_fit_bytes: bytes) -> None:
|
||||
source = tmp_path / "source.fit"
|
||||
output = tmp_path / "output.fit"
|
||||
source.write_bytes(complex_fit_bytes)
|
||||
|
||||
convert_fit_device(source, output)
|
||||
|
||||
before = source.read_bytes()
|
||||
after = output.read_bytes()
|
||||
assert len(before) == len(after)
|
||||
|
||||
changed = {index for index, (a, b) in enumerate(zip(before, after)) if a != b}
|
||||
expected_metadata_offsets = set(find_expected_device_metadata_offsets(before))
|
||||
crc_offsets = {12, 13, len(before) - 2, len(before) - 1}
|
||||
assert changed <= expected_metadata_offsets | crc_offsets
|
||||
```
|
||||
|
||||
The fixture helper `find_expected_device_metadata_offsets()` must use the test fixture's known construction offsets, not production parser code, so the test is independent.
|
||||
|
||||
- [ ] **Step 2: Add fixtures for compressed timestamps, developer fields, and local-definition replacement**
|
||||
|
||||
Construct one synthetic FIT data section containing:
|
||||
|
||||
1. a normal `file_id` definition/data pair;
|
||||
2. a `device_info` definition with creator record;
|
||||
3. a record definition with one developer field and one data record;
|
||||
4. a compressed-timestamp data header referring to a known local definition;
|
||||
5. a later replacement definition for the same local message number.
|
||||
|
||||
The test passes if conversion completes, output CRC validates, and non-target payload bytes remain identical.
|
||||
|
||||
- [ ] **Step 3: Run the focused advanced tests**
|
||||
|
||||
Run: `pytest tests/fit/test_rewriter_preservation.py -v`
|
||||
|
||||
Expected: PASS. If a parser guard fails, fix only the smallest parser defect required by the test.
|
||||
|
||||
- [ ] **Step 4: Add rejection tests for malformed/truncated definitions**
|
||||
|
||||
```python
|
||||
def test_truncated_definition_is_non_recoverable(tmp_path: Path) -> None:
|
||||
path = tmp_path / "truncated.fit"
|
||||
path.write_bytes(make_fit(bytes([0x40, 0x00, 0x00])))
|
||||
assert is_fit_file(path) is False
|
||||
```
|
||||
|
||||
Also assert that `convert_fit_device()` raises `FitFormatError` for the same input.
|
||||
|
||||
- [ ] **Step 5: Run the entire FIT suite**
|
||||
|
||||
Run: `pytest tests/fit -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/fit app/fit/rewriter.py
|
||||
git commit -m "test: verify FIT binary preservation"
|
||||
```
|
||||
1087
docs/superpowers/plans/2026-08-15-foundation-admin-data.md
Normal file
1087
docs/superpowers/plans/2026-08-15-foundation-admin-data.md
Normal file
File diff suppressed because it is too large
Load Diff
716
docs/superpowers/plans/2026-08-15-service-clients.md
Normal file
716
docs/superpowers/plans/2026-08-15-service-clients.md
Normal file
@@ -0,0 +1,716 @@
|
||||
# MyWhoosh and Garmin Service Clients Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Implement isolated per-user MyWhoosh and Garmin clients with persistent tokenstores, mockable network boundaries, MyWhoosh direct API login/activity download, Garmin `import_activity()`, duplicate handling, and MFA signaling.
|
||||
|
||||
**Architecture:** Keep external integrations behind narrow protocols so the sync engine never depends directly on `httpx` or `garminconnect`. MyWhoosh uses an injected `httpx.AsyncClient` and a per-user JSON tokenstore. Garmin uses an injected client factory and per-user `python-garminconnect` token directory; blocking Garmin calls are later run through `asyncio.to_thread` by the sync layer.
|
||||
|
||||
**Tech Stack:** Python 3.12, httpx, python-garminconnect, pytest, pytest-asyncio.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Do not automate or defeat MyWhoosh CAPTCHA/reCAPTCHA.
|
||||
- Follow the direct Android-style API login flow used by the reference `jdelrue/mywhoosh2garmin` project.
|
||||
- Cache MyWhoosh tokens per user under `/data/tokens/<user-id>/mywhoosh.json`.
|
||||
- Cache Garmin tokens per user under `/data/tokens/<user-id>/garmin/` using the library's tokenstore behavior.
|
||||
- Authentication/API changes must become explicit integration/authentication errors, not uncontrolled retries.
|
||||
- Garmin activity transfer must use `import_activity()`, not `upload_activity()`.
|
||||
- Known duplicate Garmin responses are successful terminal outcomes.
|
||||
- Garmin MFA must raise a dedicated exception so the UI can collect a one-time code.
|
||||
- Never log passwords, bearer tokens, Garmin tokens, or MFA codes.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```text
|
||||
app/mywhoosh/
|
||||
__init__.py
|
||||
models.py
|
||||
tokenstore.py
|
||||
client.py
|
||||
app/garmin/
|
||||
__init__.py
|
||||
uploader.py
|
||||
tests/mywhoosh/
|
||||
test_tokenstore.py
|
||||
test_client_auth.py
|
||||
test_client_activities.py
|
||||
tests/garmin/
|
||||
test_uploader.py
|
||||
```
|
||||
|
||||
## Task 1: Implement MyWhoosh models and tokenstore
|
||||
|
||||
**Files:**
|
||||
- Create: `app/mywhoosh/models.py`
|
||||
- Create: `app/mywhoosh/tokenstore.py`
|
||||
- Create: `tests/mywhoosh/test_tokenstore.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces `MyWhooshToken`, `MyWhooshActivity`, `MyWhooshTokenStore.load()`, `save()`, `clear()`.
|
||||
|
||||
- [ ] **Step 1: Write tokenstore tests**
|
||||
|
||||
```python
|
||||
# tests/mywhoosh/test_tokenstore.py
|
||||
from pathlib import Path
|
||||
|
||||
from app.mywhoosh.models import MyWhooshToken
|
||||
from app.mywhoosh.tokenstore import MyWhooshTokenStore
|
||||
|
||||
|
||||
def test_tokenstore_round_trip_and_permissions(tmp_path: Path) -> None:
|
||||
store = MyWhooshTokenStore(tmp_path / "tokens" / "7" / "mywhoosh.json")
|
||||
token = MyWhooshToken(access_token="access", refresh_token="refresh", whoosh_id="whoosh-7")
|
||||
store.save(token)
|
||||
|
||||
assert store.load() == token
|
||||
assert oct(store.path.stat().st_mode & 0o777) == "0o600"
|
||||
|
||||
|
||||
def test_missing_token_returns_none(tmp_path: Path) -> None:
|
||||
store = MyWhooshTokenStore(tmp_path / "missing.json")
|
||||
assert store.load() is None
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/mywhoosh/test_tokenstore.py -v`
|
||||
|
||||
Expected: import failure.
|
||||
|
||||
- [ ] **Step 3: Implement models**
|
||||
|
||||
```python
|
||||
# app/mywhoosh/models.py
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MyWhooshToken:
|
||||
access_token: str
|
||||
refresh_token: str | None
|
||||
whoosh_id: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MyWhooshActivity:
|
||||
id: str
|
||||
title: str
|
||||
activity_file_id: str
|
||||
started_at: datetime | None
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Implement atomic JSON token persistence**
|
||||
|
||||
```python
|
||||
# app/mywhoosh/tokenstore.py
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from app.mywhoosh.models import MyWhooshToken
|
||||
|
||||
|
||||
class MyWhooshTokenStore:
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = path
|
||||
|
||||
def load(self) -> MyWhooshToken | None:
|
||||
try:
|
||||
raw = json.loads(self.path.read_text("utf-8"))
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
return MyWhooshToken(
|
||||
access_token=raw["access_token"],
|
||||
refresh_token=raw.get("refresh_token"),
|
||||
whoosh_id=raw.get("whoosh_id"),
|
||||
)
|
||||
|
||||
def save(self, token: MyWhooshToken) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
tmp = self.path.with_suffix(".tmp")
|
||||
tmp.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"access_token": token.access_token,
|
||||
"refresh_token": token.refresh_token,
|
||||
"whoosh_id": token.whoosh_id,
|
||||
},
|
||||
indent=2,
|
||||
),
|
||||
"utf-8",
|
||||
)
|
||||
os.chmod(tmp, 0o600)
|
||||
tmp.replace(self.path)
|
||||
os.chmod(self.path, 0o600)
|
||||
|
||||
def clear(self) -> None:
|
||||
self.path.unlink(missing_ok=True)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run tests**
|
||||
|
||||
Run: `pytest tests/mywhoosh/test_tokenstore.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add app/mywhoosh/models.py app/mywhoosh/tokenstore.py tests/mywhoosh/test_tokenstore.py
|
||||
git commit -m "feat: add MyWhoosh token persistence"
|
||||
```
|
||||
|
||||
## Task 2: Implement MyWhoosh login and cached-session recovery
|
||||
|
||||
**Files:**
|
||||
- Create: `app/mywhoosh/client.py`
|
||||
- Create: `tests/mywhoosh/test_client_auth.py`
|
||||
- Modify: `pyproject.toml`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces exceptions `MyWhooshAuthError`, `MyWhooshTransientError`, `MyWhooshIntegrationError`.
|
||||
- Produces `MyWhooshClient.ensure_authenticated(email: str, password: str) -> None`.
|
||||
- Login endpoint: `https://services.mywhoosh.com/http-service/api/login`.
|
||||
- Login payload fields: `Username`, `Password`, `Platform="Android"`, `Action=1001`, random `CorrelationId`, random `DeviceId`, `Authorization=""`.
|
||||
|
||||
- [ ] **Step 1: Add test dependencies**
|
||||
|
||||
Add to `pyproject.toml` runtime dependencies:
|
||||
|
||||
```toml
|
||||
"httpx>=0.27,<1",
|
||||
```
|
||||
|
||||
and test dependencies:
|
||||
|
||||
```toml
|
||||
"pytest-asyncio>=0.24,<1",
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write authentication tests using `httpx.MockTransport`**
|
||||
|
||||
```python
|
||||
# tests/mywhoosh/test_client_auth.py
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.mywhoosh.client import MyWhooshClient, MyWhooshAuthError
|
||||
from app.mywhoosh.models import MyWhooshToken
|
||||
from app.mywhoosh.tokenstore import MyWhooshTokenStore
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_login_saves_access_refresh_and_whoosh_id(tmp_path) -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
assert request.url.path == "/http-service/api/login"
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"Success": True,
|
||||
"AccessToken": "new-access",
|
||||
"RefreshToken": "new-refresh",
|
||||
"WhooshId": "w-1",
|
||||
},
|
||||
)
|
||||
|
||||
store = MyWhooshTokenStore(tmp_path / "mywhoosh.json")
|
||||
client = MyWhooshClient(store, http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)))
|
||||
await client.login("rider@example.com", "secret")
|
||||
assert store.load() == MyWhooshToken("new-access", "new-refresh", "w-1")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_credentials_raise_auth_error(tmp_path) -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json={"Success": False, "Message": "Invalid credentials"})
|
||||
|
||||
client = MyWhooshClient(
|
||||
MyWhooshTokenStore(tmp_path / "mywhoosh.json"),
|
||||
http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
with pytest.raises(MyWhooshAuthError):
|
||||
await client.login("rider@example.com", "bad")
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/mywhoosh/test_client_auth.py -v`
|
||||
|
||||
Expected: missing client implementation.
|
||||
|
||||
- [ ] **Step 4: Implement exception taxonomy and login**
|
||||
|
||||
```python
|
||||
# app/mywhoosh/client.py
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
import httpx
|
||||
|
||||
from app.mywhoosh.models import MyWhooshToken
|
||||
from app.mywhoosh.tokenstore import MyWhooshTokenStore
|
||||
|
||||
LOGIN_URL = "https://services.mywhoosh.com/http-service/api/login"
|
||||
ACTIVITIES_BASE = "https://service14.mywhoosh.com/v2/"
|
||||
|
||||
|
||||
class MyWhooshError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class MyWhooshAuthError(MyWhooshError):
|
||||
pass
|
||||
|
||||
|
||||
class MyWhooshTransientError(MyWhooshError):
|
||||
pass
|
||||
|
||||
|
||||
class MyWhooshIntegrationError(MyWhooshError):
|
||||
pass
|
||||
|
||||
|
||||
class MyWhooshClient:
|
||||
def __init__(self, token_store: MyWhooshTokenStore, http_client: httpx.AsyncClient | None = None) -> None:
|
||||
self.token_store = token_store
|
||||
self.http = http_client or httpx.AsyncClient(timeout=30.0)
|
||||
self.token = token_store.load()
|
||||
|
||||
async def login(self, email: str, password: str) -> None:
|
||||
payload = {
|
||||
"Username": email,
|
||||
"Password": password,
|
||||
"Platform": "Android",
|
||||
"Action": 1001,
|
||||
"CorrelationId": str(uuid.uuid4()),
|
||||
"DeviceId": str(uuid.uuid4()),
|
||||
"Authorization": "",
|
||||
}
|
||||
try:
|
||||
response = await self.http.post(LOGIN_URL, json=payload)
|
||||
except httpx.TransportError as exc:
|
||||
raise MyWhooshTransientError("MyWhoosh login request failed") from exc
|
||||
if response.status_code >= 500:
|
||||
raise MyWhooshTransientError(f"MyWhoosh login returned HTTP {response.status_code}")
|
||||
if response.status_code >= 400:
|
||||
raise MyWhooshAuthError(f"MyWhoosh login returned HTTP {response.status_code}")
|
||||
try:
|
||||
body = response.json()
|
||||
except ValueError as exc:
|
||||
raise MyWhooshIntegrationError("MyWhoosh login returned invalid JSON") from exc
|
||||
if body.get("Success") is not True or not body.get("AccessToken"):
|
||||
raise MyWhooshAuthError(str(body.get("Message") or "MyWhoosh login failed"))
|
||||
self.token = MyWhooshToken(
|
||||
access_token=str(body["AccessToken"]),
|
||||
refresh_token=str(body["RefreshToken"]) if body.get("RefreshToken") else None,
|
||||
whoosh_id=str(body["WhooshId"]) if body.get("WhooshId") else None,
|
||||
)
|
||||
self.token_store.save(self.token)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Implement `ensure_authenticated` as cache-first validation**
|
||||
|
||||
Do not invent a refresh endpoint. Validate cached tokens using the normal activities request; on `401/403`, clear the cache, login once, and continue. The later `list_activities()` task provides the request method. Expose the intended behavior now:
|
||||
|
||||
```python
|
||||
async def ensure_authenticated(self, email: str, password: str) -> None:
|
||||
if self.token is None:
|
||||
await self.login(email, password)
|
||||
```
|
||||
|
||||
Task 3 extends this with one retry after an unauthorized activities response.
|
||||
|
||||
- [ ] **Step 6: Run authentication tests**
|
||||
|
||||
Run: `pytest tests/mywhoosh/test_client_auth.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add pyproject.toml app/mywhoosh/client.py tests/mywhoosh/test_client_auth.py
|
||||
git commit -m "feat: add MyWhoosh API login"
|
||||
```
|
||||
|
||||
## Task 3: Implement MyWhoosh activity listing and FIT download
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/mywhoosh/client.py`
|
||||
- Create: `tests/mywhoosh/test_client_activities.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces `list_activities(email: str, password: str) -> list[MyWhooshActivity]`.
|
||||
- Produces `download_fit(activity_file_id: str, email: str, password: str) -> bytes`.
|
||||
- Activities endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/activities` with `{"sortDate":"DESC","page":N}`.
|
||||
- Download endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/download-activity-file` with `{"fileId": activity_file_id}`; response `data` is a pre-signed URL which is fetched with GET.
|
||||
|
||||
- [ ] **Step 1: Write paginated activity-list test**
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_activities_paginates_and_normalizes(tmp_path) -> None:
|
||||
calls = []
|
||||
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
calls.append(str(request.url))
|
||||
if request.url.path.endswith("/activities"):
|
||||
payload = json.loads(request.content)
|
||||
page = payload["page"]
|
||||
result = {
|
||||
"data": {
|
||||
"totalPages": 2,
|
||||
"results": [{
|
||||
"id": f"a-{page}",
|
||||
"title": f"Ride {page}",
|
||||
"activityFileId": f"f-{page}",
|
||||
"startDatetime": "2026-08-15T06:00:00.000Z",
|
||||
}],
|
||||
}
|
||||
}
|
||||
return httpx.Response(200, json=result)
|
||||
raise AssertionError(request.url)
|
||||
```
|
||||
|
||||
Preload the tokenstore with `access_token="cached"`; assert two normalized `MyWhooshActivity` values are returned.
|
||||
|
||||
- [ ] **Step 2: Write expired-token reauthentication test**
|
||||
|
||||
The mock transport sequence must return `401` for the first activities request, a successful login response, then `200` for the retried activities request. Assert login is attempted exactly once and the tokenstore contains the new access token.
|
||||
|
||||
- [ ] **Step 3: Write FIT download test**
|
||||
|
||||
Mock the download-activity-file endpoint to return `{"data":"https://signed.example/activity.fit"}`, then mock that URL to return bytes beginning with a valid FIT header. Assert `download_fit()` returns those exact bytes.
|
||||
|
||||
- [ ] **Step 4: Implement one authenticated-request retry helper**
|
||||
|
||||
```python
|
||||
async def _authenticated_post(self, url: str, payload: dict, email: str, password: str) -> httpx.Response:
|
||||
await self.ensure_authenticated(email, password)
|
||||
for attempt in range(2):
|
||||
assert self.token is not None
|
||||
try:
|
||||
response = await self.http.post(
|
||||
url,
|
||||
json=payload,
|
||||
headers={"Authorization": f"Bearer {self.token.access_token}"},
|
||||
)
|
||||
except httpx.TransportError as exc:
|
||||
raise MyWhooshTransientError("MyWhoosh request failed") from exc
|
||||
if response.status_code not in {401, 403}:
|
||||
if response.status_code >= 500:
|
||||
raise MyWhooshTransientError(f"MyWhoosh returned HTTP {response.status_code}")
|
||||
return response
|
||||
if attempt == 0:
|
||||
self.token_store.clear()
|
||||
self.token = None
|
||||
await self.login(email, password)
|
||||
continue
|
||||
raise MyWhooshAuthError("MyWhoosh session rejected after reauthentication")
|
||||
raise AssertionError("unreachable")
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Implement list normalization and download**
|
||||
|
||||
Parse `startDatetime` as UTC when present. Skip malformed activity rows only if they lack no stable `id` or `activityFileId`; otherwise surface JSON/schema failures as `MyWhooshIntegrationError` so API changes are visible.
|
||||
|
||||
```python
|
||||
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
|
||||
response = await self._authenticated_post(
|
||||
ACTIVITIES_BASE + "rider/profile/download-activity-file",
|
||||
{"fileId": activity_file_id},
|
||||
email,
|
||||
password,
|
||||
)
|
||||
if response.status_code >= 400:
|
||||
raise MyWhooshIntegrationError(f"download metadata returned HTTP {response.status_code}")
|
||||
url = response.json().get("data")
|
||||
if not isinstance(url, str) or not url:
|
||||
raise MyWhooshIntegrationError("MyWhoosh download response has no URL")
|
||||
try:
|
||||
fit_response = await self.http.get(url)
|
||||
except httpx.TransportError as exc:
|
||||
raise MyWhooshTransientError("FIT download failed") from exc
|
||||
if fit_response.status_code >= 500:
|
||||
raise MyWhooshTransientError(f"FIT host returned HTTP {fit_response.status_code}")
|
||||
if fit_response.status_code >= 400:
|
||||
raise MyWhooshIntegrationError(f"FIT host returned HTTP {fit_response.status_code}")
|
||||
return fit_response.content
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run MyWhoosh tests**
|
||||
|
||||
Run: `pytest tests/mywhoosh -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add app/mywhoosh/client.py tests/mywhoosh/test_client_activities.py
|
||||
git commit -m "feat: fetch MyWhoosh activities and FIT files"
|
||||
```
|
||||
|
||||
## Task 4: Implement Garmin import adapter with duplicate and MFA handling
|
||||
|
||||
**Files:**
|
||||
- Create: `app/garmin/uploader.py`
|
||||
- Create: `tests/garmin/test_uploader.py`
|
||||
- Modify: `pyproject.toml`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces `GarminUploader.import_fit(fit_path: Path, mfa_code: str | None = None) -> UploadResult`.
|
||||
- Produces exceptions `GarminUploadBlocked`, `GarminAuthError`, `GarminTransientError`.
|
||||
- Uses `client.login(tokenstore_path)` and `client.import_activity(activity_path)`.
|
||||
|
||||
- [ ] **Step 1: Add Garmin dependency**
|
||||
|
||||
Add to runtime dependencies:
|
||||
|
||||
```toml
|
||||
"garminconnect>=0.2,<1",
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write fake-client tests based on the previously working uploader pattern**
|
||||
|
||||
```python
|
||||
# tests/garmin/test_uploader.py
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.garmin.uploader import GarminUploadBlocked, GarminUploader
|
||||
|
||||
|
||||
class FakeGarmin:
|
||||
def __init__(self, *args, prompt_mfa=None, import_result=None, login_error=None, import_error=None, **kwargs):
|
||||
self.prompt_mfa = prompt_mfa
|
||||
self.import_result = import_result or {"activityId": 42}
|
||||
self.login_error = login_error
|
||||
self.import_error = import_error
|
||||
self.login_path = None
|
||||
|
||||
def login(self, tokenstore=None):
|
||||
self.login_path = tokenstore
|
||||
if self.login_error:
|
||||
raise self.login_error
|
||||
|
||||
def import_activity(self, activity_path: str):
|
||||
if self.import_error:
|
||||
raise self.import_error
|
||||
return self.import_result
|
||||
```
|
||||
|
||||
Add these concrete tests below the fake client:
|
||||
|
||||
```python
|
||||
def test_successful_import_returns_activity_id(tmp_path: Path) -> None:
|
||||
uploader = GarminUploader(
|
||||
email="g@example.com",
|
||||
password="pw",
|
||||
tokenstore=tmp_path / "garmin",
|
||||
client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_result={"activityId": 42}),
|
||||
)
|
||||
result = uploader.import_fit(tmp_path / "ride.fit")
|
||||
assert result.status == "imported"
|
||||
assert result.garmin_activity_id == "42"
|
||||
|
||||
|
||||
def test_duplicate_is_terminal_success(tmp_path: Path) -> None:
|
||||
uploader = GarminUploader(
|
||||
email="g@example.com",
|
||||
password="pw",
|
||||
tokenstore=tmp_path / "garmin",
|
||||
client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_error=RuntimeError("409 duplicate")),
|
||||
)
|
||||
result = uploader.import_fit(tmp_path / "ride.fit")
|
||||
assert result.duplicate is True
|
||||
assert result.status == "duplicate"
|
||||
|
||||
|
||||
def test_mfa_without_code_is_blocked(tmp_path: Path) -> None:
|
||||
class MfaGarmin(FakeGarmin):
|
||||
def login(self, tokenstore=None):
|
||||
self.prompt_mfa()
|
||||
|
||||
uploader = GarminUploader(
|
||||
email="g@example.com",
|
||||
password="pw",
|
||||
tokenstore=tmp_path / "garmin",
|
||||
client_factory=MfaGarmin,
|
||||
)
|
||||
with pytest.raises(GarminUploadBlocked):
|
||||
uploader.import_fit(tmp_path / "ride.fit")
|
||||
|
||||
|
||||
def test_mfa_code_is_returned_only_to_prompt(tmp_path: Path) -> None:
|
||||
seen = []
|
||||
|
||||
class MfaGarmin(FakeGarmin):
|
||||
def login(self, tokenstore=None):
|
||||
seen.append(self.prompt_mfa())
|
||||
|
||||
uploader = GarminUploader(
|
||||
email="g@example.com",
|
||||
password="pw",
|
||||
tokenstore=tmp_path / "garmin",
|
||||
client_factory=MfaGarmin,
|
||||
)
|
||||
uploader.import_fit(tmp_path / "ride.fit", mfa_code="123456")
|
||||
assert seen == ["123456"]
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/garmin/test_uploader.py -v`
|
||||
|
||||
Expected: import failure.
|
||||
|
||||
- [ ] **Step 4: Implement the adapter**
|
||||
|
||||
```python
|
||||
# app/garmin/uploader.py
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Protocol
|
||||
|
||||
|
||||
class GarminClientProtocol(Protocol):
|
||||
def login(self, tokenstore: str | None = None) -> Any: ...
|
||||
def import_activity(self, activity_path: str) -> Any: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UploadResult:
|
||||
status: str
|
||||
duplicate: bool
|
||||
garmin_activity_id: str | None
|
||||
raw_response: Any
|
||||
|
||||
|
||||
class GarminUploadBlocked(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class GarminAuthError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class GarminTransientError(RuntimeError):
|
||||
pass
|
||||
```
|
||||
|
||||
Implement the uploader fully:
|
||||
|
||||
```python
|
||||
class GarminUploader:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
email: str,
|
||||
password: str,
|
||||
tokenstore: Path,
|
||||
client_factory: Callable[..., GarminClientProtocol] | None = None,
|
||||
) -> None:
|
||||
self.email = email
|
||||
self.password = password
|
||||
self.tokenstore = tokenstore
|
||||
self.client_factory = client_factory
|
||||
self._client: GarminClientProtocol | None = None
|
||||
self._mfa_code: str | None = None
|
||||
|
||||
def import_fit(self, fit_path: Path, mfa_code: str | None = None) -> UploadResult:
|
||||
self._mfa_code = mfa_code
|
||||
try:
|
||||
client = self._ensure_client()
|
||||
try:
|
||||
response = client.import_activity(str(fit_path))
|
||||
except Exception as exc:
|
||||
if _looks_duplicate_error(exc):
|
||||
return UploadResult("duplicate", True, None, str(exc))
|
||||
text = str(exc).lower()
|
||||
if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")):
|
||||
raise GarminTransientError("Garmin import failed transiently") from exc
|
||||
raise
|
||||
return UploadResult("imported", False, _extract_activity_id(response), response)
|
||||
finally:
|
||||
self._mfa_code = None
|
||||
|
||||
def _ensure_client(self) -> GarminClientProtocol:
|
||||
if self._client is not None:
|
||||
return self._client
|
||||
self.tokenstore.mkdir(parents=True, exist_ok=True)
|
||||
factory = self.client_factory or _default_garmin_factory
|
||||
client = factory(self.email, self.password, prompt_mfa=self._prompt_mfa)
|
||||
try:
|
||||
client.login(str(self.tokenstore))
|
||||
except GarminUploadBlocked:
|
||||
raise
|
||||
except Exception as exc:
|
||||
text = str(exc).lower()
|
||||
if "mfa" in text:
|
||||
raise GarminUploadBlocked("Garmin MFA is required") from exc
|
||||
if any(token in text for token in ("password", "credential", "unauthorized", "401")):
|
||||
raise GarminAuthError("Garmin authentication failed") from exc
|
||||
if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")):
|
||||
raise GarminTransientError("Garmin login failed transiently") from exc
|
||||
raise GarminAuthError("Garmin login failed") from exc
|
||||
self._client = client
|
||||
return client
|
||||
|
||||
def _prompt_mfa(self) -> str:
|
||||
if self._mfa_code:
|
||||
return self._mfa_code
|
||||
raise GarminUploadBlocked("Garmin requested MFA")
|
||||
|
||||
|
||||
def _default_garmin_factory(*args: Any, **kwargs: Any) -> GarminClientProtocol:
|
||||
from garminconnect import Garmin
|
||||
|
||||
return Garmin(*args, **kwargs)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Keep duplicate and response-ID extraction deterministic**
|
||||
|
||||
Use these helpers:
|
||||
|
||||
```python
|
||||
def _looks_duplicate_error(exc: Exception) -> bool:
|
||||
text = str(exc).lower()
|
||||
return any(token in text for token in ("duplicate", "already exists", "409"))
|
||||
|
||||
|
||||
def _extract_activity_id(response: Any) -> str | None:
|
||||
if not isinstance(response, dict):
|
||||
return None
|
||||
candidates = [response.get("activityId"), response.get("activity_id"), response.get("id")]
|
||||
detailed = response.get("detailedImportResult")
|
||||
if isinstance(detailed, dict):
|
||||
candidates.extend([detailed.get("uploadId"), detailed.get("activityId")])
|
||||
for key in ("successes", "success", "importedActivities"):
|
||||
items = response.get(key)
|
||||
if isinstance(items, list) and items and isinstance(items[0], dict):
|
||||
candidates.extend([items[0].get("activityId"), items[0].get("id")])
|
||||
return next((str(value) for value in candidates if value is not None), None)
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run Garmin tests**
|
||||
|
||||
Run: `pytest tests/garmin/test_uploader.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add pyproject.toml app/garmin/uploader.py tests/garmin/test_uploader.py
|
||||
git commit -m "feat: import FIT activities into Garmin"
|
||||
```
|
||||
804
docs/superpowers/plans/2026-08-15-sync-scheduler-web.md
Normal file
804
docs/superpowers/plans/2026-08-15-sync-scheduler-web.md
Normal file
@@ -0,0 +1,804 @@
|
||||
# Sync Engine, Scheduler, and Operational UI Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Integrate the database, MyWhoosh client, FIT rewriter, Garmin importer, scheduler, MFA workflow, retry behavior, and operational admin pages into a resilient multi-user sync service.
|
||||
|
||||
**Architecture:** A `SyncManager` owns per-user `asyncio.Lock` instances and executes a durable activity state machine. External clients are injected via factories for tests. A lightweight FastAPI lifespan scheduler triggers syncs at the configured interval; different users run concurrently, while each user's pipeline is serialized.
|
||||
|
||||
**Tech Stack:** Python 3.12, asyncio, FastAPI lifespan, SQLAlchemy, HTMX, Jinja2, pytest/pytest-asyncio.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Multiple users sync independently and may run concurrently.
|
||||
- At most one sync may run for a given user at a time.
|
||||
- Manual sync and scheduled sync use the same pipeline and lock.
|
||||
- Durable activity stages are `discovered`, `downloaded`, `converted`, `imported`, `duplicate`, `failed` with `last_completed_stage` retained on failure.
|
||||
- `imported` and `duplicate` are terminal.
|
||||
- Transient network/server failures retry at most once in a run; later attempts occur on future scheduler ticks.
|
||||
- Invalid MyWhoosh/Garmin credentials and Garmin MFA set `action_required`.
|
||||
- Corrupt/unsupported FIT is non-retryable per activity.
|
||||
- Failure of one user or one activity must never stop other users.
|
||||
- Original and converted FIT files remain on disk in v1.
|
||||
- MFA codes are never persisted or logged.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```text
|
||||
app/sync/
|
||||
__init__.py
|
||||
states.py
|
||||
manager.py
|
||||
scheduler.py
|
||||
app/web/
|
||||
operations.py
|
||||
templates/
|
||||
dashboard.html
|
||||
users/detail.html
|
||||
system.html
|
||||
fragments/user_card.html
|
||||
fragments/sync_result.html
|
||||
fragments/mfa_form.html
|
||||
app/db/
|
||||
repositories.py
|
||||
tests/sync/
|
||||
fakes.py
|
||||
test_manager.py
|
||||
test_concurrency.py
|
||||
test_scheduler.py
|
||||
tests/web/
|
||||
test_operations.py
|
||||
test_mfa.py
|
||||
```
|
||||
|
||||
## Task 1: Add durable activity/sync-run repository operations
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/db/repositories.py`
|
||||
- Create: `tests/db/test_sync_state.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces methods to advance activity stages, mark failures without losing `last_completed_stage`, list pending activities, and create/finalize sync runs.
|
||||
|
||||
- [ ] **Step 1: Write failing state-transition tests**
|
||||
|
||||
```python
|
||||
# tests/db/test_sync_state.py
|
||||
from app.db.models import ActivityStatus
|
||||
|
||||
|
||||
def test_failure_retains_last_completed_stage(activity_repository, seeded_activity) -> None:
|
||||
activity_repository.mark_downloaded(seeded_activity.id, "/data/activities/1/a/source.fit")
|
||||
activity_repository.mark_failed(seeded_activity.id, "Garmin timeout", retryable=True)
|
||||
activity = activity_repository.get(seeded_activity.id)
|
||||
|
||||
assert activity.status == ActivityStatus.FAILED
|
||||
assert activity.last_completed_stage == ActivityStatus.DOWNLOADED
|
||||
assert activity.retryable is True
|
||||
|
||||
|
||||
def test_converted_activity_is_pending_until_terminal(activity_repository, seeded_activity) -> None:
|
||||
activity_repository.mark_converted(seeded_activity.id, "/data/activities/1/a/converted.fit")
|
||||
ids = [item.id for item in activity_repository.list_pending_for_user(seeded_activity.user_id)]
|
||||
assert seeded_activity.id in ids
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/db/test_sync_state.py -v`
|
||||
|
||||
Expected: missing repository methods.
|
||||
|
||||
- [ ] **Step 3: Implement explicit transition methods**
|
||||
|
||||
Add methods with these exact effects:
|
||||
|
||||
```python
|
||||
def mark_downloaded(self, activity_id: int, path: str) -> Activity:
|
||||
activity = self._require(activity_id)
|
||||
activity.source_fit_path = path
|
||||
activity.status = ActivityStatus.DOWNLOADED
|
||||
activity.last_completed_stage = ActivityStatus.DOWNLOADED
|
||||
activity.last_error = None
|
||||
activity.retryable = True
|
||||
self.session.commit()
|
||||
return activity
|
||||
|
||||
|
||||
def mark_converted(self, activity_id: int, path: str) -> Activity:
|
||||
activity = self._require(activity_id)
|
||||
activity.converted_fit_path = path
|
||||
activity.status = ActivityStatus.CONVERTED
|
||||
activity.last_completed_stage = ActivityStatus.CONVERTED
|
||||
activity.last_error = None
|
||||
activity.retryable = True
|
||||
self.session.commit()
|
||||
return activity
|
||||
|
||||
|
||||
def mark_imported(self, activity_id: int, garmin_activity_id: str | None) -> Activity:
|
||||
activity = self._require(activity_id)
|
||||
activity.status = ActivityStatus.IMPORTED
|
||||
activity.last_completed_stage = ActivityStatus.IMPORTED
|
||||
activity.garmin_activity_id = garmin_activity_id
|
||||
activity.last_error = None
|
||||
activity.retryable = False
|
||||
self.session.commit()
|
||||
return activity
|
||||
|
||||
|
||||
def mark_duplicate(self, activity_id: int) -> Activity:
|
||||
activity = self._require(activity_id)
|
||||
activity.status = ActivityStatus.DUPLICATE
|
||||
activity.last_completed_stage = ActivityStatus.DUPLICATE
|
||||
activity.last_error = None
|
||||
activity.retryable = False
|
||||
self.session.commit()
|
||||
return activity
|
||||
|
||||
|
||||
def mark_failed(self, activity_id: int, error: str, *, retryable: bool) -> Activity:
|
||||
activity = self._require(activity_id)
|
||||
activity.status = ActivityStatus.FAILED
|
||||
activity.last_error = error[:2000]
|
||||
activity.retryable = retryable
|
||||
self.session.commit()
|
||||
return activity
|
||||
```
|
||||
|
||||
`list_pending_for_user()` must exclude terminal states and include failed rows only when `retryable=True`.
|
||||
|
||||
- [ ] **Step 4: Add `SyncRunRepository` create/finalize methods**
|
||||
|
||||
`start(user_id)` creates `RUNNING`; `finish(...)` sets counts, `finished_at`, status, and optional summary error. Do not store exception tracebacks in SQLite.
|
||||
|
||||
- [ ] **Step 5: Run DB state tests**
|
||||
|
||||
Run: `pytest tests/db/test_sync_state.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add app/db/repositories.py tests/db/test_sync_state.py
|
||||
git commit -m "feat: add durable sync state transitions"
|
||||
```
|
||||
|
||||
## Task 2: Implement the single-user sync state machine
|
||||
|
||||
**Files:**
|
||||
- Create: `app/sync/states.py`
|
||||
- Create: `app/sync/manager.py`
|
||||
- Create: `tests/sync/fakes.py`
|
||||
- Create: `tests/sync/test_manager.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces `SyncManager.sync_user(user_id: int, mfa_code: str | None = None) -> SyncOutcome`.
|
||||
- Constructor receives `session_factory`, `credential_cipher`, `settings`, `mywhoosh_factory`, `garmin_factory`, and `fit_converter`.
|
||||
- `mywhoosh_factory(token_store: MyWhooshTokenStore) -> MyWhooshClient`.
|
||||
- `garmin_factory(email: str, password: str, tokenstore: Path) -> GarminUploader`.
|
||||
- `fit_converter(source_path: Path, output_path: Path) -> FitConversionResult`.
|
||||
|
||||
- [ ] **Step 1: Define result models and fake integration factories**
|
||||
|
||||
```python
|
||||
# app/sync/states.py
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SyncOutcome:
|
||||
user_id: int
|
||||
status: str
|
||||
discovered: int
|
||||
imported: int
|
||||
skipped: int
|
||||
failed: int
|
||||
message: str | None = None
|
||||
```
|
||||
|
||||
Implement concrete fakes:
|
||||
|
||||
```python
|
||||
# tests/sync/fakes.py
|
||||
from app.garmin.uploader import UploadResult
|
||||
|
||||
|
||||
class FakeMyWhooshClient:
|
||||
def __init__(self, activities, fit_bytes: bytes) -> None:
|
||||
self.activities = activities
|
||||
self.fit_bytes = fit_bytes
|
||||
self.list_calls = 0
|
||||
self.download_calls = 0
|
||||
|
||||
async def list_activities(self, email: str, password: str):
|
||||
self.list_calls += 1
|
||||
return list(self.activities)
|
||||
|
||||
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
|
||||
self.download_calls += 1
|
||||
return self.fit_bytes
|
||||
|
||||
|
||||
class FakeGarminUploader:
|
||||
def __init__(self, result: UploadResult | None = None, error: Exception | None = None) -> None:
|
||||
self.result = result or UploadResult("imported", False, "g-1", {"activityId": "g-1"})
|
||||
self.error = error
|
||||
self.calls = 0
|
||||
|
||||
def import_fit(self, fit_path, mfa_code=None):
|
||||
self.calls += 1
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
return self.result
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the happy-path test**
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_activity_downloads_converts_and_imports(manager, seeded_user, tmp_path) -> None:
|
||||
outcome = await manager.sync_user(seeded_user.id)
|
||||
|
||||
assert outcome.discovered == 1
|
||||
assert outcome.imported == 1
|
||||
assert outcome.failed == 0
|
||||
activity = load_only_activity(seeded_user.id)
|
||||
assert activity.status == ActivityStatus.IMPORTED
|
||||
assert Path(activity.source_fit_path).exists()
|
||||
assert Path(activity.converted_fit_path).exists()
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Write resume tests before implementation**
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("status", "last_stage", "expected_downloads", "expected_conversions", "expected_imports"),
|
||||
[
|
||||
(ActivityStatus.DOWNLOADED, ActivityStatus.DOWNLOADED, 0, 1, 1),
|
||||
(ActivityStatus.CONVERTED, ActivityStatus.CONVERTED, 0, 0, 1),
|
||||
(ActivityStatus.IMPORTED, ActivityStatus.IMPORTED, 0, 0, 0),
|
||||
(ActivityStatus.FAILED, ActivityStatus.CONVERTED, 0, 0, 1),
|
||||
],
|
||||
)
|
||||
async def test_resume_from_durable_stage(
|
||||
manager_factory, seeded_activity_factory, status, last_stage,
|
||||
expected_downloads, expected_conversions, expected_imports,
|
||||
) -> None:
|
||||
activity = seeded_activity_factory(status=status, last_completed_stage=last_stage, retryable=True)
|
||||
manager, mywhoosh, converter, garmin = manager_factory(activity)
|
||||
await manager.sync_user(activity.user_id)
|
||||
assert mywhoosh.download_calls == expected_downloads
|
||||
assert converter.calls == expected_conversions
|
||||
assert garmin.calls == expected_imports
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/sync/test_manager.py -v`
|
||||
|
||||
Expected: missing manager.
|
||||
|
||||
- [ ] **Step 5: Implement per-activity filesystem layout and state machine**
|
||||
|
||||
Use paths:
|
||||
|
||||
```python
|
||||
activity_dir = settings.activities_dir / str(user.id) / activity.mywhoosh_activity_id
|
||||
source_path = activity_dir / "source.fit"
|
||||
converted_path = activity_dir / "edge-1030-plus.fit"
|
||||
```
|
||||
|
||||
Create per-user integration instances from decrypted credentials and isolated token paths:
|
||||
|
||||
```python
|
||||
mw_email = self.credential_cipher.decrypt(user.mywhoosh_email_enc)
|
||||
mw_password = self.credential_cipher.decrypt(user.mywhoosh_password_enc)
|
||||
garmin_email = self.credential_cipher.decrypt(user.garmin_email_enc)
|
||||
garmin_password = self.credential_cipher.decrypt(user.garmin_password_enc)
|
||||
|
||||
token_dir = self.settings.tokens_dir / str(user.id)
|
||||
mywhoosh = self.mywhoosh_factory(MyWhooshTokenStore(token_dir / "mywhoosh.json"))
|
||||
garmin = self.garmin_factory(garmin_email, garmin_password, token_dir / "garmin")
|
||||
remote_activities = await mywhoosh.list_activities(mw_email, mw_password)
|
||||
```
|
||||
|
||||
For each remote activity, call `get_or_create_discovered(...)`, then resume from `activity.last_completed_stage` when `activity.status == FAILED`; otherwise use `activity.status`.
|
||||
|
||||
Core sequence:
|
||||
|
||||
```python
|
||||
if stage == ActivityStatus.DISCOVERED:
|
||||
fit_bytes = await mywhoosh.download_fit(remote.activity_file_id, mw_email, mw_password)
|
||||
activity_dir.mkdir(parents=True, exist_ok=True)
|
||||
source_path.write_bytes(fit_bytes)
|
||||
repo.mark_downloaded(activity.id, str(source_path))
|
||||
|
||||
if stage in {ActivityStatus.DOWNLOADED}:
|
||||
fit_converter(source_path, converted_path)
|
||||
repo.mark_converted(activity.id, str(converted_path))
|
||||
|
||||
if stage in {ActivityStatus.CONVERTED}:
|
||||
upload = await asyncio.to_thread(garmin.import_fit, converted_path, mfa_code)
|
||||
if upload.duplicate:
|
||||
repo.mark_duplicate(activity.id)
|
||||
else:
|
||||
repo.mark_imported(activity.id, upload.garmin_activity_id)
|
||||
```
|
||||
|
||||
After each repository transition, update the local `stage` variable from the returned record so resume behavior is deterministic.
|
||||
|
||||
- [ ] **Step 6: Implement exception mapping**
|
||||
|
||||
Map exceptions with explicit user connection-state updates:
|
||||
|
||||
```python
|
||||
except MyWhooshTransientError as exc:
|
||||
user.health_state = HealthState.DEGRADED
|
||||
user.mywhoosh_state = "error"
|
||||
repo.mark_failed(activity.id, str(exc), retryable=True)
|
||||
except MyWhooshAuthError as exc:
|
||||
user.health_state = HealthState.ACTION_REQUIRED
|
||||
user.mywhoosh_state = "auth_required"
|
||||
user.action_reason = "mywhoosh_auth_required"
|
||||
stop_user_run = True
|
||||
except MyWhooshIntegrationError as exc:
|
||||
user.health_state = HealthState.ACTION_REQUIRED
|
||||
user.mywhoosh_state = "integration_error"
|
||||
user.action_reason = "mywhoosh_integration_changed"
|
||||
stop_user_run = True
|
||||
except GarminUploadBlocked:
|
||||
user.health_state = HealthState.ACTION_REQUIRED
|
||||
user.garmin_state = "mfa_required"
|
||||
user.action_reason = "garmin_mfa_required"
|
||||
stop_user_run = True
|
||||
except GarminAuthError as exc:
|
||||
user.health_state = HealthState.ACTION_REQUIRED
|
||||
user.garmin_state = "auth_required"
|
||||
user.action_reason = "garmin_auth_required"
|
||||
stop_user_run = True
|
||||
except GarminTransientError as exc:
|
||||
user.health_state = HealthState.DEGRADED
|
||||
user.garmin_state = "error"
|
||||
repo.mark_failed(activity.id, str(exc), retryable=True)
|
||||
except FitFormatError as exc:
|
||||
repo.mark_failed(activity.id, str(exc), retryable=False)
|
||||
```
|
||||
|
||||
On successful MyWhoosh listing set `mywhoosh_state="connected"`; on successful Garmin import set `garmin_state="connected"`. Persist the user after each connection-state change. Unexpected exceptions mark the run/user `degraded` and log only exception class plus sanitized message.
|
||||
|
||||
- [ ] **Step 7: Run manager tests**
|
||||
|
||||
Run: `pytest tests/sync/test_manager.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add app/sync/states.py app/sync/manager.py tests/sync
|
||||
git commit -m "feat: add resumable per-user sync pipeline"
|
||||
```
|
||||
|
||||
## Task 3: Add per-user locks and cross-user isolation
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/sync/manager.py`
|
||||
- Create: `tests/sync/test_concurrency.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces `SyncAlreadyRunning` and ensures only one active `sync_user()` call per user.
|
||||
|
||||
- [ ] **Step 1: Write concurrency tests**
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_same_user_cannot_run_twice(manager, seeded_user) -> None:
|
||||
first_started = asyncio.Event()
|
||||
release_first = asyncio.Event()
|
||||
manager.test_hooks = SyncTestHooks(first_started=first_started, release=release_first)
|
||||
|
||||
first = asyncio.create_task(manager.sync_user(seeded_user.id))
|
||||
await first_started.wait()
|
||||
|
||||
with pytest.raises(SyncAlreadyRunning):
|
||||
await manager.sync_user(seeded_user.id)
|
||||
|
||||
release_first.set()
|
||||
await first
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_different_users_can_run_concurrently(manager, user_a, user_b) -> None:
|
||||
results = await asyncio.gather(manager.sync_user(user_a.id), manager.sync_user(user_b.id))
|
||||
assert {result.user_id for result in results} == {user_a.id, user_b.id}
|
||||
```
|
||||
|
||||
Do not leave production-only `test_hooks`; instead inject a fake MyWhoosh client whose `list_activities()` blocks on test events.
|
||||
|
||||
- [ ] **Step 2: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/sync/test_concurrency.py -v`
|
||||
|
||||
Expected: same-user duplicate execution is not yet blocked.
|
||||
|
||||
- [ ] **Step 3: Implement lock registry**
|
||||
|
||||
```python
|
||||
class SyncAlreadyRunning(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class SyncManager:
|
||||
def __init__(...):
|
||||
self._locks: dict[int, asyncio.Lock] = {}
|
||||
self._locks_guard = asyncio.Lock()
|
||||
|
||||
async def _lock_for(self, user_id: int) -> asyncio.Lock:
|
||||
async with self._locks_guard:
|
||||
return self._locks.setdefault(user_id, asyncio.Lock())
|
||||
|
||||
async def sync_user(self, user_id: int, mfa_code: str | None = None) -> SyncOutcome:
|
||||
lock = await self._lock_for(user_id)
|
||||
if lock.locked():
|
||||
raise SyncAlreadyRunning(f"sync already running for user {user_id}")
|
||||
async with lock:
|
||||
return await self._sync_user_locked(user_id, mfa_code)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add a `sync_all_enabled()` isolation method**
|
||||
|
||||
```python
|
||||
async def sync_all_enabled(self) -> list[SyncOutcome | Exception]:
|
||||
user_ids = self._load_enabled_user_ids()
|
||||
return await asyncio.gather(
|
||||
*(self.sync_user(user_id) for user_id in user_ids),
|
||||
return_exceptions=True,
|
||||
)
|
||||
```
|
||||
|
||||
A failure for one user must appear as one list element and must not cancel sibling jobs.
|
||||
|
||||
- [ ] **Step 5: Run concurrency tests**
|
||||
|
||||
Run: `pytest tests/sync/test_concurrency.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add app/sync/manager.py tests/sync/test_concurrency.py
|
||||
git commit -m "feat: isolate concurrent user syncs"
|
||||
```
|
||||
|
||||
## Task 4: Add the periodic scheduler through FastAPI lifespan
|
||||
|
||||
**Files:**
|
||||
- Create: `app/sync/scheduler.py`
|
||||
- Modify: `app/main.py`
|
||||
- Create: `tests/sync/test_scheduler.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces `SyncScheduler.start()`, `stop()`, `run_once()`, `last_tick`, `next_tick`.
|
||||
- Scheduler interval is `Settings.sync_interval_minutes`.
|
||||
|
||||
- [ ] **Step 1: Write scheduler test with a short injected interval**
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_scheduler_calls_sync_all_and_survives_failure() -> None:
|
||||
fake = FakeSyncManager(results=[RuntimeError("one user failed")])
|
||||
scheduler = SyncScheduler(fake, interval_seconds=0.01)
|
||||
await scheduler.start()
|
||||
await asyncio.sleep(0.035)
|
||||
await scheduler.stop()
|
||||
assert fake.calls >= 2
|
||||
assert scheduler.last_tick is not None
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/sync/test_scheduler.py -v`
|
||||
|
||||
Expected: missing scheduler.
|
||||
|
||||
- [ ] **Step 3: Implement scheduler loop**
|
||||
|
||||
```python
|
||||
class SyncScheduler:
|
||||
def __init__(self, manager, *, interval_seconds: float) -> None:
|
||||
self.manager = manager
|
||||
self.interval_seconds = interval_seconds
|
||||
self._task: asyncio.Task | None = None
|
||||
self._stop = asyncio.Event()
|
||||
self.last_tick = None
|
||||
self.next_tick = None
|
||||
|
||||
async def run_once(self) -> None:
|
||||
self.last_tick = datetime.now(timezone.utc)
|
||||
await self.manager.sync_all_enabled()
|
||||
self.next_tick = datetime.now(timezone.utc) + timedelta(seconds=self.interval_seconds)
|
||||
|
||||
async def _run(self) -> None:
|
||||
while not self._stop.is_set():
|
||||
await self.run_once()
|
||||
try:
|
||||
await asyncio.wait_for(self._stop.wait(), timeout=self.interval_seconds)
|
||||
except TimeoutError:
|
||||
pass
|
||||
```
|
||||
|
||||
`stop()` sets the event and awaits the task. Never allow one `sync_all_enabled()` exception to kill the loop; log it and continue.
|
||||
|
||||
- [ ] **Step 4: Wire into FastAPI lifespan**
|
||||
|
||||
Build the concrete `SyncManager` once during app startup, store it on `app.state.sync_manager`, create `SyncScheduler(... interval_minutes * 60)`, start it, and stop it during lifespan shutdown.
|
||||
|
||||
- [ ] **Step 5: Run scheduler tests**
|
||||
|
||||
Run: `pytest tests/sync/test_scheduler.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add app/sync/scheduler.py app/main.py tests/sync/test_scheduler.py
|
||||
git commit -m "feat: schedule periodic user synchronization"
|
||||
```
|
||||
|
||||
## Task 5: Add dashboard/manual sync/system operational routes
|
||||
|
||||
**Files:**
|
||||
- Create: `app/web/operations.py`
|
||||
- Modify: `app/web/routes.py`
|
||||
- Modify: `app/web/templates/dashboard.html`
|
||||
- Create: `app/web/templates/system.html`
|
||||
- Create: `app/web/templates/fragments/sync_result.html`
|
||||
- Create: `tests/web/test_operations.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Routes: `POST /users/{id}/sync`, `POST /sync-all`, `GET /system`.
|
||||
- Manual actions use the same `SyncManager` instance and lock as the scheduler.
|
||||
|
||||
- [ ] **Step 1: Write manual-sync tests**
|
||||
|
||||
```python
|
||||
def test_manual_sync_calls_shared_manager(authenticated_client, fake_sync_manager) -> None:
|
||||
response = authenticated_client.post(
|
||||
"/users/1/sync",
|
||||
data={"csrf_token": authenticated_client.csrf_token},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert fake_sync_manager.user_calls == [1]
|
||||
|
||||
|
||||
def test_manual_sync_reports_already_running(authenticated_client, fake_sync_manager) -> None:
|
||||
fake_sync_manager.raise_already_running = True
|
||||
response = authenticated_client.post(
|
||||
"/users/1/sync",
|
||||
data={"csrf_token": authenticated_client.csrf_token},
|
||||
)
|
||||
assert response.status_code == 409
|
||||
assert "already running" in response.text.lower()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run and verify failure**
|
||||
|
||||
Run: `pytest tests/web/test_operations.py -v`
|
||||
|
||||
Expected: routes missing.
|
||||
|
||||
- [ ] **Step 3: Implement routes with admin and CSRF checks**
|
||||
|
||||
Each state-changing route must execute in this order:
|
||||
|
||||
```python
|
||||
require_admin(request)
|
||||
validate_csrf(request, csrf_token)
|
||||
```
|
||||
|
||||
Then call `await request.app.state.sync_manager.sync_user(user_id)` or `sync_all_enabled()`.
|
||||
|
||||
- [ ] **Step 4: Expand dashboard data**
|
||||
|
||||
Add a repository projection that contains only safe display fields:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class UserDashboardRow:
|
||||
id: int
|
||||
name: str
|
||||
enabled: bool
|
||||
health_state: str
|
||||
action_reason: str | None
|
||||
last_sync_at: datetime | None
|
||||
last_activity_name: str | None
|
||||
last_activity_status: str | None
|
||||
|
||||
|
||||
def dashboard_rows(self) -> list[UserDashboardRow]:
|
||||
users = self.list_all()
|
||||
rows = []
|
||||
for user in users:
|
||||
last_run = self.session.scalar(
|
||||
select(SyncRun).where(SyncRun.user_id == user.id).order_by(SyncRun.started_at.desc()).limit(1)
|
||||
)
|
||||
last_activity = self.session.scalar(
|
||||
select(Activity).where(Activity.user_id == user.id).order_by(Activity.created_at.desc()).limit(1)
|
||||
)
|
||||
rows.append(UserDashboardRow(
|
||||
id=user.id,
|
||||
name=user.name,
|
||||
enabled=user.enabled,
|
||||
health_state=user.health_state.value,
|
||||
action_reason=user.action_reason,
|
||||
last_sync_at=last_run.finished_at if last_run else None,
|
||||
last_activity_name=last_activity.activity_name if last_activity else None,
|
||||
last_activity_status=last_activity.status.value if last_activity else None,
|
||||
))
|
||||
return rows
|
||||
```
|
||||
|
||||
Pass only these rows to `dashboard.html`. Render the MFA action only when `row.action_reason == "garmin_mfa_required"`. No decrypted credential is part of this projection.
|
||||
|
||||
- [ ] **Step 5: Implement read-only system page**
|
||||
|
||||
Expose application version, configured interval, scheduler `last_tick` and `next_tick`, user count, and activity count. The only action is a CSRF-protected `sync all now` POST.
|
||||
|
||||
- [ ] **Step 6: Run tests**
|
||||
|
||||
Run: `pytest tests/web/test_operations.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add app/web tests/web/test_operations.py
|
||||
git commit -m "feat: add operational sync controls"
|
||||
```
|
||||
|
||||
## Task 6: Add Garmin MFA lifecycle and failed-activity retry
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/web/routes.py`
|
||||
- Modify: `app/web/templates/users/detail.html`
|
||||
- Create: `app/web/templates/fragments/mfa_form.html`
|
||||
- Create: `tests/web/test_mfa.py`
|
||||
- Modify: `app/sync/manager.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Route: `POST /users/{id}/garmin-mfa` with one-time `code`.
|
||||
- Route: `POST /activities/{id}/retry`.
|
||||
- MFA code exists only in request memory and the immediate `sync_user(user_id, mfa_code=code)` call.
|
||||
|
||||
- [ ] **Step 1: Write MFA lifecycle test**
|
||||
|
||||
```python
|
||||
def test_mfa_code_is_used_once_and_not_persisted(authenticated_client, fake_sync_manager, db_session) -> None:
|
||||
response = authenticated_client.post(
|
||||
"/users/1/garmin-mfa",
|
||||
data={"csrf_token": authenticated_client.csrf_token, "code": "123456"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert fake_sync_manager.mfa_calls == [(1, "123456")]
|
||||
|
||||
persisted_text = " ".join(str(row) for row in db_session.execute(text("select * from sync_runs")).all())
|
||||
assert "123456" not in persisted_text
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write retry test for non-terminal failed activity**
|
||||
|
||||
Assert the route changes a retryable failed activity back to `status=last_completed_stage`, clears `last_error`, then calls the user's normal sync. Reject retry for `retryable=False` with HTTP 409.
|
||||
|
||||
- [ ] **Step 3: Implement MFA route**
|
||||
|
||||
Validate code as a non-empty short string, never log it, and call:
|
||||
|
||||
```python
|
||||
outcome = await request.app.state.sync_manager.sync_user(user_id, mfa_code=code.strip())
|
||||
```
|
||||
|
||||
After a successful Garmin login/import, clear `action_reason` and restore health to `healthy` or `degraded` according to the resulting sync outcome.
|
||||
|
||||
- [ ] **Step 4: Implement failed-activity reset operation**
|
||||
|
||||
Repository method:
|
||||
|
||||
```python
|
||||
def reset_retryable_failure(self, activity_id: int) -> Activity:
|
||||
activity = self._require(activity_id)
|
||||
if activity.status != ActivityStatus.FAILED or not activity.retryable:
|
||||
raise ValueError("activity is not retryable")
|
||||
activity.status = activity.last_completed_stage
|
||||
activity.last_error = None
|
||||
self.session.commit()
|
||||
return activity
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run MFA/retry tests**
|
||||
|
||||
Run: `pytest tests/web/test_mfa.py tests/web/test_operations.py -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add app/web app/sync/manager.py app/db/repositories.py tests/web/test_mfa.py
|
||||
git commit -m "feat: handle Garmin MFA and activity retries"
|
||||
```
|
||||
|
||||
## Task 7: End-to-end regression and Docker acceptance
|
||||
|
||||
**Files:**
|
||||
- Modify: `docker-compose.example.yml` only if integration exposes a missing runtime configuration
|
||||
- Create: `tests/test_acceptance.py`
|
||||
|
||||
**Interfaces:**
|
||||
- No new interface; verifies the v1 acceptance criteria with fake external services.
|
||||
|
||||
- [ ] **Step 1: Add an application-level acceptance test with two users**
|
||||
|
||||
Build the app with temporary SQLite/data directories and injected fake MyWhoosh/Garmin factories. Seed two enabled users, give each one distinct remote activity IDs, run `sync_all_enabled()`, and assert:
|
||||
|
||||
```python
|
||||
assert all(result.status == "success" for result in results)
|
||||
assert count_terminal_activities(user_a.id) == 1
|
||||
assert count_terminal_activities(user_b.id) == 1
|
||||
assert user_a_source_path.parent != user_b_source_path.parent
|
||||
assert user_a_garmin_factory.tokenstore != user_b_garmin_factory.tokenstore
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add isolation acceptance test**
|
||||
|
||||
Configure User B to raise `GarminUploadBlocked`; assert User A still imports and User B ends `action_required` with no impact on User A.
|
||||
|
||||
- [ ] **Step 3: Run the full suite**
|
||||
|
||||
Run: `pytest -v`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 4: Build Docker image again**
|
||||
|
||||
Run: `docker build -t mywhoosh-garmin-sync:test .`
|
||||
|
||||
Expected: successful build with the complete dependency set.
|
||||
|
||||
- [ ] **Step 5: Start local container and exercise smoke paths**
|
||||
|
||||
Start with a temporary bind-mounted `/data`, then verify:
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:18080/healthz
|
||||
curl -I http://127.0.0.1:18080/
|
||||
```
|
||||
|
||||
Expected: health JSON and dashboard redirect to `/login` when unauthenticated.
|
||||
|
||||
- [ ] **Step 6: Verify secrets are absent from captured test logs**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
pytest -v 2>&1 | tee /tmp/mywhoosh-garmin-test.log
|
||||
! grep -F "mw-secret" /tmp/mywhoosh-garmin-test.log
|
||||
! grep -F "garmin-secret" /tmp/mywhoosh-garmin-test.log
|
||||
! grep -F "123456" /tmp/mywhoosh-garmin-test.log
|
||||
```
|
||||
|
||||
Expected: all three negated `grep` commands succeed.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/test_acceptance.py docker-compose.example.yml
|
||||
git commit -m "test: cover multi-user sync acceptance"
|
||||
```
|
||||
620
docs/superpowers/specs/2026-08-15-mywhoosh-garmin-sync-design.md
Normal file
620
docs/superpowers/specs/2026-08-15-mywhoosh-garmin-sync-design.md
Normal file
@@ -0,0 +1,620 @@
|
||||
# MyWhoosh -> Garmin Sync Service — Design
|
||||
|
||||
Date: 2026-08-15
|
||||
Status: Draft for user review
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Build a self-hosted Docker service that periodically checks MyWhoosh for new cycling activities for multiple configured users, downloads each new FIT file, rewrites the device metadata to a Garmin Edge 1030 Plus, and imports the activity into Garmin Connect without forwarding it to Strava.
|
||||
|
||||
The service is administered through a local-only web interface protected by a single admin password.
|
||||
|
||||
## 2. Scope
|
||||
|
||||
### In scope for v1
|
||||
|
||||
- Multiple independent sync users.
|
||||
- Local-only admin UI.
|
||||
- Admin password supplied via environment variable.
|
||||
- SQLite persistence.
|
||||
- Encrypted MyWhoosh and Garmin credentials at rest.
|
||||
- Persistent per-user MyWhoosh and Garmin token stores.
|
||||
- Periodic background sync at a configurable interval.
|
||||
- Manual "sync now" actions.
|
||||
- MyWhoosh activity discovery and FIT download.
|
||||
- Binary FIT metadata patching to Garmin Edge 1030 Plus.
|
||||
- FIT CRC validation and repair.
|
||||
- Garmin Connect import via `import_activity()`.
|
||||
- No intentional forwarding to Strava.
|
||||
- Per-user error isolation, status and retry handling.
|
||||
- Garmin MFA handling through the admin UI when required.
|
||||
- Activity history and sync-run history.
|
||||
|
||||
### Out of scope for v1
|
||||
|
||||
- Public Internet exposure.
|
||||
- Multi-admin or per-user web logins.
|
||||
- OAuth/OIDC for the admin UI.
|
||||
- Editing deployment configuration from the web UI.
|
||||
- Strava integration.
|
||||
- Mobile app.
|
||||
- Distributed workers or external queues.
|
||||
- Automatic CAPTCHA solving or browser automation for MyWhoosh login.
|
||||
|
||||
## 3. High-level architecture
|
||||
|
||||
The application runs as one Docker container with a persistent `/data` volume.
|
||||
|
||||
Components:
|
||||
|
||||
1. FastAPI application.
|
||||
2. Server-rendered Jinja2 admin UI with HTMX for small interactive actions.
|
||||
3. SQLite database.
|
||||
4. Scheduler.
|
||||
5. Sync manager.
|
||||
6. Per-user MyWhoosh client.
|
||||
7. FIT rewriter.
|
||||
8. Per-user Garmin client/uploader.
|
||||
9. Credential encryption service.
|
||||
|
||||
Data flow:
|
||||
|
||||
MyWhoosh -> download FIT -> validate FIT -> patch device metadata -> rewrite CRC -> Garmin `import_activity()` -> persist result.
|
||||
|
||||
## 4. Deployment configuration
|
||||
|
||||
Configuration is supplied through environment variables, for example:
|
||||
|
||||
- `ADMIN_PASSWORD`
|
||||
- `SECRET_KEY`
|
||||
- `CREDENTIAL_ENCRYPTION_KEY`
|
||||
- `SYNC_INTERVAL_MINUTES`
|
||||
- `DATABASE_URL=sqlite:////data/app.db`
|
||||
- `DATA_DIR=/data`
|
||||
|
||||
The web server binds inside the container and is published only to the trusted local network by Docker configuration.
|
||||
|
||||
The web UI does not modify these deployment-level settings.
|
||||
|
||||
## 5. Admin authentication
|
||||
|
||||
The application has one admin login with no username.
|
||||
|
||||
- The password is read from `ADMIN_PASSWORD`.
|
||||
- The password is never persisted in SQLite.
|
||||
- Successful login establishes a signed session using `SECRET_KEY`.
|
||||
- Authentication failures reveal no account details.
|
||||
- Session cookies should be `HttpOnly` and `SameSite=Lax`.
|
||||
- If TLS is later placed in front of the service, `Secure` should be enabled for the cookie.
|
||||
|
||||
Because the service is intended for LAN-only use, v1 does not introduce a separate identity provider.
|
||||
|
||||
## 6. User model
|
||||
|
||||
Each sync user is independent.
|
||||
|
||||
A user contains:
|
||||
|
||||
- id
|
||||
- display name
|
||||
- enabled flag
|
||||
- health state
|
||||
- encrypted MyWhoosh email/password
|
||||
- encrypted Garmin email/password
|
||||
- created/updated timestamps
|
||||
|
||||
Credentials are encrypted before being written to SQLite. The encryption key comes exclusively from `CREDENTIAL_ENCRYPTION_KEY`.
|
||||
|
||||
Stored passwords are never returned to the browser. When editing a user, an empty password field means "keep the existing password".
|
||||
|
||||
## 7. Token storage
|
||||
|
||||
Authentication/session tokens are separated per user.
|
||||
|
||||
Suggested filesystem layout:
|
||||
|
||||
```text
|
||||
/data/
|
||||
app.db
|
||||
tokens/
|
||||
<user-id>/
|
||||
mywhoosh.json
|
||||
garmin/
|
||||
activities/
|
||||
<user-id>/
|
||||
```
|
||||
|
||||
The Garmin tokenstore mechanism from `python-garminconnect` should be reused rather than reimplemented.
|
||||
|
||||
The MyWhoosh token cache stores the access token and, where usable, the refresh token and associated account metadata.
|
||||
|
||||
No user may read or reuse another user's tokenstore.
|
||||
|
||||
## 8. MyWhoosh authentication
|
||||
|
||||
The service should follow the same direct API-login pattern used by `jdelrue/mywhoosh2garmin` rather than automating the MyWhoosh web login page.
|
||||
|
||||
The intended flow is:
|
||||
|
||||
1. Load cached per-user token.
|
||||
2. Attempt an authenticated activities request.
|
||||
3. If accepted, continue.
|
||||
4. If unauthorized, attempt API login using the configured MyWhoosh credentials and the Android-style login payload used by the reference project.
|
||||
5. Persist fresh token data.
|
||||
6. Retry the operation once.
|
||||
|
||||
The implementation must not attempt to defeat or automate CAPTCHA/reCAPTCHA challenges.
|
||||
|
||||
The MyWhoosh endpoints are not treated as a stable public API. Changes in these endpoints should surface as a clear `action_required`/authentication or integration failure rather than causing uncontrolled retries.
|
||||
|
||||
## 9. MyWhoosh activity discovery
|
||||
|
||||
For each enabled user, the MyWhoosh client retrieves recent activities using the authenticated bearer token.
|
||||
|
||||
Each MyWhoosh activity must have a stable external activity identifier. The pair `(user_id, mywhoosh_activity_id)` is unique in SQLite.
|
||||
|
||||
This makes discovery idempotent: the same activity may be returned on every scheduler run but is processed only once unless it previously failed at a retryable stage.
|
||||
|
||||
The client is responsible only for:
|
||||
|
||||
- authentication,
|
||||
- listing activities,
|
||||
- normalizing metadata,
|
||||
- downloading the original FIT file.
|
||||
|
||||
It has no knowledge of Garmin or FIT rewriting.
|
||||
|
||||
## 10. FIT rewriting
|
||||
|
||||
The FIT rewriter follows the binary-patching approach from the existing working Python implementation rather than fully decoding and re-encoding the activity.
|
||||
|
||||
### 10.1 Device identity
|
||||
|
||||
Target device:
|
||||
|
||||
- manufacturer: Garmin (`1`)
|
||||
- product: Edge 1030 Plus (`3570`)
|
||||
- product name: `Edge 1030 Plus`
|
||||
- serial number: optional; if absent, the original serial field is left unchanged unless a later compatibility requirement proves otherwise
|
||||
|
||||
### 10.2 Patched messages
|
||||
|
||||
`file_id` fields when present:
|
||||
|
||||
- manufacturer
|
||||
- product
|
||||
- optional serial number
|
||||
- product name
|
||||
|
||||
`device_info` fields are patched only for the creator device (`device_index == 0`) when a usable device index is present.
|
||||
|
||||
Other sensor/device records should remain unchanged so a trainer, HR sensor or power meter does not become an Edge 1030 Plus accidentally.
|
||||
|
||||
If the source FIT lacks enough information to identify creator-specific `device_info` safely, `file_id` remains mandatory and `device_info` patching should be conservative rather than rewriting all device messages.
|
||||
|
||||
### 10.3 Binary preservation
|
||||
|
||||
The rewriter must preserve all bytes not deliberately changed, except FIT CRC fields.
|
||||
|
||||
It must support:
|
||||
|
||||
- 12-byte and 14-byte FIT headers,
|
||||
- little- and big-endian definition architectures,
|
||||
- compressed timestamp records,
|
||||
- developer fields,
|
||||
- changing local message definitions.
|
||||
|
||||
### 10.4 Validation
|
||||
|
||||
Before patching:
|
||||
|
||||
- validate `.FIT` signature,
|
||||
- validate declared length,
|
||||
- validate header CRC when present,
|
||||
- validate file CRC.
|
||||
|
||||
After patching:
|
||||
|
||||
- rewrite header CRC when present,
|
||||
- rewrite file CRC,
|
||||
- validate the output again,
|
||||
- verify expected target metadata is readable.
|
||||
|
||||
Invalid FIT input is a non-retryable activity error unless the original file is later replaced/redownloaded.
|
||||
|
||||
## 11. Garmin import
|
||||
|
||||
The existing Garmin uploader pattern is reused with `python-garminconnect`.
|
||||
|
||||
The final activity is sent using `import_activity()` rather than `upload_activity()` because the desired behavior is to import into Garmin Connect without intentional onward synchronization to Strava.
|
||||
|
||||
Per user:
|
||||
|
||||
1. Reuse Garmin tokenstore where possible.
|
||||
2. Login/refresh when required.
|
||||
3. Call `import_activity()` with the converted FIT path.
|
||||
4. Record the returned Garmin activity/import identifier if available.
|
||||
5. Treat known duplicate responses as completed `duplicate`, not as fatal failures.
|
||||
|
||||
## 12. Garmin MFA
|
||||
|
||||
MFA is modeled as an explicit user state.
|
||||
|
||||
If Garmin requires MFA and there is no one-time code available:
|
||||
|
||||
- the user's health becomes `action_required`,
|
||||
- that user's Garmin import attempts pause,
|
||||
- other users continue syncing normally,
|
||||
- the dashboard shows that MFA is required.
|
||||
|
||||
The admin can submit the one-time MFA code through the local UI.
|
||||
|
||||
The code:
|
||||
|
||||
- is used only for that login attempt,
|
||||
- is never written to SQLite,
|
||||
- is never written to logs,
|
||||
- is discarded immediately after use.
|
||||
|
||||
On successful authentication the Garmin tokenstore is persisted and the user returns to normal sync behavior.
|
||||
|
||||
## 13. Activity state machine
|
||||
|
||||
An activity progresses through durable stages:
|
||||
|
||||
- `discovered`
|
||||
- `downloaded`
|
||||
- `converted`
|
||||
- `imported`
|
||||
- `duplicate`
|
||||
- `failed`
|
||||
|
||||
Persisted activity fields include:
|
||||
|
||||
- internal id
|
||||
- user id
|
||||
- MyWhoosh activity id
|
||||
- activity date/time
|
||||
- activity name
|
||||
- original FIT path
|
||||
- converted FIT path
|
||||
- current status
|
||||
- Garmin activity/import id when known
|
||||
- last error
|
||||
- created/updated timestamps
|
||||
|
||||
Completed terminal states are `imported` and `duplicate`.
|
||||
|
||||
A failure must retain the latest successfully completed stage so a retry can resume without repeating unnecessary work.
|
||||
|
||||
## 14. Sync-run model
|
||||
|
||||
Each user sync invocation creates a sync-run record containing:
|
||||
|
||||
- id
|
||||
- user id
|
||||
- start time
|
||||
- finish time
|
||||
- status (`running`, `success`, `partial`, `failed`)
|
||||
- discovered count
|
||||
- imported count
|
||||
- skipped count
|
||||
- failed count
|
||||
- summary error when relevant
|
||||
|
||||
Detailed application logs remain on stdout; SQLite stores only UI-relevant summaries.
|
||||
|
||||
## 15. Scheduler and concurrency
|
||||
|
||||
A central scheduler triggers every `SYNC_INTERVAL_MINUTES`.
|
||||
|
||||
On each tick:
|
||||
|
||||
1. Load enabled users.
|
||||
2. Schedule one sync job per user.
|
||||
3. Allow different users to run concurrently.
|
||||
4. Enforce at most one active sync per user with a per-user lock.
|
||||
|
||||
Manual "sync now" uses exactly the same sync pipeline and the same lock.
|
||||
|
||||
If a manual request arrives while that user is already syncing, the application should return a clear "already running" result rather than start another run.
|
||||
|
||||
The scheduler must not block because one account is slow, broken or waiting for user action.
|
||||
|
||||
## 16. Retry policy
|
||||
|
||||
Retries depend on failure type.
|
||||
|
||||
### Retryable automatically
|
||||
|
||||
- transient network errors
|
||||
- timeouts
|
||||
- temporary MyWhoosh/Garmin server errors
|
||||
- expired session/token after one reauthentication attempt
|
||||
|
||||
Within one sync run, use at most a small bounded retry (for example one retry). Further retry occurs on the next scheduler tick.
|
||||
|
||||
### Action required
|
||||
|
||||
- invalid MyWhoosh credentials
|
||||
- MyWhoosh login/API behavior changed in a way that prevents authentication
|
||||
- Garmin MFA required
|
||||
- invalid Garmin credentials
|
||||
|
||||
### Non-retryable per activity
|
||||
|
||||
- corrupt/invalid FIT file
|
||||
- unsupported FIT structure that cannot be safely patched
|
||||
|
||||
The admin UI can expose an explicit "retry" action for failed activities after the underlying issue is fixed.
|
||||
|
||||
## 17. User health state
|
||||
|
||||
Each user has a concise operational state:
|
||||
|
||||
- `healthy`
|
||||
- `syncing`
|
||||
- `degraded`
|
||||
- `action_required`
|
||||
- `disabled`
|
||||
|
||||
This state is derived from configuration and recent sync/authentication outcomes and is shown prominently on the dashboard.
|
||||
|
||||
## 18. Admin UI
|
||||
|
||||
### 18.1 Login
|
||||
|
||||
Single password field and submit action.
|
||||
|
||||
### 18.2 Dashboard
|
||||
|
||||
Shows all users with:
|
||||
|
||||
- name
|
||||
- health state
|
||||
- MyWhoosh connection state
|
||||
- Garmin connection state
|
||||
- last sync
|
||||
- last imported activity
|
||||
- primary error/action if any
|
||||
- "sync now"
|
||||
- "details"
|
||||
- MFA action when needed
|
||||
|
||||
Includes "add account".
|
||||
|
||||
### 18.3 User create/edit
|
||||
|
||||
Fields:
|
||||
|
||||
- display name
|
||||
- MyWhoosh email
|
||||
- MyWhoosh password
|
||||
- Garmin email
|
||||
- Garmin password
|
||||
- enabled flag
|
||||
|
||||
Actions:
|
||||
|
||||
- save
|
||||
- test connection
|
||||
|
||||
Existing passwords are never rendered back to the browser.
|
||||
|
||||
### 18.4 User details
|
||||
|
||||
Shows:
|
||||
|
||||
- current connection and health states
|
||||
- most recent sync-run summary
|
||||
- recent activities and status
|
||||
- latest errors
|
||||
|
||||
Actions:
|
||||
|
||||
- sync now
|
||||
- retry failed activity
|
||||
- enter Garmin MFA when required
|
||||
|
||||
### 18.5 System page
|
||||
|
||||
Read-only operational information:
|
||||
|
||||
- application version
|
||||
- configured sync interval
|
||||
- last scheduler tick
|
||||
- next expected tick
|
||||
- account count
|
||||
- activity count
|
||||
|
||||
Action:
|
||||
|
||||
- sync all now
|
||||
|
||||
## 19. UI technology
|
||||
|
||||
Use:
|
||||
|
||||
- FastAPI
|
||||
- Jinja2
|
||||
- HTMX
|
||||
- small application-specific CSS
|
||||
|
||||
Do not introduce Angular, React, Tailwind or Bootstrap for v1 unless requirements change.
|
||||
|
||||
HTMX is used for bounded actions such as:
|
||||
|
||||
- sync now
|
||||
- test connection
|
||||
- submit MFA
|
||||
- retry activity
|
||||
|
||||
The application remains server-rendered and easy to operate as one container.
|
||||
|
||||
## 20. Security requirements
|
||||
|
||||
- Never log passwords, bearer tokens, session tokens, encryption keys or MFA codes.
|
||||
- Encrypt stored MyWhoosh and Garmin credentials.
|
||||
- Keep tokenstores under the persistent data directory with restrictive filesystem permissions where possible.
|
||||
- Escape all user-visible data rendered into HTML.
|
||||
- Protect state-changing web requests against CSRF.
|
||||
- Validate all IDs against the authenticated admin session rather than trusting client-provided paths blindly.
|
||||
- Use prepared/ORM parameterized database access.
|
||||
- Do not expose decrypted credentials through API responses or templates.
|
||||
|
||||
## 21. Cleanup and retention
|
||||
|
||||
For v1, original and converted FIT files are retained because they are valuable for debugging failed imports.
|
||||
|
||||
Automated retention/cleanup can be added later after operating behavior is known.
|
||||
|
||||
## 22. Error isolation
|
||||
|
||||
Failure of one user must never prevent other users from syncing.
|
||||
|
||||
Examples:
|
||||
|
||||
- User A imports successfully while User B requires Garmin MFA.
|
||||
- User C may have invalid MyWhoosh credentials without affecting scheduler execution for A or B.
|
||||
- A corrupt activity file affects only that activity and user.
|
||||
|
||||
## 23. Testing strategy
|
||||
|
||||
### FIT rewriter tests
|
||||
|
||||
- valid 12-byte header FIT
|
||||
- valid 14-byte header FIT
|
||||
- invalid header CRC
|
||||
- invalid file CRC
|
||||
- malformed/truncated definitions
|
||||
- developer fields preserved
|
||||
- compressed timestamp records handled
|
||||
- Edge 1030 Plus manufacturer/product patched correctly
|
||||
- non-creator `device_info` unchanged
|
||||
- output CRC valid
|
||||
- bytes outside expected metadata and CRC locations unchanged
|
||||
|
||||
### MyWhoosh client tests
|
||||
|
||||
Use mocked HTTP responses for:
|
||||
|
||||
- valid cached token
|
||||
- expired token followed by successful login
|
||||
- invalid credentials
|
||||
- transient server error
|
||||
- activity listing
|
||||
- FIT download
|
||||
|
||||
Do not make live MyWhoosh requests in the normal unit test suite.
|
||||
|
||||
### Garmin uploader tests
|
||||
|
||||
Use a fake/protocol-compatible Garmin client for:
|
||||
|
||||
- tokenstore login
|
||||
- successful import
|
||||
- duplicate
|
||||
- MFA required
|
||||
- invalid login
|
||||
- transient import failure
|
||||
|
||||
### Sync manager tests
|
||||
|
||||
- new activity full happy path
|
||||
- discovered activity is not duplicated
|
||||
- resume from downloaded
|
||||
- resume from converted
|
||||
- retry after transient Garmin failure
|
||||
- one user's failure does not affect another
|
||||
- per-user lock prevents concurrent duplicate sync
|
||||
|
||||
### Web tests
|
||||
|
||||
- admin login success/failure
|
||||
- unauthenticated routes redirect/reject
|
||||
- create/edit/disable user
|
||||
- password never returned
|
||||
- manual sync action
|
||||
- MFA submission lifecycle
|
||||
- CSRF on mutating requests
|
||||
|
||||
## 24. Suggested module boundaries
|
||||
|
||||
```text
|
||||
app/
|
||||
main.py
|
||||
config.py
|
||||
|
||||
auth/
|
||||
admin.py
|
||||
|
||||
db/
|
||||
models.py
|
||||
session.py
|
||||
repositories.py
|
||||
|
||||
security/
|
||||
credentials.py
|
||||
|
||||
mywhoosh/
|
||||
client.py
|
||||
models.py
|
||||
tokenstore.py
|
||||
|
||||
fit/
|
||||
rewriter.py
|
||||
crc.py
|
||||
models.py
|
||||
|
||||
garmin/
|
||||
uploader.py
|
||||
|
||||
sync/
|
||||
manager.py
|
||||
scheduler.py
|
||||
states.py
|
||||
|
||||
web/
|
||||
routes.py
|
||||
forms.py
|
||||
templates/
|
||||
static/
|
||||
|
||||
tests/
|
||||
```
|
||||
|
||||
Each unit should depend on explicit interfaces/protocols where external services are involved so tests do not require live accounts.
|
||||
|
||||
## 25. Primary design decisions
|
||||
|
||||
1. One local admin instead of user-facing authentication.
|
||||
2. Multiple independent sync accounts.
|
||||
3. SQLite for durable application state.
|
||||
4. Environment variables for deployment secrets/configuration.
|
||||
5. Encrypted service credentials at rest.
|
||||
6. Per-user tokenstores.
|
||||
7. Direct MyWhoosh API login pattern; no CAPTCHA automation.
|
||||
8. Binary FIT patching rather than decode/re-encode.
|
||||
9. Garmin Edge 1030 Plus product ID `3570`.
|
||||
10. Conservative creator-device patching.
|
||||
11. Garmin `import_activity()` for Garmin-only import behavior.
|
||||
12. FastAPI + Jinja2 + HTMX for a small single-container admin UI.
|
||||
13. Parallel sync across users, serialized sync within each user.
|
||||
14. Durable activity stages for resumable/idempotent sync.
|
||||
|
||||
## 26. Acceptance criteria for v1
|
||||
|
||||
The system is ready for v1 when:
|
||||
|
||||
1. It runs from Docker with persistent `/data` storage.
|
||||
2. The admin can log in locally using the environment-configured password.
|
||||
3. The admin can add at least two independent users.
|
||||
4. Each user can authenticate independently to MyWhoosh and Garmin.
|
||||
5. New MyWhoosh activities are discovered automatically on schedule.
|
||||
6. FIT files are downloaded and patched to Garmin Edge 1030 Plus metadata with valid CRCs.
|
||||
7. Converted activities are imported into the corresponding Garmin Connect account using `import_activity()`.
|
||||
8. Already processed activities are not imported again.
|
||||
9. Garmin MFA for one user can be resolved through the UI and does not block other users.
|
||||
10. A failure for one user or one activity does not stop the scheduler.
|
||||
11. The dashboard shows current state, recent syncs and actionable errors.
|
||||
12. Secrets and MFA codes do not appear in logs or browser responses.
|
||||
Reference in New Issue
Block a user