805 lines
26 KiB
Markdown
805 lines
26 KiB
Markdown
# Sync Engine, Scheduler, and Operational UI Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Integrate the database, MyWhoosh client, FIT rewriter, Garmin importer, scheduler, MFA workflow, retry behavior, and operational admin pages into a resilient multi-user sync service.
|
|
|
|
**Architecture:** A `SyncManager` owns per-user `asyncio.Lock` instances and executes a durable activity state machine. External clients are injected via factories for tests. A lightweight FastAPI lifespan scheduler triggers syncs at the configured interval; different users run concurrently, while each user's pipeline is serialized.
|
|
|
|
**Tech Stack:** Python 3.12, asyncio, FastAPI lifespan, SQLAlchemy, HTMX, Jinja2, pytest/pytest-asyncio.
|
|
|
|
## Global Constraints
|
|
|
|
- Multiple users sync independently and may run concurrently.
|
|
- At most one sync may run for a given user at a time.
|
|
- Manual sync and scheduled sync use the same pipeline and lock.
|
|
- Durable activity stages are `discovered`, `downloaded`, `converted`, `imported`, `duplicate`, `failed` with `last_completed_stage` retained on failure.
|
|
- `imported` and `duplicate` are terminal.
|
|
- Transient network/server failures retry at most once in a run; later attempts occur on future scheduler ticks.
|
|
- Invalid MyWhoosh/Garmin credentials and Garmin MFA set `action_required`.
|
|
- Corrupt/unsupported FIT is non-retryable per activity.
|
|
- Failure of one user or one activity must never stop other users.
|
|
- Original and converted FIT files remain on disk in v1.
|
|
- MFA codes are never persisted or logged.
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
```text
|
|
app/sync/
|
|
__init__.py
|
|
states.py
|
|
manager.py
|
|
scheduler.py
|
|
app/web/
|
|
operations.py
|
|
templates/
|
|
dashboard.html
|
|
users/detail.html
|
|
system.html
|
|
fragments/user_card.html
|
|
fragments/sync_result.html
|
|
fragments/mfa_form.html
|
|
app/db/
|
|
repositories.py
|
|
tests/sync/
|
|
fakes.py
|
|
test_manager.py
|
|
test_concurrency.py
|
|
test_scheduler.py
|
|
tests/web/
|
|
test_operations.py
|
|
test_mfa.py
|
|
```
|
|
|
|
## Task 1: Add durable activity/sync-run repository operations
|
|
|
|
**Files:**
|
|
- Modify: `app/db/repositories.py`
|
|
- Create: `tests/db/test_sync_state.py`
|
|
|
|
**Interfaces:**
|
|
- Produces methods to advance activity stages, mark failures without losing `last_completed_stage`, list pending activities, and create/finalize sync runs.
|
|
|
|
- [ ] **Step 1: Write failing state-transition tests**
|
|
|
|
```python
|
|
# tests/db/test_sync_state.py
|
|
from app.db.models import ActivityStatus
|
|
|
|
|
|
def test_failure_retains_last_completed_stage(activity_repository, seeded_activity) -> None:
|
|
activity_repository.mark_downloaded(seeded_activity.id, "/data/activities/1/a/source.fit")
|
|
activity_repository.mark_failed(seeded_activity.id, "Garmin timeout", retryable=True)
|
|
activity = activity_repository.get(seeded_activity.id)
|
|
|
|
assert activity.status == ActivityStatus.FAILED
|
|
assert activity.last_completed_stage == ActivityStatus.DOWNLOADED
|
|
assert activity.retryable is True
|
|
|
|
|
|
def test_converted_activity_is_pending_until_terminal(activity_repository, seeded_activity) -> None:
|
|
activity_repository.mark_converted(seeded_activity.id, "/data/activities/1/a/converted.fit")
|
|
ids = [item.id for item in activity_repository.list_pending_for_user(seeded_activity.user_id)]
|
|
assert seeded_activity.id in ids
|
|
```
|
|
|
|
- [ ] **Step 2: Run and verify failure**
|
|
|
|
Run: `pytest tests/db/test_sync_state.py -v`
|
|
|
|
Expected: missing repository methods.
|
|
|
|
- [ ] **Step 3: Implement explicit transition methods**
|
|
|
|
Add methods with these exact effects:
|
|
|
|
```python
|
|
def mark_downloaded(self, activity_id: int, path: str) -> Activity:
|
|
activity = self._require(activity_id)
|
|
activity.source_fit_path = path
|
|
activity.status = ActivityStatus.DOWNLOADED
|
|
activity.last_completed_stage = ActivityStatus.DOWNLOADED
|
|
activity.last_error = None
|
|
activity.retryable = True
|
|
self.session.commit()
|
|
return activity
|
|
|
|
|
|
def mark_converted(self, activity_id: int, path: str) -> Activity:
|
|
activity = self._require(activity_id)
|
|
activity.converted_fit_path = path
|
|
activity.status = ActivityStatus.CONVERTED
|
|
activity.last_completed_stage = ActivityStatus.CONVERTED
|
|
activity.last_error = None
|
|
activity.retryable = True
|
|
self.session.commit()
|
|
return activity
|
|
|
|
|
|
def mark_imported(self, activity_id: int, garmin_activity_id: str | None) -> Activity:
|
|
activity = self._require(activity_id)
|
|
activity.status = ActivityStatus.IMPORTED
|
|
activity.last_completed_stage = ActivityStatus.IMPORTED
|
|
activity.garmin_activity_id = garmin_activity_id
|
|
activity.last_error = None
|
|
activity.retryable = False
|
|
self.session.commit()
|
|
return activity
|
|
|
|
|
|
def mark_duplicate(self, activity_id: int) -> Activity:
|
|
activity = self._require(activity_id)
|
|
activity.status = ActivityStatus.DUPLICATE
|
|
activity.last_completed_stage = ActivityStatus.DUPLICATE
|
|
activity.last_error = None
|
|
activity.retryable = False
|
|
self.session.commit()
|
|
return activity
|
|
|
|
|
|
def mark_failed(self, activity_id: int, error: str, *, retryable: bool) -> Activity:
|
|
activity = self._require(activity_id)
|
|
activity.status = ActivityStatus.FAILED
|
|
activity.last_error = error[:2000]
|
|
activity.retryable = retryable
|
|
self.session.commit()
|
|
return activity
|
|
```
|
|
|
|
`list_pending_for_user()` must exclude terminal states and include failed rows only when `retryable=True`.
|
|
|
|
- [ ] **Step 4: Add `SyncRunRepository` create/finalize methods**
|
|
|
|
`start(user_id)` creates `RUNNING`; `finish(...)` sets counts, `finished_at`, status, and optional summary error. Do not store exception tracebacks in SQLite.
|
|
|
|
- [ ] **Step 5: Run DB state tests**
|
|
|
|
Run: `pytest tests/db/test_sync_state.py -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add app/db/repositories.py tests/db/test_sync_state.py
|
|
git commit -m "feat: add durable sync state transitions"
|
|
```
|
|
|
|
## Task 2: Implement the single-user sync state machine
|
|
|
|
**Files:**
|
|
- Create: `app/sync/states.py`
|
|
- Create: `app/sync/manager.py`
|
|
- Create: `tests/sync/fakes.py`
|
|
- Create: `tests/sync/test_manager.py`
|
|
|
|
**Interfaces:**
|
|
- Produces `SyncManager.sync_user(user_id: int, mfa_code: str | None = None) -> SyncOutcome`.
|
|
- Constructor receives `session_factory`, `credential_cipher`, `settings`, `mywhoosh_factory`, `garmin_factory`, and `fit_converter`.
|
|
- `mywhoosh_factory(token_store: MyWhooshTokenStore) -> MyWhooshClient`.
|
|
- `garmin_factory(email: str, password: str, tokenstore: Path) -> GarminUploader`.
|
|
- `fit_converter(source_path: Path, output_path: Path) -> FitConversionResult`.
|
|
|
|
- [ ] **Step 1: Define result models and fake integration factories**
|
|
|
|
```python
|
|
# app/sync/states.py
|
|
from dataclasses import dataclass
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SyncOutcome:
|
|
user_id: int
|
|
status: str
|
|
discovered: int
|
|
imported: int
|
|
skipped: int
|
|
failed: int
|
|
message: str | None = None
|
|
```
|
|
|
|
Implement concrete fakes:
|
|
|
|
```python
|
|
# tests/sync/fakes.py
|
|
from app.garmin.uploader import UploadResult
|
|
|
|
|
|
class FakeMyWhooshClient:
|
|
def __init__(self, activities, fit_bytes: bytes) -> None:
|
|
self.activities = activities
|
|
self.fit_bytes = fit_bytes
|
|
self.list_calls = 0
|
|
self.download_calls = 0
|
|
|
|
async def list_activities(self, email: str, password: str):
|
|
self.list_calls += 1
|
|
return list(self.activities)
|
|
|
|
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
|
|
self.download_calls += 1
|
|
return self.fit_bytes
|
|
|
|
|
|
class FakeGarminUploader:
|
|
def __init__(self, result: UploadResult | None = None, error: Exception | None = None) -> None:
|
|
self.result = result or UploadResult("imported", False, "g-1", {"activityId": "g-1"})
|
|
self.error = error
|
|
self.calls = 0
|
|
|
|
def import_fit(self, fit_path, mfa_code=None):
|
|
self.calls += 1
|
|
if self.error is not None:
|
|
raise self.error
|
|
return self.result
|
|
```
|
|
|
|
- [ ] **Step 2: Write the happy-path test**
|
|
|
|
```python
|
|
@pytest.mark.asyncio
|
|
async def test_new_activity_downloads_converts_and_imports(manager, seeded_user, tmp_path) -> None:
|
|
outcome = await manager.sync_user(seeded_user.id)
|
|
|
|
assert outcome.discovered == 1
|
|
assert outcome.imported == 1
|
|
assert outcome.failed == 0
|
|
activity = load_only_activity(seeded_user.id)
|
|
assert activity.status == ActivityStatus.IMPORTED
|
|
assert Path(activity.source_fit_path).exists()
|
|
assert Path(activity.converted_fit_path).exists()
|
|
```
|
|
|
|
- [ ] **Step 3: Write resume tests before implementation**
|
|
|
|
```python
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
("status", "last_stage", "expected_downloads", "expected_conversions", "expected_imports"),
|
|
[
|
|
(ActivityStatus.DOWNLOADED, ActivityStatus.DOWNLOADED, 0, 1, 1),
|
|
(ActivityStatus.CONVERTED, ActivityStatus.CONVERTED, 0, 0, 1),
|
|
(ActivityStatus.IMPORTED, ActivityStatus.IMPORTED, 0, 0, 0),
|
|
(ActivityStatus.FAILED, ActivityStatus.CONVERTED, 0, 0, 1),
|
|
],
|
|
)
|
|
async def test_resume_from_durable_stage(
|
|
manager_factory, seeded_activity_factory, status, last_stage,
|
|
expected_downloads, expected_conversions, expected_imports,
|
|
) -> None:
|
|
activity = seeded_activity_factory(status=status, last_completed_stage=last_stage, retryable=True)
|
|
manager, mywhoosh, converter, garmin = manager_factory(activity)
|
|
await manager.sync_user(activity.user_id)
|
|
assert mywhoosh.download_calls == expected_downloads
|
|
assert converter.calls == expected_conversions
|
|
assert garmin.calls == expected_imports
|
|
```
|
|
|
|
- [ ] **Step 4: Run and verify failure**
|
|
|
|
Run: `pytest tests/sync/test_manager.py -v`
|
|
|
|
Expected: missing manager.
|
|
|
|
- [ ] **Step 5: Implement per-activity filesystem layout and state machine**
|
|
|
|
Use paths:
|
|
|
|
```python
|
|
activity_dir = settings.activities_dir / str(user.id) / activity.mywhoosh_activity_id
|
|
source_path = activity_dir / "source.fit"
|
|
converted_path = activity_dir / "edge-1030-plus.fit"
|
|
```
|
|
|
|
Create per-user integration instances from decrypted credentials and isolated token paths:
|
|
|
|
```python
|
|
mw_email = self.credential_cipher.decrypt(user.mywhoosh_email_enc)
|
|
mw_password = self.credential_cipher.decrypt(user.mywhoosh_password_enc)
|
|
garmin_email = self.credential_cipher.decrypt(user.garmin_email_enc)
|
|
garmin_password = self.credential_cipher.decrypt(user.garmin_password_enc)
|
|
|
|
token_dir = self.settings.tokens_dir / str(user.id)
|
|
mywhoosh = self.mywhoosh_factory(MyWhooshTokenStore(token_dir / "mywhoosh.json"))
|
|
garmin = self.garmin_factory(garmin_email, garmin_password, token_dir / "garmin")
|
|
remote_activities = await mywhoosh.list_activities(mw_email, mw_password)
|
|
```
|
|
|
|
For each remote activity, call `get_or_create_discovered(...)`, then resume from `activity.last_completed_stage` when `activity.status == FAILED`; otherwise use `activity.status`.
|
|
|
|
Core sequence:
|
|
|
|
```python
|
|
if stage == ActivityStatus.DISCOVERED:
|
|
fit_bytes = await mywhoosh.download_fit(remote.activity_file_id, mw_email, mw_password)
|
|
activity_dir.mkdir(parents=True, exist_ok=True)
|
|
source_path.write_bytes(fit_bytes)
|
|
repo.mark_downloaded(activity.id, str(source_path))
|
|
|
|
if stage in {ActivityStatus.DOWNLOADED}:
|
|
fit_converter(source_path, converted_path)
|
|
repo.mark_converted(activity.id, str(converted_path))
|
|
|
|
if stage in {ActivityStatus.CONVERTED}:
|
|
upload = await asyncio.to_thread(garmin.import_fit, converted_path, mfa_code)
|
|
if upload.duplicate:
|
|
repo.mark_duplicate(activity.id)
|
|
else:
|
|
repo.mark_imported(activity.id, upload.garmin_activity_id)
|
|
```
|
|
|
|
After each repository transition, update the local `stage` variable from the returned record so resume behavior is deterministic.
|
|
|
|
- [ ] **Step 6: Implement exception mapping**
|
|
|
|
Map exceptions with explicit user connection-state updates:
|
|
|
|
```python
|
|
except MyWhooshTransientError as exc:
|
|
user.health_state = HealthState.DEGRADED
|
|
user.mywhoosh_state = "error"
|
|
repo.mark_failed(activity.id, str(exc), retryable=True)
|
|
except MyWhooshAuthError as exc:
|
|
user.health_state = HealthState.ACTION_REQUIRED
|
|
user.mywhoosh_state = "auth_required"
|
|
user.action_reason = "mywhoosh_auth_required"
|
|
stop_user_run = True
|
|
except MyWhooshIntegrationError as exc:
|
|
user.health_state = HealthState.ACTION_REQUIRED
|
|
user.mywhoosh_state = "integration_error"
|
|
user.action_reason = "mywhoosh_integration_changed"
|
|
stop_user_run = True
|
|
except GarminUploadBlocked:
|
|
user.health_state = HealthState.ACTION_REQUIRED
|
|
user.garmin_state = "mfa_required"
|
|
user.action_reason = "garmin_mfa_required"
|
|
stop_user_run = True
|
|
except GarminAuthError as exc:
|
|
user.health_state = HealthState.ACTION_REQUIRED
|
|
user.garmin_state = "auth_required"
|
|
user.action_reason = "garmin_auth_required"
|
|
stop_user_run = True
|
|
except GarminTransientError as exc:
|
|
user.health_state = HealthState.DEGRADED
|
|
user.garmin_state = "error"
|
|
repo.mark_failed(activity.id, str(exc), retryable=True)
|
|
except FitFormatError as exc:
|
|
repo.mark_failed(activity.id, str(exc), retryable=False)
|
|
```
|
|
|
|
On successful MyWhoosh listing set `mywhoosh_state="connected"`; on successful Garmin import set `garmin_state="connected"`. Persist the user after each connection-state change. Unexpected exceptions mark the run/user `degraded` and log only exception class plus sanitized message.
|
|
|
|
- [ ] **Step 7: Run manager tests**
|
|
|
|
Run: `pytest tests/sync/test_manager.py -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 8: Commit**
|
|
|
|
```bash
|
|
git add app/sync/states.py app/sync/manager.py tests/sync
|
|
git commit -m "feat: add resumable per-user sync pipeline"
|
|
```
|
|
|
|
## Task 3: Add per-user locks and cross-user isolation
|
|
|
|
**Files:**
|
|
- Modify: `app/sync/manager.py`
|
|
- Create: `tests/sync/test_concurrency.py`
|
|
|
|
**Interfaces:**
|
|
- Produces `SyncAlreadyRunning` and ensures only one active `sync_user()` call per user.
|
|
|
|
- [ ] **Step 1: Write concurrency tests**
|
|
|
|
```python
|
|
@pytest.mark.asyncio
|
|
async def test_same_user_cannot_run_twice(manager, seeded_user) -> None:
|
|
first_started = asyncio.Event()
|
|
release_first = asyncio.Event()
|
|
manager.test_hooks = SyncTestHooks(first_started=first_started, release=release_first)
|
|
|
|
first = asyncio.create_task(manager.sync_user(seeded_user.id))
|
|
await first_started.wait()
|
|
|
|
with pytest.raises(SyncAlreadyRunning):
|
|
await manager.sync_user(seeded_user.id)
|
|
|
|
release_first.set()
|
|
await first
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_different_users_can_run_concurrently(manager, user_a, user_b) -> None:
|
|
results = await asyncio.gather(manager.sync_user(user_a.id), manager.sync_user(user_b.id))
|
|
assert {result.user_id for result in results} == {user_a.id, user_b.id}
|
|
```
|
|
|
|
Do not leave production-only `test_hooks`; instead inject a fake MyWhoosh client whose `list_activities()` blocks on test events.
|
|
|
|
- [ ] **Step 2: Run and verify failure**
|
|
|
|
Run: `pytest tests/sync/test_concurrency.py -v`
|
|
|
|
Expected: same-user duplicate execution is not yet blocked.
|
|
|
|
- [ ] **Step 3: Implement lock registry**
|
|
|
|
```python
|
|
class SyncAlreadyRunning(RuntimeError):
|
|
pass
|
|
|
|
|
|
class SyncManager:
|
|
def __init__(...):
|
|
self._locks: dict[int, asyncio.Lock] = {}
|
|
self._locks_guard = asyncio.Lock()
|
|
|
|
async def _lock_for(self, user_id: int) -> asyncio.Lock:
|
|
async with self._locks_guard:
|
|
return self._locks.setdefault(user_id, asyncio.Lock())
|
|
|
|
async def sync_user(self, user_id: int, mfa_code: str | None = None) -> SyncOutcome:
|
|
lock = await self._lock_for(user_id)
|
|
if lock.locked():
|
|
raise SyncAlreadyRunning(f"sync already running for user {user_id}")
|
|
async with lock:
|
|
return await self._sync_user_locked(user_id, mfa_code)
|
|
```
|
|
|
|
- [ ] **Step 4: Add a `sync_all_enabled()` isolation method**
|
|
|
|
```python
|
|
async def sync_all_enabled(self) -> list[SyncOutcome | Exception]:
|
|
user_ids = self._load_enabled_user_ids()
|
|
return await asyncio.gather(
|
|
*(self.sync_user(user_id) for user_id in user_ids),
|
|
return_exceptions=True,
|
|
)
|
|
```
|
|
|
|
A failure for one user must appear as one list element and must not cancel sibling jobs.
|
|
|
|
- [ ] **Step 5: Run concurrency tests**
|
|
|
|
Run: `pytest tests/sync/test_concurrency.py -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add app/sync/manager.py tests/sync/test_concurrency.py
|
|
git commit -m "feat: isolate concurrent user syncs"
|
|
```
|
|
|
|
## Task 4: Add the periodic scheduler through FastAPI lifespan
|
|
|
|
**Files:**
|
|
- Create: `app/sync/scheduler.py`
|
|
- Modify: `app/main.py`
|
|
- Create: `tests/sync/test_scheduler.py`
|
|
|
|
**Interfaces:**
|
|
- Produces `SyncScheduler.start()`, `stop()`, `run_once()`, `last_tick`, `next_tick`.
|
|
- Scheduler interval is `Settings.sync_interval_minutes`.
|
|
|
|
- [ ] **Step 1: Write scheduler test with a short injected interval**
|
|
|
|
```python
|
|
@pytest.mark.asyncio
|
|
async def test_scheduler_calls_sync_all_and_survives_failure() -> None:
|
|
fake = FakeSyncManager(results=[RuntimeError("one user failed")])
|
|
scheduler = SyncScheduler(fake, interval_seconds=0.01)
|
|
await scheduler.start()
|
|
await asyncio.sleep(0.035)
|
|
await scheduler.stop()
|
|
assert fake.calls >= 2
|
|
assert scheduler.last_tick is not None
|
|
```
|
|
|
|
- [ ] **Step 2: Run and verify failure**
|
|
|
|
Run: `pytest tests/sync/test_scheduler.py -v`
|
|
|
|
Expected: missing scheduler.
|
|
|
|
- [ ] **Step 3: Implement scheduler loop**
|
|
|
|
```python
|
|
class SyncScheduler:
|
|
def __init__(self, manager, *, interval_seconds: float) -> None:
|
|
self.manager = manager
|
|
self.interval_seconds = interval_seconds
|
|
self._task: asyncio.Task | None = None
|
|
self._stop = asyncio.Event()
|
|
self.last_tick = None
|
|
self.next_tick = None
|
|
|
|
async def run_once(self) -> None:
|
|
self.last_tick = datetime.now(timezone.utc)
|
|
await self.manager.sync_all_enabled()
|
|
self.next_tick = datetime.now(timezone.utc) + timedelta(seconds=self.interval_seconds)
|
|
|
|
async def _run(self) -> None:
|
|
while not self._stop.is_set():
|
|
await self.run_once()
|
|
try:
|
|
await asyncio.wait_for(self._stop.wait(), timeout=self.interval_seconds)
|
|
except TimeoutError:
|
|
pass
|
|
```
|
|
|
|
`stop()` sets the event and awaits the task. Never allow one `sync_all_enabled()` exception to kill the loop; log it and continue.
|
|
|
|
- [ ] **Step 4: Wire into FastAPI lifespan**
|
|
|
|
Build the concrete `SyncManager` once during app startup, store it on `app.state.sync_manager`, create `SyncScheduler(... interval_minutes * 60)`, start it, and stop it during lifespan shutdown.
|
|
|
|
- [ ] **Step 5: Run scheduler tests**
|
|
|
|
Run: `pytest tests/sync/test_scheduler.py -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add app/sync/scheduler.py app/main.py tests/sync/test_scheduler.py
|
|
git commit -m "feat: schedule periodic user synchronization"
|
|
```
|
|
|
|
## Task 5: Add dashboard/manual sync/system operational routes
|
|
|
|
**Files:**
|
|
- Create: `app/web/operations.py`
|
|
- Modify: `app/web/routes.py`
|
|
- Modify: `app/web/templates/dashboard.html`
|
|
- Create: `app/web/templates/system.html`
|
|
- Create: `app/web/templates/fragments/sync_result.html`
|
|
- Create: `tests/web/test_operations.py`
|
|
|
|
**Interfaces:**
|
|
- Routes: `POST /users/{id}/sync`, `POST /sync-all`, `GET /system`.
|
|
- Manual actions use the same `SyncManager` instance and lock as the scheduler.
|
|
|
|
- [ ] **Step 1: Write manual-sync tests**
|
|
|
|
```python
|
|
def test_manual_sync_calls_shared_manager(authenticated_client, fake_sync_manager) -> None:
|
|
response = authenticated_client.post(
|
|
"/users/1/sync",
|
|
data={"csrf_token": authenticated_client.csrf_token},
|
|
)
|
|
assert response.status_code == 200
|
|
assert fake_sync_manager.user_calls == [1]
|
|
|
|
|
|
def test_manual_sync_reports_already_running(authenticated_client, fake_sync_manager) -> None:
|
|
fake_sync_manager.raise_already_running = True
|
|
response = authenticated_client.post(
|
|
"/users/1/sync",
|
|
data={"csrf_token": authenticated_client.csrf_token},
|
|
)
|
|
assert response.status_code == 409
|
|
assert "already running" in response.text.lower()
|
|
```
|
|
|
|
- [ ] **Step 2: Run and verify failure**
|
|
|
|
Run: `pytest tests/web/test_operations.py -v`
|
|
|
|
Expected: routes missing.
|
|
|
|
- [ ] **Step 3: Implement routes with admin and CSRF checks**
|
|
|
|
Each state-changing route must execute in this order:
|
|
|
|
```python
|
|
require_admin(request)
|
|
validate_csrf(request, csrf_token)
|
|
```
|
|
|
|
Then call `await request.app.state.sync_manager.sync_user(user_id)` or `sync_all_enabled()`.
|
|
|
|
- [ ] **Step 4: Expand dashboard data**
|
|
|
|
Add a repository projection that contains only safe display fields:
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class UserDashboardRow:
|
|
id: int
|
|
name: str
|
|
enabled: bool
|
|
health_state: str
|
|
action_reason: str | None
|
|
last_sync_at: datetime | None
|
|
last_activity_name: str | None
|
|
last_activity_status: str | None
|
|
|
|
|
|
def dashboard_rows(self) -> list[UserDashboardRow]:
|
|
users = self.list_all()
|
|
rows = []
|
|
for user in users:
|
|
last_run = self.session.scalar(
|
|
select(SyncRun).where(SyncRun.user_id == user.id).order_by(SyncRun.started_at.desc()).limit(1)
|
|
)
|
|
last_activity = self.session.scalar(
|
|
select(Activity).where(Activity.user_id == user.id).order_by(Activity.created_at.desc()).limit(1)
|
|
)
|
|
rows.append(UserDashboardRow(
|
|
id=user.id,
|
|
name=user.name,
|
|
enabled=user.enabled,
|
|
health_state=user.health_state.value,
|
|
action_reason=user.action_reason,
|
|
last_sync_at=last_run.finished_at if last_run else None,
|
|
last_activity_name=last_activity.activity_name if last_activity else None,
|
|
last_activity_status=last_activity.status.value if last_activity else None,
|
|
))
|
|
return rows
|
|
```
|
|
|
|
Pass only these rows to `dashboard.html`. Render the MFA action only when `row.action_reason == "garmin_mfa_required"`. No decrypted credential is part of this projection.
|
|
|
|
- [ ] **Step 5: Implement read-only system page**
|
|
|
|
Expose application version, configured interval, scheduler `last_tick` and `next_tick`, user count, and activity count. The only action is a CSRF-protected `sync all now` POST.
|
|
|
|
- [ ] **Step 6: Run tests**
|
|
|
|
Run: `pytest tests/web/test_operations.py -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 7: Commit**
|
|
|
|
```bash
|
|
git add app/web tests/web/test_operations.py
|
|
git commit -m "feat: add operational sync controls"
|
|
```
|
|
|
|
## Task 6: Add Garmin MFA lifecycle and failed-activity retry
|
|
|
|
**Files:**
|
|
- Modify: `app/web/routes.py`
|
|
- Modify: `app/web/templates/users/detail.html`
|
|
- Create: `app/web/templates/fragments/mfa_form.html`
|
|
- Create: `tests/web/test_mfa.py`
|
|
- Modify: `app/sync/manager.py`
|
|
|
|
**Interfaces:**
|
|
- Route: `POST /users/{id}/garmin-mfa` with one-time `code`.
|
|
- Route: `POST /activities/{id}/retry`.
|
|
- MFA code exists only in request memory and the immediate `sync_user(user_id, mfa_code=code)` call.
|
|
|
|
- [ ] **Step 1: Write MFA lifecycle test**
|
|
|
|
```python
|
|
def test_mfa_code_is_used_once_and_not_persisted(authenticated_client, fake_sync_manager, db_session) -> None:
|
|
response = authenticated_client.post(
|
|
"/users/1/garmin-mfa",
|
|
data={"csrf_token": authenticated_client.csrf_token, "code": "123456"},
|
|
)
|
|
assert response.status_code == 200
|
|
assert fake_sync_manager.mfa_calls == [(1, "123456")]
|
|
|
|
persisted_text = " ".join(str(row) for row in db_session.execute(text("select * from sync_runs")).all())
|
|
assert "123456" not in persisted_text
|
|
```
|
|
|
|
- [ ] **Step 2: Write retry test for non-terminal failed activity**
|
|
|
|
Assert the route changes a retryable failed activity back to `status=last_completed_stage`, clears `last_error`, then calls the user's normal sync. Reject retry for `retryable=False` with HTTP 409.
|
|
|
|
- [ ] **Step 3: Implement MFA route**
|
|
|
|
Validate code as a non-empty short string, never log it, and call:
|
|
|
|
```python
|
|
outcome = await request.app.state.sync_manager.sync_user(user_id, mfa_code=code.strip())
|
|
```
|
|
|
|
After a successful Garmin login/import, clear `action_reason` and restore health to `healthy` or `degraded` according to the resulting sync outcome.
|
|
|
|
- [ ] **Step 4: Implement failed-activity reset operation**
|
|
|
|
Repository method:
|
|
|
|
```python
|
|
def reset_retryable_failure(self, activity_id: int) -> Activity:
|
|
activity = self._require(activity_id)
|
|
if activity.status != ActivityStatus.FAILED or not activity.retryable:
|
|
raise ValueError("activity is not retryable")
|
|
activity.status = activity.last_completed_stage
|
|
activity.last_error = None
|
|
self.session.commit()
|
|
return activity
|
|
```
|
|
|
|
- [ ] **Step 5: Run MFA/retry tests**
|
|
|
|
Run: `pytest tests/web/test_mfa.py tests/web/test_operations.py -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add app/web app/sync/manager.py app/db/repositories.py tests/web/test_mfa.py
|
|
git commit -m "feat: handle Garmin MFA and activity retries"
|
|
```
|
|
|
|
## Task 7: End-to-end regression and Docker acceptance
|
|
|
|
**Files:**
|
|
- Modify: `docker-compose.example.yml` only if integration exposes a missing runtime configuration
|
|
- Create: `tests/test_acceptance.py`
|
|
|
|
**Interfaces:**
|
|
- No new interface; verifies the v1 acceptance criteria with fake external services.
|
|
|
|
- [ ] **Step 1: Add an application-level acceptance test with two users**
|
|
|
|
Build the app with temporary SQLite/data directories and injected fake MyWhoosh/Garmin factories. Seed two enabled users, give each one distinct remote activity IDs, run `sync_all_enabled()`, and assert:
|
|
|
|
```python
|
|
assert all(result.status == "success" for result in results)
|
|
assert count_terminal_activities(user_a.id) == 1
|
|
assert count_terminal_activities(user_b.id) == 1
|
|
assert user_a_source_path.parent != user_b_source_path.parent
|
|
assert user_a_garmin_factory.tokenstore != user_b_garmin_factory.tokenstore
|
|
```
|
|
|
|
- [ ] **Step 2: Add isolation acceptance test**
|
|
|
|
Configure User B to raise `GarminUploadBlocked`; assert User A still imports and User B ends `action_required` with no impact on User A.
|
|
|
|
- [ ] **Step 3: Run the full suite**
|
|
|
|
Run: `pytest -v`
|
|
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 4: Build Docker image again**
|
|
|
|
Run: `docker build -t mywhoosh-garmin-sync:test .`
|
|
|
|
Expected: successful build with the complete dependency set.
|
|
|
|
- [ ] **Step 5: Start local container and exercise smoke paths**
|
|
|
|
Start with a temporary bind-mounted `/data`, then verify:
|
|
|
|
```bash
|
|
curl -fsS http://127.0.0.1:18080/healthz
|
|
curl -I http://127.0.0.1:18080/
|
|
```
|
|
|
|
Expected: health JSON and dashboard redirect to `/login` when unauthenticated.
|
|
|
|
- [ ] **Step 6: Verify secrets are absent from captured test logs**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
pytest -v 2>&1 | tee /tmp/mywhoosh-garmin-test.log
|
|
! grep -F "mw-secret" /tmp/mywhoosh-garmin-test.log
|
|
! grep -F "garmin-secret" /tmp/mywhoosh-garmin-test.log
|
|
! grep -F "123456" /tmp/mywhoosh-garmin-test.log
|
|
```
|
|
|
|
Expected: all three negated `grep` commands succeed.
|
|
|
|
- [ ] **Step 7: Commit**
|
|
|
|
```bash
|
|
git add tests/test_acceptance.py docker-compose.example.yml
|
|
git commit -m "test: cover multi-user sync acceptance"
|
|
```
|