26 KiB
Sync Engine, Scheduler, and Operational UI Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Integrate the database, MyWhoosh client, FIT rewriter, Garmin importer, scheduler, MFA workflow, retry behavior, and operational admin pages into a resilient multi-user sync service.
Architecture: A SyncManager owns per-user asyncio.Lock instances and executes a durable activity state machine. External clients are injected via factories for tests. A lightweight FastAPI lifespan scheduler triggers syncs at the configured interval; different users run concurrently, while each user's pipeline is serialized.
Tech Stack: Python 3.12, asyncio, FastAPI lifespan, SQLAlchemy, HTMX, Jinja2, pytest/pytest-asyncio.
Global Constraints
- Multiple users sync independently and may run concurrently.
- At most one sync may run for a given user at a time.
- Manual sync and scheduled sync use the same pipeline and lock.
- Durable activity stages are
discovered,downloaded,converted,imported,duplicate,failedwithlast_completed_stageretained on failure. importedandduplicateare terminal.- Transient network/server failures retry at most once in a run; later attempts occur on future scheduler ticks.
- Invalid MyWhoosh/Garmin credentials and Garmin MFA set
action_required. - Corrupt/unsupported FIT is non-retryable per activity.
- Failure of one user or one activity must never stop other users.
- Original and converted FIT files remain on disk in v1.
- MFA codes are never persisted or logged.
File Structure
app/sync/
__init__.py
states.py
manager.py
scheduler.py
app/web/
operations.py
templates/
dashboard.html
users/detail.html
system.html
fragments/user_card.html
fragments/sync_result.html
fragments/mfa_form.html
app/db/
repositories.py
tests/sync/
fakes.py
test_manager.py
test_concurrency.py
test_scheduler.py
tests/web/
test_operations.py
test_mfa.py
Task 1: Add durable activity/sync-run repository operations
Files:
- Modify:
app/db/repositories.py - Create:
tests/db/test_sync_state.py
Interfaces:
-
Produces methods to advance activity stages, mark failures without losing
last_completed_stage, list pending activities, and create/finalize sync runs. -
Step 1: Write failing state-transition tests
# tests/db/test_sync_state.py
from app.db.models import ActivityStatus
def test_failure_retains_last_completed_stage(activity_repository, seeded_activity) -> None:
activity_repository.mark_downloaded(seeded_activity.id, "/data/activities/1/a/source.fit")
activity_repository.mark_failed(seeded_activity.id, "Garmin timeout", retryable=True)
activity = activity_repository.get(seeded_activity.id)
assert activity.status == ActivityStatus.FAILED
assert activity.last_completed_stage == ActivityStatus.DOWNLOADED
assert activity.retryable is True
def test_converted_activity_is_pending_until_terminal(activity_repository, seeded_activity) -> None:
activity_repository.mark_converted(seeded_activity.id, "/data/activities/1/a/converted.fit")
ids = [item.id for item in activity_repository.list_pending_for_user(seeded_activity.user_id)]
assert seeded_activity.id in ids
- Step 2: Run and verify failure
Run: pytest tests/db/test_sync_state.py -v
Expected: missing repository methods.
- Step 3: Implement explicit transition methods
Add methods with these exact effects:
def mark_downloaded(self, activity_id: int, path: str) -> Activity:
activity = self._require(activity_id)
activity.source_fit_path = path
activity.status = ActivityStatus.DOWNLOADED
activity.last_completed_stage = ActivityStatus.DOWNLOADED
activity.last_error = None
activity.retryable = True
self.session.commit()
return activity
def mark_converted(self, activity_id: int, path: str) -> Activity:
activity = self._require(activity_id)
activity.converted_fit_path = path
activity.status = ActivityStatus.CONVERTED
activity.last_completed_stage = ActivityStatus.CONVERTED
activity.last_error = None
activity.retryable = True
self.session.commit()
return activity
def mark_imported(self, activity_id: int, garmin_activity_id: str | None) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.IMPORTED
activity.last_completed_stage = ActivityStatus.IMPORTED
activity.garmin_activity_id = garmin_activity_id
activity.last_error = None
activity.retryable = False
self.session.commit()
return activity
def mark_duplicate(self, activity_id: int) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.DUPLICATE
activity.last_completed_stage = ActivityStatus.DUPLICATE
activity.last_error = None
activity.retryable = False
self.session.commit()
return activity
def mark_failed(self, activity_id: int, error: str, *, retryable: bool) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.FAILED
activity.last_error = error[:2000]
activity.retryable = retryable
self.session.commit()
return activity
list_pending_for_user() must exclude terminal states and include failed rows only when retryable=True.
- Step 4: Add
SyncRunRepositorycreate/finalize methods
start(user_id) creates RUNNING; finish(...) sets counts, finished_at, status, and optional summary error. Do not store exception tracebacks in SQLite.
- Step 5: Run DB state tests
Run: pytest tests/db/test_sync_state.py -v
Expected: PASS.
- Step 6: Commit
git add app/db/repositories.py tests/db/test_sync_state.py
git commit -m "feat: add durable sync state transitions"
Task 2: Implement the single-user sync state machine
Files:
- Create:
app/sync/states.py - Create:
app/sync/manager.py - Create:
tests/sync/fakes.py - Create:
tests/sync/test_manager.py
Interfaces:
-
Produces
SyncManager.sync_user(user_id: int, mfa_code: str | None = None) -> SyncOutcome. -
Constructor receives
session_factory,credential_cipher,settings,mywhoosh_factory,garmin_factory, andfit_converter. -
mywhoosh_factory(token_store: MyWhooshTokenStore) -> MyWhooshClient. -
garmin_factory(email: str, password: str, tokenstore: Path) -> GarminUploader. -
fit_converter(source_path: Path, output_path: Path) -> FitConversionResult. -
Step 1: Define result models and fake integration factories
# app/sync/states.py
from dataclasses import dataclass
@dataclass(frozen=True)
class SyncOutcome:
user_id: int
status: str
discovered: int
imported: int
skipped: int
failed: int
message: str | None = None
Implement concrete fakes:
# tests/sync/fakes.py
from app.garmin.uploader import UploadResult
class FakeMyWhooshClient:
def __init__(self, activities, fit_bytes: bytes) -> None:
self.activities = activities
self.fit_bytes = fit_bytes
self.list_calls = 0
self.download_calls = 0
async def list_activities(self, email: str, password: str):
self.list_calls += 1
return list(self.activities)
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
self.download_calls += 1
return self.fit_bytes
class FakeGarminUploader:
def __init__(self, result: UploadResult | None = None, error: Exception | None = None) -> None:
self.result = result or UploadResult("imported", False, "g-1", {"activityId": "g-1"})
self.error = error
self.calls = 0
def import_fit(self, fit_path, mfa_code=None):
self.calls += 1
if self.error is not None:
raise self.error
return self.result
- Step 2: Write the happy-path test
@pytest.mark.asyncio
async def test_new_activity_downloads_converts_and_imports(manager, seeded_user, tmp_path) -> None:
outcome = await manager.sync_user(seeded_user.id)
assert outcome.discovered == 1
assert outcome.imported == 1
assert outcome.failed == 0
activity = load_only_activity(seeded_user.id)
assert activity.status == ActivityStatus.IMPORTED
assert Path(activity.source_fit_path).exists()
assert Path(activity.converted_fit_path).exists()
- Step 3: Write resume tests before implementation
@pytest.mark.asyncio
@pytest.mark.parametrize(
("status", "last_stage", "expected_downloads", "expected_conversions", "expected_imports"),
[
(ActivityStatus.DOWNLOADED, ActivityStatus.DOWNLOADED, 0, 1, 1),
(ActivityStatus.CONVERTED, ActivityStatus.CONVERTED, 0, 0, 1),
(ActivityStatus.IMPORTED, ActivityStatus.IMPORTED, 0, 0, 0),
(ActivityStatus.FAILED, ActivityStatus.CONVERTED, 0, 0, 1),
],
)
async def test_resume_from_durable_stage(
manager_factory, seeded_activity_factory, status, last_stage,
expected_downloads, expected_conversions, expected_imports,
) -> None:
activity = seeded_activity_factory(status=status, last_completed_stage=last_stage, retryable=True)
manager, mywhoosh, converter, garmin = manager_factory(activity)
await manager.sync_user(activity.user_id)
assert mywhoosh.download_calls == expected_downloads
assert converter.calls == expected_conversions
assert garmin.calls == expected_imports
- Step 4: Run and verify failure
Run: pytest tests/sync/test_manager.py -v
Expected: missing manager.
- Step 5: Implement per-activity filesystem layout and state machine
Use paths:
activity_dir = settings.activities_dir / str(user.id) / activity.mywhoosh_activity_id
source_path = activity_dir / "source.fit"
converted_path = activity_dir / "edge-1030-plus.fit"
Create per-user integration instances from decrypted credentials and isolated token paths:
mw_email = self.credential_cipher.decrypt(user.mywhoosh_email_enc)
mw_password = self.credential_cipher.decrypt(user.mywhoosh_password_enc)
garmin_email = self.credential_cipher.decrypt(user.garmin_email_enc)
garmin_password = self.credential_cipher.decrypt(user.garmin_password_enc)
token_dir = self.settings.tokens_dir / str(user.id)
mywhoosh = self.mywhoosh_factory(MyWhooshTokenStore(token_dir / "mywhoosh.json"))
garmin = self.garmin_factory(garmin_email, garmin_password, token_dir / "garmin")
remote_activities = await mywhoosh.list_activities(mw_email, mw_password)
For each remote activity, call get_or_create_discovered(...), then resume from activity.last_completed_stage when activity.status == FAILED; otherwise use activity.status.
Core sequence:
if stage == ActivityStatus.DISCOVERED:
fit_bytes = await mywhoosh.download_fit(remote.activity_file_id, mw_email, mw_password)
activity_dir.mkdir(parents=True, exist_ok=True)
source_path.write_bytes(fit_bytes)
repo.mark_downloaded(activity.id, str(source_path))
if stage in {ActivityStatus.DOWNLOADED}:
fit_converter(source_path, converted_path)
repo.mark_converted(activity.id, str(converted_path))
if stage in {ActivityStatus.CONVERTED}:
upload = await asyncio.to_thread(garmin.import_fit, converted_path, mfa_code)
if upload.duplicate:
repo.mark_duplicate(activity.id)
else:
repo.mark_imported(activity.id, upload.garmin_activity_id)
After each repository transition, update the local stage variable from the returned record so resume behavior is deterministic.
- Step 6: Implement exception mapping
Map exceptions with explicit user connection-state updates:
except MyWhooshTransientError as exc:
user.health_state = HealthState.DEGRADED
user.mywhoosh_state = "error"
repo.mark_failed(activity.id, str(exc), retryable=True)
except MyWhooshAuthError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.mywhoosh_state = "auth_required"
user.action_reason = "mywhoosh_auth_required"
stop_user_run = True
except MyWhooshIntegrationError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.mywhoosh_state = "integration_error"
user.action_reason = "mywhoosh_integration_changed"
stop_user_run = True
except GarminUploadBlocked:
user.health_state = HealthState.ACTION_REQUIRED
user.garmin_state = "mfa_required"
user.action_reason = "garmin_mfa_required"
stop_user_run = True
except GarminAuthError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.garmin_state = "auth_required"
user.action_reason = "garmin_auth_required"
stop_user_run = True
except GarminTransientError as exc:
user.health_state = HealthState.DEGRADED
user.garmin_state = "error"
repo.mark_failed(activity.id, str(exc), retryable=True)
except FitFormatError as exc:
repo.mark_failed(activity.id, str(exc), retryable=False)
On successful MyWhoosh listing set mywhoosh_state="connected"; on successful Garmin import set garmin_state="connected". Persist the user after each connection-state change. Unexpected exceptions mark the run/user degraded and log only exception class plus sanitized message.
- Step 7: Run manager tests
Run: pytest tests/sync/test_manager.py -v
Expected: PASS.
- Step 8: Commit
git add app/sync/states.py app/sync/manager.py tests/sync
git commit -m "feat: add resumable per-user sync pipeline"
Task 3: Add per-user locks and cross-user isolation
Files:
- Modify:
app/sync/manager.py - Create:
tests/sync/test_concurrency.py
Interfaces:
-
Produces
SyncAlreadyRunningand ensures only one activesync_user()call per user. -
Step 1: Write concurrency tests
@pytest.mark.asyncio
async def test_same_user_cannot_run_twice(manager, seeded_user) -> None:
first_started = asyncio.Event()
release_first = asyncio.Event()
manager.test_hooks = SyncTestHooks(first_started=first_started, release=release_first)
first = asyncio.create_task(manager.sync_user(seeded_user.id))
await first_started.wait()
with pytest.raises(SyncAlreadyRunning):
await manager.sync_user(seeded_user.id)
release_first.set()
await first
@pytest.mark.asyncio
async def test_different_users_can_run_concurrently(manager, user_a, user_b) -> None:
results = await asyncio.gather(manager.sync_user(user_a.id), manager.sync_user(user_b.id))
assert {result.user_id for result in results} == {user_a.id, user_b.id}
Do not leave production-only test_hooks; instead inject a fake MyWhoosh client whose list_activities() blocks on test events.
- Step 2: Run and verify failure
Run: pytest tests/sync/test_concurrency.py -v
Expected: same-user duplicate execution is not yet blocked.
- Step 3: Implement lock registry
class SyncAlreadyRunning(RuntimeError):
pass
class SyncManager:
def __init__(...):
self._locks: dict[int, asyncio.Lock] = {}
self._locks_guard = asyncio.Lock()
async def _lock_for(self, user_id: int) -> asyncio.Lock:
async with self._locks_guard:
return self._locks.setdefault(user_id, asyncio.Lock())
async def sync_user(self, user_id: int, mfa_code: str | None = None) -> SyncOutcome:
lock = await self._lock_for(user_id)
if lock.locked():
raise SyncAlreadyRunning(f"sync already running for user {user_id}")
async with lock:
return await self._sync_user_locked(user_id, mfa_code)
- Step 4: Add a
sync_all_enabled()isolation method
async def sync_all_enabled(self) -> list[SyncOutcome | Exception]:
user_ids = self._load_enabled_user_ids()
return await asyncio.gather(
*(self.sync_user(user_id) for user_id in user_ids),
return_exceptions=True,
)
A failure for one user must appear as one list element and must not cancel sibling jobs.
- Step 5: Run concurrency tests
Run: pytest tests/sync/test_concurrency.py -v
Expected: PASS.
- Step 6: Commit
git add app/sync/manager.py tests/sync/test_concurrency.py
git commit -m "feat: isolate concurrent user syncs"
Task 4: Add the periodic scheduler through FastAPI lifespan
Files:
- Create:
app/sync/scheduler.py - Modify:
app/main.py - Create:
tests/sync/test_scheduler.py
Interfaces:
-
Produces
SyncScheduler.start(),stop(),run_once(),last_tick,next_tick. -
Scheduler interval is
Settings.sync_interval_minutes. -
Step 1: Write scheduler test with a short injected interval
@pytest.mark.asyncio
async def test_scheduler_calls_sync_all_and_survives_failure() -> None:
fake = FakeSyncManager(results=[RuntimeError("one user failed")])
scheduler = SyncScheduler(fake, interval_seconds=0.01)
await scheduler.start()
await asyncio.sleep(0.035)
await scheduler.stop()
assert fake.calls >= 2
assert scheduler.last_tick is not None
- Step 2: Run and verify failure
Run: pytest tests/sync/test_scheduler.py -v
Expected: missing scheduler.
- Step 3: Implement scheduler loop
class SyncScheduler:
def __init__(self, manager, *, interval_seconds: float) -> None:
self.manager = manager
self.interval_seconds = interval_seconds
self._task: asyncio.Task | None = None
self._stop = asyncio.Event()
self.last_tick = None
self.next_tick = None
async def run_once(self) -> None:
self.last_tick = datetime.now(timezone.utc)
await self.manager.sync_all_enabled()
self.next_tick = datetime.now(timezone.utc) + timedelta(seconds=self.interval_seconds)
async def _run(self) -> None:
while not self._stop.is_set():
await self.run_once()
try:
await asyncio.wait_for(self._stop.wait(), timeout=self.interval_seconds)
except TimeoutError:
pass
stop() sets the event and awaits the task. Never allow one sync_all_enabled() exception to kill the loop; log it and continue.
- Step 4: Wire into FastAPI lifespan
Build the concrete SyncManager once during app startup, store it on app.state.sync_manager, create SyncScheduler(... interval_minutes * 60), start it, and stop it during lifespan shutdown.
- Step 5: Run scheduler tests
Run: pytest tests/sync/test_scheduler.py -v
Expected: PASS.
- Step 6: Commit
git add app/sync/scheduler.py app/main.py tests/sync/test_scheduler.py
git commit -m "feat: schedule periodic user synchronization"
Task 5: Add dashboard/manual sync/system operational routes
Files:
- Create:
app/web/operations.py - Modify:
app/web/routes.py - Modify:
app/web/templates/dashboard.html - Create:
app/web/templates/system.html - Create:
app/web/templates/fragments/sync_result.html - Create:
tests/web/test_operations.py
Interfaces:
-
Routes:
POST /users/{id}/sync,POST /sync-all,GET /system. -
Manual actions use the same
SyncManagerinstance and lock as the scheduler. -
Step 1: Write manual-sync tests
def test_manual_sync_calls_shared_manager(authenticated_client, fake_sync_manager) -> None:
response = authenticated_client.post(
"/users/1/sync",
data={"csrf_token": authenticated_client.csrf_token},
)
assert response.status_code == 200
assert fake_sync_manager.user_calls == [1]
def test_manual_sync_reports_already_running(authenticated_client, fake_sync_manager) -> None:
fake_sync_manager.raise_already_running = True
response = authenticated_client.post(
"/users/1/sync",
data={"csrf_token": authenticated_client.csrf_token},
)
assert response.status_code == 409
assert "already running" in response.text.lower()
- Step 2: Run and verify failure
Run: pytest tests/web/test_operations.py -v
Expected: routes missing.
- Step 3: Implement routes with admin and CSRF checks
Each state-changing route must execute in this order:
require_admin(request)
validate_csrf(request, csrf_token)
Then call await request.app.state.sync_manager.sync_user(user_id) or sync_all_enabled().
- Step 4: Expand dashboard data
Add a repository projection that contains only safe display fields:
@dataclass(frozen=True)
class UserDashboardRow:
id: int
name: str
enabled: bool
health_state: str
action_reason: str | None
last_sync_at: datetime | None
last_activity_name: str | None
last_activity_status: str | None
def dashboard_rows(self) -> list[UserDashboardRow]:
users = self.list_all()
rows = []
for user in users:
last_run = self.session.scalar(
select(SyncRun).where(SyncRun.user_id == user.id).order_by(SyncRun.started_at.desc()).limit(1)
)
last_activity = self.session.scalar(
select(Activity).where(Activity.user_id == user.id).order_by(Activity.created_at.desc()).limit(1)
)
rows.append(UserDashboardRow(
id=user.id,
name=user.name,
enabled=user.enabled,
health_state=user.health_state.value,
action_reason=user.action_reason,
last_sync_at=last_run.finished_at if last_run else None,
last_activity_name=last_activity.activity_name if last_activity else None,
last_activity_status=last_activity.status.value if last_activity else None,
))
return rows
Pass only these rows to dashboard.html. Render the MFA action only when row.action_reason == "garmin_mfa_required". No decrypted credential is part of this projection.
- Step 5: Implement read-only system page
Expose application version, configured interval, scheduler last_tick and next_tick, user count, and activity count. The only action is a CSRF-protected sync all now POST.
- Step 6: Run tests
Run: pytest tests/web/test_operations.py -v
Expected: PASS.
- Step 7: Commit
git add app/web tests/web/test_operations.py
git commit -m "feat: add operational sync controls"
Task 6: Add Garmin MFA lifecycle and failed-activity retry
Files:
- Modify:
app/web/routes.py - Modify:
app/web/templates/users/detail.html - Create:
app/web/templates/fragments/mfa_form.html - Create:
tests/web/test_mfa.py - Modify:
app/sync/manager.py
Interfaces:
-
Route:
POST /users/{id}/garmin-mfawith one-timecode. -
Route:
POST /activities/{id}/retry. -
MFA code exists only in request memory and the immediate
sync_user(user_id, mfa_code=code)call. -
Step 1: Write MFA lifecycle test
def test_mfa_code_is_used_once_and_not_persisted(authenticated_client, fake_sync_manager, db_session) -> None:
response = authenticated_client.post(
"/users/1/garmin-mfa",
data={"csrf_token": authenticated_client.csrf_token, "code": "123456"},
)
assert response.status_code == 200
assert fake_sync_manager.mfa_calls == [(1, "123456")]
persisted_text = " ".join(str(row) for row in db_session.execute(text("select * from sync_runs")).all())
assert "123456" not in persisted_text
- Step 2: Write retry test for non-terminal failed activity
Assert the route changes a retryable failed activity back to status=last_completed_stage, clears last_error, then calls the user's normal sync. Reject retry for retryable=False with HTTP 409.
- Step 3: Implement MFA route
Validate code as a non-empty short string, never log it, and call:
outcome = await request.app.state.sync_manager.sync_user(user_id, mfa_code=code.strip())
After a successful Garmin login/import, clear action_reason and restore health to healthy or degraded according to the resulting sync outcome.
- Step 4: Implement failed-activity reset operation
Repository method:
def reset_retryable_failure(self, activity_id: int) -> Activity:
activity = self._require(activity_id)
if activity.status != ActivityStatus.FAILED or not activity.retryable:
raise ValueError("activity is not retryable")
activity.status = activity.last_completed_stage
activity.last_error = None
self.session.commit()
return activity
- Step 5: Run MFA/retry tests
Run: pytest tests/web/test_mfa.py tests/web/test_operations.py -v
Expected: PASS.
- Step 6: Commit
git add app/web app/sync/manager.py app/db/repositories.py tests/web/test_mfa.py
git commit -m "feat: handle Garmin MFA and activity retries"
Task 7: End-to-end regression and Docker acceptance
Files:
- Modify:
docker-compose.example.ymlonly if integration exposes a missing runtime configuration - Create:
tests/test_acceptance.py
Interfaces:
-
No new interface; verifies the v1 acceptance criteria with fake external services.
-
Step 1: Add an application-level acceptance test with two users
Build the app with temporary SQLite/data directories and injected fake MyWhoosh/Garmin factories. Seed two enabled users, give each one distinct remote activity IDs, run sync_all_enabled(), and assert:
assert all(result.status == "success" for result in results)
assert count_terminal_activities(user_a.id) == 1
assert count_terminal_activities(user_b.id) == 1
assert user_a_source_path.parent != user_b_source_path.parent
assert user_a_garmin_factory.tokenstore != user_b_garmin_factory.tokenstore
- Step 2: Add isolation acceptance test
Configure User B to raise GarminUploadBlocked; assert User A still imports and User B ends action_required with no impact on User A.
- Step 3: Run the full suite
Run: pytest -v
Expected: PASS.
- Step 4: Build Docker image again
Run: docker build -t mywhoosh-garmin-sync:test .
Expected: successful build with the complete dependency set.
- Step 5: Start local container and exercise smoke paths
Start with a temporary bind-mounted /data, then verify:
curl -fsS http://127.0.0.1:18080/healthz
curl -I http://127.0.0.1:18080/
Expected: health JSON and dashboard redirect to /login when unauthenticated.
- Step 6: Verify secrets are absent from captured test logs
Run:
pytest -v 2>&1 | tee /tmp/mywhoosh-garmin-test.log
! grep -F "mw-secret" /tmp/mywhoosh-garmin-test.log
! grep -F "garmin-secret" /tmp/mywhoosh-garmin-test.log
! grep -F "123456" /tmp/mywhoosh-garmin-test.log
Expected: all three negated grep commands succeed.
- Step 7: Commit
git add tests/test_acceptance.py docker-compose.example.yml
git commit -m "test: cover multi-user sync acceptance"