HomeSense
Multimodal smart-home activity-recognition platform: ML research code plus a full web product around it
Interface Gallery (13 Screens)













Sign in provider:The sign-in page, which starts an OpenID Connect authorisation-code flow (PKCE).
System Overview
A model that recognises human activity from smart-home sensor events on the CASAS datasets (Aruba, Milan, Cairo, Kyoto, Orange). Each window of sensor events is turned into four views (a sequence, a behaviour image, a sensor graph and a text description) and fused. Around it sits a product: houses with floor plans and published sensor metadata, validated datasets, model registry with lifecycle checks and shadow runs, batch prediction jobs, a predictions explorer, anomaly detection, a live simulator, and role-based access through OpenID Connect.
Infrastructure & Stack
Python ML core (har_core: sequence, image, graph and text modalities fused into one classifier), FastAPI service, background worker, streaming services (windows, predictor, anomaly, shadow model, sink), MLflow, MinIO, PostgreSQL, React + TypeScript web app, Expo mobile app, a dev OpenID Connect provider
Smart-Home Sensor Playback Lab
Step through synthetic smart-home motion and door sensor events over an apartment floor plan. See how the stream is dynamically projected into four synchronized representation modalities: Sequence, Text, Raster, and Graph for multimodal activity recognition.
- Active (ON/open)
- Inactive
- Current event
The dashed line is an inferred path joining the last few motion activations. It is a spatial trajectory estimate between sensor positions, not continuous camera tracking.
Current event (synthetic)
- Timestamp
- 06:52:10
- Sensor
- M01 (Bedroom motion, Bedroom)
- State
- ON
- Annotation
- Wake upHand-written label for this synthetic day (not model prediction).
Events Stream
Click any event row to jump to that timestamp.
Four Views of the Same Event Window
The window is the last 8 events up to the current one. All four views are computed from it in real-time.
Each event in the window becomes a row of numbers, in time order. A transformer reads this to see order and timing. The current window holds 1 event.
| event | sensor | state | time in window | log gap | hour sin / cos |
|---|---|---|---|---|---|
| 1 | M01 | 1 | 0.00 | 0.00 | 0.97 / -0.23 |
Container Topology & Ports
From the mydocker dashboard: type "mmsar" then "up" (the first start builds many images and takes several minutes), then "open".
Engineering Observations & Notes
The web build expects the identity provider on localhost:9400 and the API on localhost:8000, and the API only allows the browser origin http://localhost:5173. The mydocker dashboard publishes them on 5012 and 4004 and the site on 3005, so signing in through a plain browser fails until those values are rebuilt or configured to match (build-time API/OIDC base URLs and CORS_ORIGINS=http://localhost:3005). To take these screenshots the browser's requests to 9400 and 8000 were redirected to 5012 and 4004 and the CORS headers added at the browser; the application itself was not changed. The signed-in pages above show data that already existed in the local database (five houses, eleven prediction jobs).
Complete Flow & Screen Breakdown

The sign-in page, which starts an OpenID Connect authorisation-code flow (PKCE).

The dev identity provider: choose a predefined user (admin, researcher, viewer) or a custom one and authorise. Roles decide what the app lets you do.

Operations overview: houses, datasets, models and prediction-job counts, and recent prediction jobs.

Houses: five published configurations with floor-plan size and version.

Validated sensor-event collections pinned to published house metadata.

Registered model artifacts matched to house versions.

Model lifecycle: checks, shadow run and promotion.

Batch inference with compatible dataset and model pairs.

Predictions explorer: the activity timeline coloured by predicted activity (opacity follows confidence), class distribution, the average per-modality signal (text, sequence, image, graph) and a window list with evidence.

Unusual patterns raised by the detectors.

Scenarios replayed through the same pipeline as real sensors.

Predictions explorer: the activity timeline coloured by predicted activity (opacity follows confidence), class distribution, the average per-modality signal (text, sequence, image, graph) and a window list with evidence.

One house: live view, floor-plan annotation, published configuration versions (checksum, sensors) and validity.