HomeSenseThesis PlatformJune to October 2026Technical Case Study

HomeSense

Multimodal smart-home activity-recognition platform: ML research code plus a full web product around it

Interface Gallery (13 Screens)

Use arrows or dots to navigate
The sign-in page, which starts an OpenID Connect authorisation-code flow (PKCE).The dev identity provider: choose a predefined user (admin, researcher, viewer) or a custom one and authorise. Roles decide what the app lets you do.Operations overview: houses, datasets, models and prediction-job counts, and recent prediction jobs.Houses: five published configurations with floor-plan size and version.Validated sensor-event collections pinned to published house metadata.Registered model artifacts matched to house versions.Model lifecycle: checks, shadow run and promotion.Batch inference with compatible dataset and model pairs.Predictions explorer: the activity timeline coloured by predicted activity (opacity follows confidence), class distribution, the average per-modality signal (text, sequence, image, graph) and a window list with evidence.Unusual patterns raised by the detectors.Scenarios replayed through the same pipeline as real sensors.Predictions explorer: the activity timeline coloured by predicted activity (opacity follows confidence), class distribution, the average per-modality signal (text, sequence, image, graph) and a window list with evidence.One house: live view, floor-plan annotation, published configuration versions (checksum, sensors) and validity.
1/13

Sign in provider:The sign-in page, which starts an OpenID Connect authorisation-code flow (PKCE).

System Overview

A model that recognises human activity from smart-home sensor events on the CASAS datasets (Aruba, Milan, Cairo, Kyoto, Orange). Each window of sensor events is turned into four views (a sequence, a behaviour image, a sensor graph and a text description) and fused. Around it sits a product: houses with floor plans and published sensor metadata, validated datasets, model registry with lifecycle checks and shadow runs, batch prediction jobs, a predictions explorer, anomaly detection, a live simulator, and role-based access through OpenID Connect.

Infrastructure & Stack

Python ML core (har_core: sequence, image, graph and text modalities fused into one classifier), FastAPI service, background worker, streaming services (windows, predictor, anomaly, shadow model, sink), MLflow, MinIO, PostgreSQL, React + TypeScript web app, Expo mobile app, a dev OpenID Connect provider

Python
FastAPI
Kafka
PostgreSQL
React
PyTorch
Kubernetes
Redis
Helm
MLflow
LIVE INTERACTIVE LAB (FROM MYPORTFOLIO)

Smart-Home Sensor Playback Lab

Step through synthetic smart-home motion and door sensor events over an apartment floor plan. See how the stream is dynamically projected into four synchronized representation modalities: Sequence, Text, Raster, and Graph for multimodal activity recognition.

BedroomBathroomKitchenHallwayLiving area01M0102M0203M0304M0405M0506M06D01 closed
  • Active (ON/open)
  • Inactive
  • Current event

The dashed line is an inferred path joining the last few motion activations. It is a spatial trajectory estimate between sensor positions, not continuous camera tracking.

Current event (synthetic)

Timestamp
06:52:10
Sensor
M01 (Bedroom motion, Bedroom)
State
ON
Annotation
Wake upHand-written label for this synthetic day (not model prediction).
Playback speed
Event 1 of 26: 06:52:10, M01 ON
06:52:1007:21:30

Events Stream

Click any event row to jump to that timestamp.

Four Views of the Same Event Window

The window is the last 8 events up to the current one. All four views are computed from it in real-time.

Each event in the window becomes a row of numbers, in time order. A transformer reads this to see order and timing. The current window holds 1 event.

Per-event features for the current window
eventsensorstatetime in windowlog gaphour sin / cos
1M0110.000.000.97 / -0.23

Container Topology & Ports

$ services: web http://localhost:3005, API http://localhost:4004, inference 4005 and shadow inference 4006, MLflow 5011, idp 5012, MinIO 5013 (about fifteen containers)
$ how to run:

From the mydocker dashboard: type "mmsar" then "up" (the first start builds many images and takes several minutes), then "open".

Engineering Observations & Notes

The web build expects the identity provider on localhost:9400 and the API on localhost:8000, and the API only allows the browser origin http://localhost:5173. The mydocker dashboard publishes them on 5012 and 4004 and the site on 3005, so signing in through a plain browser fails until those values are rebuilt or configured to match (build-time API/OIDC base URLs and CORS_ORIGINS=http://localhost:3005). To take these screenshots the browser's requests to 9400 and 8000 were redirected to 5012 and 4004 and the CORS headers added at the browser; the application itself was not changed. The signed-in pages above show data that already existed in the local database (five houses, eleven prediction jobs).

Complete Flow & Screen Breakdown

Sign in provider
1. Sign in provider

The sign-in page, which starts an OpenID Connect authorisation-code flow (PKCE).

Idp login
2. Idp login

The dev identity provider: choose a predefined user (admin, researcher, viewer) or a custom one and authorise. Roles decide what the app lets you do.

Dashboard
3. Dashboard

Operations overview: houses, datasets, models and prediction-job counts, and recent prediction jobs.

Houses
4. Houses

Houses: five published configurations with floor-plan size and version.

Datasets
5. Datasets

Validated sensor-event collections pinned to published house metadata.

Models
6. Models

Registered model artifacts matched to house versions.

Lifecycle
7. Lifecycle

Model lifecycle: checks, shadow run and promotion.

Prediction jobs
8. Prediction jobs

Batch inference with compatible dataset and model pairs.

Predictions
9. Predictions

Predictions explorer: the activity timeline coloured by predicted activity (opacity follows confidence), class distribution, the average per-modality signal (text, sequence, image, graph) and a window list with evidence.

Anomalies
10. Anomalies

Unusual patterns raised by the detectors.

Simulator
11. Simulator

Scenarios replayed through the same pipeline as real sensors.

Predictions timeline
12. Predictions timeline

Predictions explorer: the activity timeline coloured by predicted activity (opacity follows confidence), class distribution, the average per-modality signal (text, sequence, image, graph) and a window list with evidence.

House detail
13. House detail

One house: live view, floor-plan annotation, published configuration versions (checksum, sensors) and validity.