Heart Rate Estimation for Remote Elderly Monitoring in Retirement Homes
This project involved the development of a non-contact, real-time heart rate estimation (BPM) system designed for use in retirement homes. Its main purpose is to help caregivers quickly identify critical cardiac events, such as a complete loss of heartbeat or sudden, irregular changes in heart rhythm, during routine video-based supervision. Many elderly residents cannot or prefer not to wear conventional biometric sensors, making this approach a practical alternative for discreet, continuous monitoring. Instead of analyzing skin tone, the system detects micro-movements in facial landmarks, capturing the subtle mechanical oscillations of the head and face. These signals are filtered to eliminate unrelated motion and then transformed into heartbeat data. The outcome is a passive, contact-free safety mechanism that strengthens remote oversight and enables faster responses to potentially life-threatening cardiac anomalies.
On this page: Overview, Reflection, Process, Steps, Technical Architecture and Workflow, Challenges & Solutions, Project Demo, Conclusion,
Overview
My Role
I led the end-to-end development of the system as a Machine Learning and Signal Processing Engineer. My responsibilities included:
- Designing a contactless heart rate estimation system based on micro-movements of facial landmarks
- Implementing facial landmark detection and tracking using MediaPipe and OpenCV
- Developing algorithms to prune irrelevant movement and isolate pulse-like oscillations from tracked facial features
- Extracting and analyzing periodic motion signals to compute heart rate in real time
- Building a WebRTC-based pipeline for live video acquisition from remote endpoints
- Using WebSocket technology to stream BPM estimates continuously with low latency
- Testing system performance under real-world conditions, including varying lighting, motion, and webcam quality
- Collaborating with cross-functional teams to ensure integration with existing monitoring dashboards and adherence to privacy and safety requirements
Collaborators
This project was developed in close collaboration with:
Stackeholders – defined system goals based on operational needs within retirement homes and aligned the feature set with safety priorities
Project Manager (PM) – coordinated timelines, deployment phases, and feedback collection from field testing
Elder Care Advisor – provided insights into typical resident behavior, movement patterns, and critical thresholds for heart rate anomalies
Front-End Developer – integrated real-time BPM feedback into the caregiver dashboard and ensured usability for non-technical staff
Duration Time
Total Duration: 7 months across 3 development phases:
2022 (4 months): Initial desktop-based prototype. Focused on designing the facial landmark–based signal extraction pipeline, testing motion-based pulse estimation techniques, and validating feasibility on local video recordings.
2024 (3 months): Final WebRTC-based version. Integrated real-time video streaming, remote processing, and live BPM monitoring through a WebSocket-enabled backend to support deployment in actual retirement home environments.
Client
A private retirement home group focused on enhancing safety and proactive health monitoring through non-intrusive, AI-driven technologies. The organization sought to implement a contact-free solution for detecting cardiac emergencies during remote check-ins, reducing reliance on wearable devices while increasing response speed in critical scenarios.
Completed Date
Completed: Final version delivered and piloted in retirement home environments in mid-2024.
Briefing
In response to the need for unobtrusive health monitoring in elderly care, this project introduced a real-time heart rate estimation system that operates entirely through video. Designed for use in retirement homes, the system enables caregivers to detect sudden cardiac anomalies such as loss of heartbeat or sharp BPM fluctuations without requiring residents to wear any sensors. By tracking micro-movements of facial landmarks and filtering out non-pulse-related motion, the system extracts a clean signal suitable for live BPM monitoring. The solution was developed in two phases: a desktop prototype to validate the motion-based approach, and a WebRTC-powered version for real-time deployment. This innovation adds a vital safety layer to remote oversight, empowering caregivers with timely, automated physiological insights.
Outcomes
I developed a functional, real-time heart rate estimation system integrated into a video-based monitoring platform for retirement home staff. The system captures live webcam input, tracks facial landmarks, and extracts periodic micro-movements to infer BPM without requiring any physical sensors. By optimizing the motion filtering and signal extraction algorithms, I achieved stable pulse detection under common environmental conditions, including minor head movements and non-ideal lighting. The integration with WebRTC and WebSocket enabled seamless, low-latency streaming of BPM data to caregivers during remote check-ins. The result is a contact-free safety mechanism that enhances monitoring coverage, reduces response time to emergencies, and increases confidence in remote elderly supervision practices. The platform is now ready for expansion with features such as anomaly alerting, session logging, and integration with digital health dashboards.
Problem Statements – The 4 Ws
As a Machine Learning Engineer, I applied the “4 Ws – Problem Statements” framework to clearly define and communicate the core challenge driving this project. This approach helped structure the technical and ethical scope of our work, ensuring the development of a solution that was both clinically relevant and technically feasible within a real-time, privacy-conscious context.
Who is affected?
Elderly residents in retirement homes who may experience sudden cardiac events and are often unable or unwilling to use wearable health monitors.
What is the problem?
Traditional methods for monitoring heart rate rely on physical sensors, which can be uncomfortable, impractical, or ignored by elderly individuals, leading to missed detections of serious anomalies such as complete heart stoppage or abrupt changes in rhythm.
Where does the problem occur?
During routine supervision or remote video check-ins where staff rely on observation alone, without access to biometric data or automated alerts.
Why is it important?
Timely detection of cardiac anomalies is critical in elderly care. Without a passive, contact-free system, staff may be unaware of life-threatening conditions until it’s too late, compromising resident safety and emergency response.
Reflection
“Sometimes, there’s no warning before a crisis. Having even a small automated cue, like a missing pulse, can save lives.”
Care staff member during pilot testing
This project was one of the most meaningful applications of my engineering skills. It was not about precision in lab benchmarks, but about building a system that could work in the unpredictability of real life, with elderly residents who may move unpredictably, remain idle for extended periods, or exhibit minimal facial activity. Developing a heartbeat estimation method based solely on facial landmark motion was technically challenging and required careful signal processing to distinguish true pulses from ambient movement. Transitioning from a prototype to a real-time streaming system sharpened my understanding of concurrency, latency, and fail-safe design. Knowing that this tool could one day alert staff to a life-threatening event in time to intervene gave the work a real sense of urgency and purpose.
Process
From a machine learning and systems engineering perspective, this project was not just about extracting a signal, it was about building a real-time, contact-free system that could reliably detect heart activity in dynamic care environments. I structured my approach around the Double Diamond model, adapting each phase to address both signal accuracy and deployment feasibility
Steps
Please choose the following steps to discover the steps of the project.
1. Discover
This phase was focused on exploring whether heart rate could be inferred from facial landmark dynamics alone. I reviewed existing literature on motion-based physiological signal extraction and remote vital sign monitoring. I also conducted exploratory experiments using recorded videos of elderly subjects to analyze common head movement patterns and how they affect signal stability.
Key output: Verified the potential of landmark-derived motion signals for BPM estimation, and identified challenges like voluntary movement, camera jitter, and frame sampling consistency.
2. Define
Based on early experiments, I defined core system specifications:
• Use facial landmarks as the only input (no color-based or wearable sensors)
• Support real-time inference over video streams (≥ 15 FPS)
• Detect both absence of pulse and significant anomalies in BPM trends
• Output BPM estimates in under one second, with stable smoothing and anomaly tolerance
Key output: A signal processing pipeline design combining landmark tracking, pruning logic, and periodicity analysis, with clear latency and accuracy targets for deployment.
3. Develop
This was the most iterative phase, split across two engineering cycles:
Local prototype – Implemented facial landmark tracking using MediaPipe; extracted raw motion trajectories; applied filters and pruning techniques to isolate biologically plausible oscillations. Developed visualization tools for waveform inspection and BPM estimation validation.
Real-time version – Integrated the system into a WebRTC pipeline for live video capture; used asynchronous queues to decouple inference from streaming; added WebSocket-based data transmission to caregiver dashboards.
Key output: A working system capable of estimating and broadcasting heart rate data from live facial video in under 500ms average delay.
4. Deliver
The final version was deployed in a staging environment and tested with simulated use cases in a retirement home setting. I delivered the complete codebase, model pipeline, system documentation, and integration API to the team. I also proposed enhancements such as BPM anomaly flagging and historical trend visualization.
Key output: A modular, real-time BPM estimation system designed to support further deployment, scaling, and alert integration in elderly care settings.
Technical Architecture and Workflow
The FER system was architected as a modular pipeline optimized for low latency, real-time inference, and ease of deployment. Below is an overview of its core components and how they interact:
Client Side
Video Capture: Runs in a browser or native application using WebRTC; initiates live facial video transmission from the resident’s device.
Consent Interface: Ensures that video monitoring is only activated with authorized caregiver approval, in compliance with ethical standards.
Network Layer
WebRTC + coturn: Enables peer-to-peer, low-latency video streaming; coturn provides NAT traversal support and fallback relaying for restrictive network environments.
WebSocket API: Facilitates real-time transmission of BPM values and system health metadata to the monitoring interface.
Backend (Server Side)
Frame Receiver Module: Extracts individual frames from the live video stream for processing.
Facial Landmark Tracker: Uses MediaPipe to detect and follow facial keypoints across frames with subpixel accuracy.
Signal Processing Engine: Filters out non-pulse-related movement, applies computes heart rate from landmark oscillation patterns.
Anomaly Detection Logic (optional): Flags abrupt drops, absence of signal, or abnormal variability for caregiver attention.
Data Streaming Module: Broadcasts live BPM values over WebSocket in less than 1s intervals.
Deployment
Containerized with Docker
Reverse Proxy via Nginx
Compatible with GPU acceleration (optional)
Prepared for integration with health monitoring dashboards through secured REST endpoints or WebSocket subscriptions
Challenges & Solutions
A project of this scope came with several technical and ethical challenges. Here’s how I addressed them:
Balancing Latency vs. Accuracy
Many elderly individuals cannot tolerate or maintain the use of chest straps, smartwatches, or adhesive sensors — making contact-based BPM monitoring unreliable.
Solution:
I developed a contactless heart rate estimation pipeline using facial landmark movement patterns. The system tracks sub-millimeter oscillations in specific facial regions and converts them into heartbeat signals using motion analysis — eliminating the need for any worn device.
DEMO
“Interactive BPM Showcase — Hosam Zolfonoon Portfolio”
This video presents an interactive BPM (Biometric Pulse Monitoring) demo from Hosam Zolfonoon’s portfolio, showcasing real-time heart rate estimation using facial landmark tracking and signal processing. Designed for elder care and remote health monitoring, it combines AI, MediaPipe, and WebSocket technologies. To request access to a live test demo, please get in touch via: contact@hosamzolfonoon.pt.
Conclusion
This project demonstrates how real-time, contactless technology can enhance safety and responsiveness in elderly care settings. By enabling remote detection of cardiac anomalies without relying on wearable devices, the system offers a new layer of protection for residents in retirement homes. Through facial landmark tracking and motion-based signal extraction, it provides caregivers with live heart rate data and the ability to recognize sudden emergencies such as heart stoppage or dangerous BPM shifts.
From early feasibility tests to full WebRTC integration, each stage was guided by a commitment to unobtrusive monitoring and practical deployment. The final result is a modular, low-latency system that complements existing care routines and extends the reach of supervision without disrupting resident comfort or autonomy. As retirement homes continue to integrate digital tools for proactive health management, this project serves as a blueprint for how machine learning and real-time video can be responsibly applied to life-critical use cases.
Let’s Connect
Are you working on a project that bridges machine learning, real-time systems, or digital health?
Whether you’re building something innovative, looking for a technical collaborator, or just want to exchange ideas — I’d love to hear from you.
Feel free to reach out for a chat about projects, collaborations, or research.
Email me at: contact@hosamzolfonoon.pt
Let’s build technology that truly makes a difference.
