Smart Kitchen

The Smart Kitchen System is an AI-driven assistive technology designed to support individuals with cognitive decline in completing daily kitchen tasks independently and safely.

The Smart Kitchen System is an interdisciplinary initiative integrating engineering and occupational therapy to develop an AI-driven solution that supports independent living for individuals experiencing subjective cognitive decline (SCD).

Definition: Subjective Cognitive Decline (SCD) refers to self-reported reductions in cognitive function, impacting daily life through challenges in attention, memory, and thinking. SCD is a significant public health issue, affecting 1 in 9 people over age 45 and posing risks to independence and safety. For individuals with SCD, especially those living alone, this system aims to foster autonomy and lessen dependence on caregivers by offering timely support during daily kitchen activities.

Significance of the Smart Kitchen System

People with SCD often encounter challenges in managing daily tasks, which can lead to dependency and social isolation. The Smart Kitchen System addresses these issues by providing real-time, AI-driven assistance with kitchen activities, thus promoting safety and independence at home. This technology also reduces the burden on caregivers, embodying digital transformation by offering accessible, customized cues when cognitive challenges arise.

Project Aims

  • Aim 1: Data Collection and Dataset Creation
    High-quality video data will be collected of participants performing the oatmeal preparation task, capturing cognitive breakdowns and task errors. This dataset will enable training of deep learning algorithms to identify and understand performance patterns.

  • Aim 2: Development of Cognitive Performance Breakdown Recognizer and Adaptive Cueing Agent
    Using this dataset, we will develop two core components: the Cognitive Performance Breakdown Recognizer (CPBR), which employs deep human action recognition models to detect cognitive errors in real-time, and the Adaptive Cueing Agent (ACA), which dynamically supports users with context-relevant cues based on detected errors. These tools will leverage advanced models for accurate and responsive task support.

Method

Oatmeal Preparation

Our primary task focuses on oatmeal preparation, serving as a foundational pilot activity for developing the Smart Kitchen System. Participants follow a structured sequence of steps, such as measuring oats, pouring water, stirring, and heating. By monitoring cognitive breakdowns at each stage, the system can provide tailored cues to help participants navigate the task. This setup builds the foundation for a smart kitchen that detects errors and offers supportive cues in real-time using computer vision and deep learning.

Data Collection

Data will be collected in a real kitchen environment, with a webcam mounted above the countertop to capture participants’ actions. The recordings will focus on the oatmeal preparation steps, allowing us to gather detailed, real-world data for training action recognition models. Each video will be annotated with action labels, documenting each step and any cognitive errors. This extensive annotation process will support the development of models that accurately recognize actions and identify performance breakdowns in real-time.

Testing and Deployment

The Smart Kitchen System will undergo initial testing in a controlled lab kitchen to evaluate its accuracy and effectiveness in detecting actions and providing cues. Once validated, the system will be adapted and tested in real home environments to gain insights into its performance under everyday conditions.

To expand its impact, we also plan to create a mobile-friendly version of the Smart Kitchen System, enabling deployment directly on smartphones. This mobile approach has the potential to make assistive technology more accessible and adaptable to various kitchen settings, ultimately helping more individuals maintain their independence at home.

Future Steps

The project will yield a comprehensive dataset of cognitive performance errors during cooking tasks, alongside user profiles that indicate who may benefit most from this assistive technology. Following this phase, we will test the Smart Kitchen System in both lab environments and users’ homes. This project could pave the way for future assistive technologies applicable to various daily tasks beyond cooking, offering an innovative, affordable solution for enhancing independence and safety in individuals with cognitive impairment.

Collaborators

Last Edited:


References