Ruiqi Wang

Minimally obtrusive support for individuals with subjective cognitive decline (SCD) is important for fostering independence in completing daily tasks. In overseeing these tasks, occupational therapists may choose to help as errors arise and provide corrective courses of action. To accomplish this, therapists must be able to recognize task-specific actions, as well as the appropriate sequence for them to occur. However, manual monitoring by therapists is not always feasible in real-world environments, motivating the need for automated systems capable of recognizing actions and detecting sequencing errors. To address this, we present CHEF-VL, an online Cognitive Human Error Detection Framework with Vision-Language Models in smart kitchen environments. CHEF-VL combines two novel vision-language models, with one fine-tuned for online human action recognition and the other specially engineered to track key environmental states. An Action-State Merger integrates these two streams of information to reduce prediction noise and correct misrecognized actions. A two-year occupational therapy project of over 100 participants with and without SCD was organized to collect video data for task evaluation. Empirical results demonstrate that CHEF-VL improves both action recognition and sequencing error detection performance, offering a promising solution for real-world assistive technologies in smart home settings.

Advances in computer vision and deep learning have made video-based Human Action Recognition (HAR) increasingly feasible. However, running HAR on live video streams encounters significant delays on embedded platforms due to computational demands. This work addresses real-time HAR performance challenges through four key contributions: (1) an experimental study identifying standard Optical Flow (OF) extraction as the primary latency bottleneck in a state-of-the-art HAR pipeline, (2) an analysis of the latency-accuracy trade-off between traditional and deep learning-based OF methods, underscoring the need for an efficient motion feature extractor with minimal impact on accuracy, (3) the design of Integrated Motion Feature Extractor (IMFE), a novel unified neural network architecture that substantially reduces motion feature extraction latency, and (4) the development of RT-HARE, a real-time HAR system optimized for embedded platforms. Experiments on three benchmark datasets of various characteristics using the Nvidia Jetson Xavier NX platform demonstrate that RT-HARE achieves real-time HAR with lower and more stable latency, reduced power consumption, and a smaller memory footprint while maintaining recognition accuracy comparable to more complex server-based HAR models.

IoT devices are increasingly the source of data for machine learning (ML) applications running on edge servers. Data transmissions from devices to servers are often over local wireless networks whose bandwidth is not just limited but, more importantly, variable. Furthermore, in cyber-physical systems interacting with the physical environment, image offloading is also commonly subject to timing constraints. It is, therefore, important to develop an adaptive approach that maximizes the inference performance of ML applications under timing constraints and the resource constraints of IoT devices. In this paper, we use image classification as our target application and propose progressive neural compression (PNC) as an efficient solution to this problem. Although neural compression has been used to compress images for different ML applications, existing solutions often produce fixed-size outputs that are unsuitable for timing-constrained offloading over variable bandwidth. To address this limitation, we train a multi-objective rateless autoencoder that optimizes for multiple compression rates via stochastic taildrop to create a compression solution that produces features ordered according to their importance to inference performance. Features are then transmitted in that order based on available bandwidth, with classification ultimately performed using the (sub)set of features received by the deadline. We demonstrate the benefits of PNC over state-of-the-art neural compression approaches and traditional compression methods on a testbed comprising an IoT device and an edge server connected over a wireless network with varying bandwidth.

This article considers a setting where embedded devices are used to acquire and classify images. Because of limited computing capacity, embedded devices rely on a parsimonious classification model with uneven accuracy. When local classification is deemed inaccurate, devices can decide to offload the image to an edge server with a more accurate but resource-intensive model. Resource constraints, e.g., network bandwidth, however, require regulating such transmissions to avoid congestion and high latency. This article investigates this offloading problem when transmissions regulation is through a token bucket, a mechanism commonly used for such purposes. The goal is to devise a lightweight, online offloading policy that optimizes an application-specific metric (e.g., classification accuracy) under the constraints of the token bucket. This article develops a policy based on a deep Q-network (DQN), and demonstrates both its efficacy and the feasibility of its deployment on embedded devices. Of note is the fact that the policy can handle complex input patterns, including correlation in image arrivals and classification accuracy. The evaluation is carried out by performing image classification over a local testbed using synthetic traces generated from the ImageNet image classification benchmark.