Kinect camera sensor-based pointing motion and magnification function: ARETL, France 2021

The sudden shift of schools to distance learning due to the COVID-19 pandemic has also impacted hospitalized children with severe medical, physical, or neurological conditions. We developed a system that enables children to attend classes using an alter-ego robot that lets them virtually experience an actual classroom environment. However, due to the inability of existing cameras and monitors to automatically adjust and magnify pivotal information that teachers present, another challenge is to draw children’s attention and increase or maintain their concentration and focus during remote classes. Thus, we proposed and developed a system that automatically magnifies the view around where a pointing motion is detected. In this study, we described the design of our proposed camera-sensor-based pointing motion recognition with magnification function system using Kinect, which consists of an RGB (color) camera, an IR (infrared) camera, an emitter-based Depth sensor, and four microphone arrays. We also conducted multiple experiments to extract pointing motions and evaluate the ability of the system to detect them using different parameters such as magnification rate, change-of-position tracking rate, and reset-position rate, which were consequently used for Semantic Differential analysis. The optimal conditions and algorithms (e.g., upper body limb positioning, camera-blackboard-body distance, height measurements, color dimensions, user-monitor distance, etc.) were also investigated for system stability.