Page 6 - Read Online

Page 4 of 11 Kim et al. Soft Sci 2024;4:12 https://dx.doi.org/10.20517/ss.2023.50

pattern and specializes in characterizing spatial derivatives such as deformation and strain. The technique
does not require a high-power illumination source or high-speed imaging, making it suitable for standard
applications with relatively fast processing times. DIC can be easily combined with 3D reconstruction
techniques to achieve 3D stereoscopic deformations on a curved surface, such as the neck and wrist, where
[12]
the associated deformations can be highly three-dimensional . Applications in soft electronics include
validating haptic and strain-related sensors [8,13] .

Particle tracking velocimetry (PTV) is particularly useful for characterizing fluid flows and structure
motions by tracking particles and fiducial points in the Lagrangian frame of reference . Unlike PIV and
[14]
DIC, PTV tracks particles in the global coordinate, allowing for estimating particle trajectories. With a
multicamera setup, PTV can measure in 3D domains. It offers advantages for quantifying flows in drug
delivery systems or validating motion-tracking devices . A relatively recent technique, the Markerless
[15]
[16]
Pose Estimation (MPE), tracks a limited number of objects without fiducial points based on transfer
learning with deep neural networks. The technique is ideal for monitoring animal behaviors, such as
tracking multiple body parts of a mouse during an odor-guided navigation task and a fruit fly behaving in a
3D chamber .
[17]
Eulerian Video Magnification (EVM) , Structure from Motion (SfM) , and area tracking represent
[20]
[18]
[19]
advanced computer vision techniques that have substantial implications for soft electronics. These methods
are particularly advantageous because they circumvent the need for fiducial points and can support
approaches for various soft electronic concepts.

The combination of the above-mentioned computer vision methods could be implemented in skin-
interfaced electronics concerning their functionalities and applications. For instance, PIV could be used to
correlate respiratory flow with signals from the MA sensor [Figure 2A]. PTV would be useful for
quantifying biomarkers requiring high temporal resolution, including vibrations of the neck during speech
and respiratory activities, to address design and placement strategies of a wearable sensor [Figure 2B]. DIC
can quantify mechanics with high spatial resolution, such as strains on the skin induced by haptic actuators
or deformations on the neck during swallowing [Figure 2C]. EVM can enhance subtle motions either
generated by a weak haptic actuator [Supplementary Video 1] or pulse wave velocity for developing a
wearable pulse oximeter. Area tracking and MPE could be implemented to measure the deformation of
organs or implantable devices where adding fiducial markers is difficult. The following sections discuss two
sets of examples of computer vision methods used in skin-interfaced electronics: one involves measuring
various biomarkers to validate a series of MA sensors, and the other involves quantifying the mechanics
induced by various vibrotactile actuators in haptic devices.

COUPLED MECHANICS IN MECHANO-ACOUSTIC SENSORS
Skin-interfaced sensor technologies offer a vast range of multimodal, clinical- and consumer-grade,
continuous monitoring of physiological biomarkers with high accuracy and immunity to external noises in
hospital and in-home settings. The MA device, a thin, soft sensor with a high-bandwidth accelerometer
conformally coupled to the skin, has demonstrated its effectiveness in providing precise measurements of
MA signals from subtle vibrations of the skin (~10 m/s ) to large motions of the entire body (~10 m/s ) .
2 [21]
2
-3
When interfaced at unique anatomical locations, such as the suprasternal notch (SN) at the base of the neck,
this technology offers a rich blend of MA information related to various classes of underlying body and
physiological processes . In conjunction with the coupled biomechanics through computer vision
[22]
methods, advanced versions of the MA system have been developed to monitor bio-signals tailored for