The procedure of our gaze estimation system is as follows [Figure 2A]: Firstly, we employ an improved You Only Look Once (YOLO) model^[27] to locate and crop eye images. We utilize an Intel RealSense depth camera^[28] to calculate the three-dimensional spatial positions of the eyes and mouth. Next, we utilize a multimodal gaze point network to predict screen gaze points. To accomplish this, we need to create and collect two datasets: one for training the YOLO object detection model and another for training the Gaze-Point-Net (GPN) gaze regression model. In our implementation, the resolution of the feature maps is increased in the detection head to improve the recognition of small-sized objects (eyes and mouth regions in the facial images). Moreover, the left-eye and right-eye classes are merged into a single “eye” class, since the two eyes exhibit highly similar appearance characteristics. The eyes and mouth are detected simultaneously during facial landmark detection. While the eye images are used in the subsequent gaze estimation process, the detected mouth position provides an additional facial landmark for characterizing the facial geometry. Combined with the binocular eye positions captured by the Intel RealSense depth camera, the three landmarks (left eye, right eye, and mouth) define a facial plane whose normal vector provides two angular degrees of freedom corresponding to head pitch and yaw in the camera coordinate system. The 3D spatial coordinates of the left and right eyes, together with these two angular components, constitute the 8-dimensional position/posture vector used in this study. Roll is not explicitly represented as an independent angular variable but is implicitly encoded in the relative 3D spatial configuration of the two eyes.

Figure 2B illustrates the training process of GPN. We collected training data from volunteers, consisting of standardized binocular images and corresponding three-dimensional information computed from image depth. These datasets were assembled and utilized to train a universal model.

We trained our model on a large dataset, and to further improve its fit to each user, we designed a calibration function [Figure 2C]. With this function, we can enhance the accuracy of the model’s predictions and deploy the same model on screens of different sizes to maintain consistent task accuracy when the camera position may not be stable.

Traditional gaze estimation tasks are usually performed under the condition of a fixed head, so traditional methods do not need to consider the head’s spatial position and pose information. Since the six degrees of freedom of the head are fixed, neural network models in traditional methods accept only binocular images or use a monocular model and compute binocular images through a mirror-flipping method.

To address the asymmetry between pupil direction and eyelid features in binocular images, we designed GPN based on LeNet, similar to existing studies^[2]. Since binocular images are different, the feature extractors cannot use the same parameters for feature extraction. Therefore, we adopted a double-channel convolutional neural network feature extractor with non-shared parameters to extract features from binocular images and used multiple fully connected layers to compress and extract features. Additionally, we connected (concatenated) head spatial position and pose vectors in the hidden layer to construct a multimodal neural network for completing the multimodal gaze position prediction task. Below is the network structure of GPN [Figure 3].

Our neural network architecture mainly consists of two parts: the SELayer^[29] and GAZE attention. The SELayer is a module used to enhance the neural network’s feature expression ability. It takes the input feature map and performs adaptive average pooling to obtain a global feature vector. After processing through two fully connected layers and an activation function, the SELayer generates a weight vector used to scale the input feature map. The SELayer includes an adaptive average pooling layer, two fully connected layers, a rectified linear unit (ReLU) activation function to construct nonlinear combinations, and a Sigmoid activation function to limit the parameter range.

The GAZE attention is a neural network consisting of convolutional and fully connected layers. Its input includes two monochrome eye images (36 × 60 pixels each) and head position and pose information. The output is a 2D tensor representing the gaze point coordinates.

The convolutional layers of this neural network adopt a non-shared-parameter dual-channel convolutional neural network structure. Each channel contains two convolutional modules. The first convolutional module consists of 20 convolutional kernels with a size of 5 × 5, a stride of 1, and a padding of 2 to keep the feature size the same. This is followed by a batch normalization layer, a ReLU activation function, and a 2 × 2 max-pooling layer. After the first pooling operation, the attention mechanism introduced earlier enhances the network’s focus on the input data. This mechanism adaptively weights each channel, allowing the neural network to focus more on extracting important features while ignoring irrelevant ones. The second convolutional module consists of 50 convolutional kernels with a size of 5 × 5, a stride of 1, and a padding of 2 × 2 to ensure that the feature size remains the same. This is followed by a batch normalization layer, a ReLU activation function, and a maximum pooling layer with a size of 2.

During training, we found that the model underestimated the y-coordinate of the predicted gaze point. Therefore, we redesigned the loss function. Our data and labels are both represented as two-dimensional coordinates, denoted by c1 = (x1, y1) and c2 = (x2, y2), respectively, where c1 and c2 represent the predicted and ground-truth gaze points. Here, x1 and y1 denote the predicted horizontal and vertical coordinates, while x2 and y2 denote the corresponding ground-truth coordinates. The proposed loss function consists of the absolute errors of the x- and y-coordinates, the squared error of the y-coordinate, and the Euclidean distance between the predicted and ground-truth gaze points, weighted by 2:2:1:4, respectively [Equation (1)]. These weighting coefficients were selected empirically through preliminary experiments to balance the contributions of the individual loss terms and promote stable model convergence. The squared y-coordinate error term further emphasizes deviations in the vertical coordinate, particularly at larger y-coordinate values.

(1) $$ L(c1,c2)=2|x1-x2|+2|y1-y2|+|y1-y2|^2+4||(x1,y1)-(x2,y2)||2 $$

A total of 20 participants were randomly divided at the participant level into training, validation, and test sets at an 8:1:1 ratio, comprising 16, 2, and 2 participants, respectively. All samples from each participant were assigned exclusively to the corresponding subset, ensuring that no participant appeared in more than one subset.

During training, we set the number of epochs to 50, the batch size to 16, and used the redesigned loss function as the objective function. To optimize the objective function, we used the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.0001 and a dynamic learning rate strategy. Specifically, we gradually reduced the learning rate by a factor of 0.1 at epochs 20 and 30. After each training batch, we used the validation set to calculate the mean absolute error (MAE) and determine whether to update the best model.

The GPN model was trained using only the data from participants in the training subset, while data from the validation and test subsets were reserved for their respective evaluation purposes. Therefore, we can further improve the model’s accuracy and performance under specific users’ usage conditions through calibration. To this end, we designed the following calibration steps, following a similar approach to that described in^[30].

First, we collected calibration data using a method similar to that used in collecting training data. The calibration data were collected by presenting nine gray boxes arranged in two rectangles on the display screen in sequence. The collected data include standardized binocular images, head position and posture vectors, and corresponding label data. Next, the collected data were assembled into a batch of calibration datasets, and the model was fine-tuned using the same loss function and optimizer as in training. Importantly, model fine-tuning at this stage used exclusively the calibration data from volunteers included in the training set.

During model calibration, to prevent overfitting at the nine calibration points, we employed an approach similar to transfer learning by adjusting only the parameters of the last fully connected layer during fine-tuning. This method enables us to further improve the model’s accuracy and performance under specific user conditions.

Moreover, a mean filter is applied to the calibrated gaze-point predictions. Specifically, the current gaze location is estimated as the mean of the five most recent predicted gaze locations. The filtered and unfiltered predictions are then compared to evaluate the effect of filtering on gaze estimation performance.

We conducted sequential point testing and collected 6,572 sets of experimental data. The results of this testing, along with error analysis at multiple scales, are presented in Figures 5 and 6. Moreover, the performance of the eyes-mouth detection model is evaluated.

During the error analysis, we considered various variables such as screen gaze position, head spatial position, head spatial pose, and lighting conditions. This comprehensive analysis allowed us to identify the factors that contribute to errors in our experimental data. By combining sequential point testing with thorough error analysis, we obtained a comprehensive understanding of our system’s performance and limitations under different conditions.

Figure 5 presents the error analysis for various variables. The horizontal axis represents the variables, while the vertical axis represents the average angular error within the corresponding interval. Figure 5A and B display the relationship between the horizontal and vertical positions of the target point on the screen and the angular error. Figure 5C presents a three-dimensional plot showing the relationship between the target point’s position on the screen and the angular error. The x and y axes represent the horizontal and vertical positions of the target point on the screen, respectively, while the z-axis represents the magnitude of the angular error. The error distribution ranges from 6 to 10 degrees.

Figure 5D illustrates the relationship between the angular error and the horizontal distance between the observer’s gaze and the camera. The independent variable is the horizontal position of the midpoint between the eyes in the camera coordinate system, which represents the observer’s horizontal displacement. The horizontal distance is distributed from -0.1 to 0.15 meters, and the error distribution ranges from 4 to 10 degrees. Figure 5E demonstrates the relationship between the angular error and the vertical distance. The independent variable is the vertical position of the midpoint between the eyes in the camera coordinate system, representing the observer’s vertical displacement. The vertical distance values range from -0.15 to 0 meters, while the angular error distribution ranges from 5 to 10 degrees. Figure 5F presents the relationship between the angular error and the axial distance between the observer’s gaze and the camera. The independent variable is the axial position of the midpoint between the eyes in the camera coordinate system, which represents the observer’s forward and backward displacement and the distance between the observer and the display. The axial distance ranges from 0.5 to 0.85 meters, and the angular error distribution ranges from 4 to 8 degrees.

Finally, Figure 5G displays the relationship between the angular error and the pitch angle of the observer’s head, representing the vertical swing amplitude. The pitch angle ranges from -70 to 20 degrees, and the angular error distribution ranges from 7 to 9 degrees. Figure 5H shows the relationship between the angular error and the yaw angle of the observer’s head, representing the horizontal swing amplitude. The yaw angle ranges from -10 to 38 degrees, and the angular error distribution ranges from 5 to 9 degrees.

Figure 6 illustrates the relationship between the mean and range of Red, Green, Blue (RGB) values and the angular error. The mean RGB values represent the intensity of the experimental ambient light, which is distributed in the range of 40-120. The range of RGB values represents the angle of the laboratory light source, where a larger light angle corresponds to a larger RGB range, distributed in the range of 80-200. In both cases, the errors are distributed within 5-10 degrees.

We collected the results of random point tests for analysis. First, we plotted the distribution of points and the predicted standard deviation using the actual values and the predicted values. These are shown in Figure 7. In the left panel, the light-colored large circle represents the fixed target point in the test, with its center at the center of the target block and its diameter equal to the target block’s diagonal length. The dark-colored small circles represent the scattered positions of the predicted fixation points by the model. In the right part, the dark small circles represent the actual fixed target point locations, while the light large circles surrounding them indicate the range of predicted points corresponding to the labels. This figure shows the distribution of predicted points around the ground-truth values and the predictive standard deviation.

During the experimental process, we retained the predicted data from the uncalibrated model and the unfiltered results, resulting in four data groups. These data are illustrated in Figure 8. The left graph depicts the functional relationship between angular error and the true values on the X-axis, where the scatter points represent the distribution of the data. The scatter points are plotted at a ratio of 1:100. The four curves represent the quadratic polynomial fit for each dataset, illustrating how the average angular error varies with the x-coordinate. The right graph presents a similar error analysis, with the Y-axis representing the true values. As shown in the graph, filtering the raw data has minimal effect on reducing the average error, whereas filtering after calibration substantially reduces it.

To analyze the performance of the model, we conducted a series of control experiments and tested the following mathematical models: (1) K-nearest neighbors (KNN). In machine learning, KNN is commonly used for pattern recognition tasks. We expanded the image into a one-dimensional vector and concatenated it with the head position and attitude information, then used the KNN method with a neighborhood size of 3 for testing^[30]; (2) Random forest (RF)^[32]. RF is an effective regression method in machine learning. We tested the model using the same data-processing method, with a decision tree size of 300, a maximum depth of 20, and a maximum of 65 features at each node; (3) Linear regression (LR)^[33]. LR is a simple regression method in machine learning and has been applied effectively to medical image analysis tasks^[33]. We used the same data processing method and tested the model using LR.

Finally, utilizing various models including KNN, RF, LR, GPN, and calibrated GPN, we obtained the prediction results for random-point testing. The results from different models were compared, and the error analysis results are presented in Figure 9. Figure 9A shows the error analysis graph with the true values on the x-axis, which has the same pattern as Figure 8, consisting of an error scatter plot and a fitting curve. In contrast, Figure 9B shows the error analysis graph with the true values on the y-axis.

Scattered points represent the error distribution of data sampled at a certain proportion, and a quadratic function is used to fit the scattered points into a curve. The blue, cyan, and yellow colors represent the results of KNN, LR, and RF, respectively. Pink represents the uncalibrated model results, while red represents the calibrated model results with added data-filtering operations. Our model shows a significantly smaller error distribution and lower standard deviation. Additionally, applying the filtering operations after calibration substantially improves the model’s performance.

Figure 9C and D present numerical comparisons of the average angular error and average pixel error for the aforementioned models, respectively. The two figures depict the KNN, RF, LR, general GPN, and calibrated GPN models from left to right, representing progressively lighter shades. The results indicate that our model’s performance is comparable to RF before calibration but improves significantly after calibration.

Table 1 presents the mean and standard deviation of pixel error and angular error for different models. The screen has a pixel range of 2202, and its visible viewing angle ranges from 60 to 70 degrees; the calibrated GPN also includes the subsequent data-filtering operation. Our calibrated GPN achieves a pixel error of 185.19 ± 116.57 and a angular error of 4.52 ± 2.87, which is much smaller than that of KNN, RF, LR, and GPN.

We employed multiple regression task evaluation metrics to analyze the performance of the models used in our study, as shown in Table 2. As in Table 1, the calibrated GPN includes the subsequent data-filtering operation. The metrics include the root mean squared error (RMSE), MAE, R-squared (R²), mean absolute percentage error (MAPE), and mean squared percentage error (MSPE).

Based on the analysis of RMSE and MAE, our model exhibits significantly smaller errors compared to other models. Furthermore, for MAPE and MSPE, our model shows smaller relative errors and fewer instances of large errors. Additionally, the R² coefficient indicates a stronger correlation between our model’s regression results and the reference standards.

The relationship between the horizontal and vertical coordinates of the ground truth values on the screen and the angular error is illustrated in Figure 10 using a three-dimensional plot. From the final coordinate error analysis graph, the model’s prediction error was within 5° in most locations. After calibration and filtering, the average angular prediction error is substantially reduced to 4.52°, demonstrating the effectiveness of the proposed method in achieving accurate gaze estimation across different target locations.

The results of the trajectory tracking test are shown in Figure 11. The left part displays the distribution heatmap of prediction errors. The heatmap depicts the trajectory of the moving target, with brighter colors indicating larger errors and darker colors indicating smaller errors. The right part shows the distribution of scattered points within a certain confidence region. The light-colored circular rings in the background represent tolerance boundaries around the target trajectory, with the radius indicating the allowable distance from the trajectory. Scatter plots depict the distribution of predicted gaze points. A confidence radius is defined to determine whether predicted gaze points fall within a specified distance from the target trajectory and are therefore considered valid tracks. With a tolerance radius of 300 pixels, the tracking accuracy reaches 93.75%. When the tolerance radius is reduced to 200 pixels, the accuracy is 81.91%.

Figure 12 presents the browsing-mode analysis, in which the gaze positions of a participant were simultaneously recorded using a professional eye tracker (Tobii Pro) and the proposed eye-tracking system while freely viewing each image^[34]. The first column [Figure 12A] shows four representative images viewed by the participant. The second and fourth columns [Figure 12B and D] present the heat maps generated by the professional eye tracker and the proposed system, respectively, while the third and fifth columns [Figure 12C and E] show the corresponding gaze trajectory maps. The heat maps illustrate the distribution and density of gaze points, with deeper red indicating higher gaze density and thus the observer’s focal areas of attention. The trajectory maps plot gaze points in chronological order, illustrating the temporal progression of visual attention across the image. Overall, the heat maps and trajectory maps generated by the proposed system closely resemble those obtained from the professional eye tracker, with only minor differences in gaze-point locations. These results demonstrate that the proposed system can provide gaze-tracking patterns comparable to those of the professional eye tracker during free-viewing tasks.

In this study, we designed a comprehensive system for gaze estimation and evaluated its performance through various experiments, drawing on the concept of explicit gaze control in neural talking-head synthesis^[35]. We demonstrated the feasibility of gaze estimation without infrared cameras and showed that a multimodal approach incorporating head spatial position and pose information can enhance the accuracy and generalization performance of gaze estimation. Furthermore, while training the object detection model, we found that increasing the resolution of the feature maps in YOLOv5 can improve the model’s ability to recognize small-sized objects. Additionally, combining similar but different object categories (the left and right eyes) into the same class can significantly improve detection performance for that class.

In the computational workflow, we discovered that in the process of using neural networks for calculations, separating and then fusing features - namely, first computing the head’s position and pose information while using perspective transformation to remove pose information from eye images, and then utilizing neural networks for feature fusion - improved the accuracy to some extent.

We directly regress the target fixation point, which reduces computational complexity and makes calibration more efficient. Calibrating the gaze estimation model before each prediction task greatly improves its accuracy.

Finally, our model performs much better in fixed-point gaze tasks (Random point test) compared to scanning tasks (Sequential point test) involving moving points. This is because our random-point testing procedure averages angular errors and standard deviations after the random point remains fixed for a certain period, yielding slightly lower errors than sequential-point testing involving continuous movement of the target point.

Since existing gaze estimation studies often adopt different datasets, input modalities, and evaluation protocols, direct performance comparisons across datasets may not be fully reliable. Therefore, Table 3 provides a reference comparison with representative methods reported in the literature, rather than a strict quantitative benchmark. The results highlight the effectiveness of our integrated gaze estimation system under the experimental setting adopted in this study.

In existing appearance-based gaze estimation methods, it has been demonstrated that combining head spatial position and three-dimensional posture methods is highly effective for gaze estimation under free-viewing conditions^[36]. Our method includes a process for correcting the head roll angle. When the head undergoes angular changes along the axial rotation, we use a matrix transformation to correct the inverse rotation. This eliminates roll-angle features in the image. We incorporate this corrected feature as an input to the model alongside other modalities, using the multimodal approach we propose.

The gaze estimation method for head rotation in Ref.^[37] directly computes on images without reverse-rotation correction. In comparison, their method relies on separate feature extraction, whereas our approach more thoroughly extracts rotation features. Our method first disentangles the features before fusing them, resulting in improved performance. Ultimately, our method achieves an error 1.18° lower than that of their approach.

Similarly, our approach incorporates perspective transformation for separating the head’s yaw and pitch angles. As a result, our method demonstrates a certain degree of adaptability to both horizontal and vertical head rotations. The method described in^[38] focuses on gaze estimation for baseline angles. They trained their model on datasets with significant angle variations and achieved an error of approximately 8 degrees. Figure 5G and H illustrate the error analysis results of our method in these two dimensions. Compared with this specialized approach tailored to the specific conditions, our method achieves a performance close to its capabilities.

We acquired data under various lighting conditions by adjusting the illumination. Furthermore, during the image preprocessing phase, techniques such as histogram equalization and data standardization were applied to mitigate the impact of variations in image RGB values. In contrast to previous work^[39], which focused on dim-lighting scenarios (resulting in an error of 9.52 degrees), our approach demonstrates an average error reduction of 5.00 degrees. As illustrated in Figure 6, our method appears to be less affected by variations in light intensity and direction, thus exhibiting strong performance across different lighting conditions.

The method mentioned in^[40] is a gaze tracking approach for mobile devices. It utilizes the front camera of a smartphone to capture images, locates 68 facial landmarks using a model, and performs estimation. Our method, by contrast, is deployed on desktop platforms and benefits from higher computational power. We employ the YOLO model for target detection; in comparison, our method exhibits higher fault tolerance, faster computation speed, and lower resource consumption. On desktop platforms, we achieve performance similar to theirs with greater efficiency and resource savings.

Our approach shares some similarities with the method proposed in^[41], as both utilize facial-related information for gaze estimation. However, Ref.^[41] employs an long short-term memory (LSTM)-based framework to model temporal dependencies from continuous facial sequences, whereas our method focuses on individual-frame gaze estimation by integrating head pose information, binocular features, and calibration strategies. This design reduces the dependency on temporal information and enables a more lightweight implementation for scenarios where continuous data acquisition is unavailable or unnecessary. Although Ref.^[41] demonstrates the effectiveness of temporal modeling for gaze estimation, a direct quantitative comparison between the two methods is difficult due to differences in datasets, input settings, and experimental protocols. Nevertheless, the comparison highlights different design considerations: temporal modeling can improve performance when sequential data are available, while our framework provides a practical alternative for frame-based gaze estimation with reduced dependence on continuous recordings. Future work will further investigate this issue using unified experimental protocols and larger datasets.

Compared to existing methods, one advantage is our calibration approach. We fine-tune neural networks for calibration, enabling robust calibration under various conditions such as changes in characters, lighting, camera parameters, camera field of view, camera position, screen size, and more. This calibration significantly enhances accuracy. During system deployment, our approach eliminates the need for fixed camera placement, head fixation, and external conditions like controlled lighting. This allows tasks to be accomplished without setting up a series of external conditions, providing a more flexible and efficient solution.

Despite the promising performance of the proposed gaze estimation system, several limitations remain.

First, the current system relies solely on images captured under visible light conditions, without incorporating infrared sensors. Consequently, its performance is sensitive to illumination variations. Although data augmentation and training strategies were employed to improve robustness, illumination effects cannot be fully eliminated. This limitation could be addressed by introducing advanced data enhancement techniques, such as diffusion-based models, to improve image quality under challenging lighting conditions and enhance robustness.

Second, the target detection module operates on a per-frame basis without enforcing temporal consistency. This leads to fluctuations in detection results across consecutive frames, causing inconsistencies in the extracted eye regions and resulting in random directional deviations in gaze predictions. To overcome this limitation, temporal consistency constraints could be incorporated, and gaze estimation models designed for continuous video streams could be developed to leverage inter-frame information and reduce prediction drift.

Third, the current pipeline lacks fine-grained preprocessing of eye images, such as explicit segmentation of the iris and eyelids, which limits the effectiveness and accuracy of feature extraction. In addition, the absence of temporal modeling prevents mitigation of randomness inherent in single-frame predictions. This issue could be alleviated by integrating eye-region segmentation methods to isolate key anatomical structures and adopting sequence-based modeling approaches to better exploit temporal dynamics.

Finally, although the current study adopts a participant-level data partitioning strategy to ensure subject-disjoint training, validation, and testing, further evaluation using more rigorous subject-independent protocols, such as leave-one-subject-out cross-validation, would provide additional evidence of the proposed method’s robustness and generalizability. Moreover, a systematic ablation analysis of individual components within the proposed framework would further clarify the contribution of each module and provide deeper insights into the model design. Future studies will investigate larger and more diverse cohorts, incorporate more comprehensive cross-participant validation strategies, and conduct extensive ablation studies to further assess the model’s applicability and reliability in real-world scenarios.