Shank3B KO mice (stock No. 017688, The Jackson Laboratory) were used in this study and maintained as heterozygous breeders. Wild-type (WT) mice were littermate controls derived from the same breeding pairs. A total of 20 adult male mice were used in this study (Shank3B KO: n = 8; WT littermates: n = 12), aged 8-12 weeks and weighing 22-28 g at the time of testing. Mice were housed under controlled conditions on a 12 h light/12 h dark cycle (lights on from 08:00 to 20:00) at a constant room temperature of 22-24 °C, with ad libitum access to food and water. All procedures were approved by the Institutional Animal Care and Use Committee (IACUC) of the Fourth Military Medical University (Approval No.251529) and followed the ARRIVE 2.0 guidelines, in accordance with the National Institutes of Health (NIH) Guide for the Care and Use of Laboratory Animals.

The three-chamber test was performed following the procedures reported previously^[26]. Mice from both groups were habituated to the testing room for 2 h before behavioral assessment. The three-chamber apparatus consisted of a central chamber (40 cm × 20 cm) and two side chambers (40 cm × 20 cm each), with a wire cage positioned in each side chamber for stimulus mice. At the start of the test, the subject mouse was placed in the central chamber and allowed to freely explore all chambers for 5 min for habituation. In the sociability phase (Stage 1), an unfamiliar male mouse with no prior contact with the subject was placed inside one wire cage in a side chamber, while the opposite cage remained empty. The partitions were opened, and the subject mouse was allowed to explore freely for 10 min. In the social novelty phase (Stage 2), a second unfamiliar male mouse was placed in the previously empty cage, and the subject mouse was again allowed to explore for 10 min. Behavior was recorded throughout each stage. The apparatus was cleaned with 75% ethanol between trials to eliminate olfactory cues.

Grooming test was performed following the procedures reported previously^[27]. Mice from both groups were habituated to the testing room for 2 h prior to the experiment. Each mouse was then transferred individually from its home cage to a testing chamber (30 cm × 20 cm × 20 cm), where behavior was continuously recorded for 30 min. After each trial, the chamber was cleaned with 75% ethanol to remove residual odor cues before testing the next animal.

Open field test was performed following the procedures reported previously^[28]. Mice from both groups were habituated to the testing room for 2 h before the experiment. Each mouse was then transferred individually from its home cage to the center of a cubic open-field arena (50 cm × 50 cm × 50 cm). Spontaneous locomotor and exploratory (Active, variable movements for environmental investigation) behavior was recorded for 10 min. The arena was cleaned with 75% ethanol between trials to eliminate olfactory cues.

Elevated plus-maze test was performed following the procedures reported previously^[29]. Mice were habituated to the testing room for 2 h prior to testing. The elevated plus maze was constructed of black Plexiglas and consisted of two open arms (50 cm × 10 cm), two enclosed arms of the same size with black acrylic walls, and a central platform (10 cm × 10 cm). At the start of each trial, the mouse was placed on the central platform facing an open arm and allowed to explore freely for 10 min while behavior was recorded. The maze was cleaned with 75% ethanol between trials.

Behavioral data from the three-chamber, grooming, open field, and elevated plus maze tests were processed using a unified computational pipeline. Discrete behavioral events were grouped into higher-level behavioral states, and per-animal metrics, including event counts, total duration, and time allocation proportions, were calculated. Genotype differences were evaluated using two-tailed Student’s t-tests or Mann-Whitney U tests as appropriate. Results are presented as mean ± SEM or SD with individual data points shown.

Behavioral transition probability matrices were constructed for each genotype to characterize state-switching dynamics. Differences in transition probabilities were assessed using Fisher’s exact tests or chi-square tests of independence. Multiple comparisons were corrected using the Benjamini-Hochberg false discovery rate (FDR) procedure where applicable. Transition structures were visualized using probability heatmaps. The calculation methods of the trajectory entropy, exploration index and anxiety index are provided in Supplementary Figure 1.

To identify latent behavioral structure, each behavioral event was encoded as a multidimensional feature vector incorporating behavioral state identity, temporal context, duration measures, and kinematic or spatial features. Features were standardized using robust scaling prior to analysis. T-SNE was applied for dimensionality reduction and visualization of behavioral clusters. T-SNE was performed with a perplexity of 30 and a fixed random state for reproducibility. This was followed by k-means clustering in the embedded space to define behavioral clusters. Cluster distributions were visualized in low-dimensional projections and summarized by behavioral composition. Genotype differences in cluster usage were evaluated using Mann-Whitney U tests, and enrichment ratios were calculated to adjust for unequal event counts.

To assess the discriminative power of extracted behavioral features, genotype classification was performed using a support vector machine (SVM) classifier. A linear kernel was used with the default regularization parameter C = 1.0. No hyperparameter tuning was performed beyond the default setting to avoid overfitting given the limited sample size. Model performance was evaluated using leave-one-animal-out (LOAO) cross-validation, repeated 10 times with random sample shuffling (random seed = 42). Performance metrics (accuracy and area under the ROC curve, AUC) are reported as mean ± SD.

For conventional behavioral metrics (e.g., time in zones, distance traveled, entries, speed), normality was first assessed using the Shapiro-Wilk test. For data meeting normality (P > 0.05) and homogeneity of variance, two-tailed Student's t-test was applied; otherwise, the Mann-Whitney U test was used. These predefined primary outcome measures were considered significant at P < 0.05 without multiple comparison correction.

For behavioral subtype proportions (e.g., grooming subcategories, time allocation across behavioral states), transition probability matrices (analyzed using Fisher's exact test), cluster composition (enrichment of WT/KO events across unsupervised clusters), and genotype distribution across clusters, the Benjamini-Hochberg false discovery rate (FDR) procedure was applied for multiple comparison correction. Adjusted q-values are reported, and significance was defined as q < 0.05.

Classifier performance was evaluated using a linear support vector machine (SVM, kernel = “linear”, C = 1.0) with leave-one-animal-out (LOAO) cross-validation, repeated 10 times with random sample shuffling. A fixed random seed (42) was set for the SVM classifier to ensure reproducibility. Performance metrics (accuracy and area under the ROC curve, AUC) are reported as mean ± SD. The 95% confidence interval for AUC was estimated via bootstrap (1,000 resamples). No hyperparameter tuning was performed beyond the default setting to avoid overfitting given the limited sample size. No additional multiple comparison correction was applied, as each behavioral assay involved an independent classification task and cross-validation inherently accounts for generalization error.

To maintain visual clarity in the main figures, significance is indicated by asterisks (*). Exact p-values and FDR-adjusted q values for all key comparisons are provided in Supplementary Table 1.

All statistical analyses were performed using Python (v3.9) with scikit-learn (v1.2.0) for machine learning analyses, and GraphPad Prism (v9.0) for conventional statistical tests.

To biologically interpret the latent behavioral clusters identified by unsupervised clustering, we performed a systematic ground-truthing procedure as follows.

First, for each cluster identified by k-means, we extracted the centroid event—the behavioral sample whose feature vector had the smallest Euclidean distance to the cluster center. Additionally, we selected the 10 events closest to the centroid to capture the core behavioral characteristics of the cluster.

Second, a trained observer, blinded to both genotype and cluster assignment, reviewed the video segments corresponding to these centroid and near-centroid events. Each segment was examined at normal speed and, when necessary, frame-by-frame. The predominant behavioral motif was recorded for each event.

Third, the manual annotations derived from video inspection were cross-validated against the quantitative distribution of predefined behavioral categories within each cluster. For example, a cluster was annotated as "leg-focused grooming" only if the video inspection consistently identified leg-licking and the majority of events in that cluster belonged to the predefined "leg lick" category.

Fourth, for clusters that shared similar behavioral labels, we further characterized their temporal and sequential properties, including duration, self-transition probability, and behavioral context (preceding and following states), to distinguish between functionally distinct but superficially similar clusters.

This procedure ensured that each latent cluster received a biologically meaningful and empirically validated annotation.

To evaluate whether fine-grained behavioral analysis applied to classical paradigms can sensitively capture genotype-related social phenotypes, we first examined social behavior in Shank3B KO and WT littermates using the three-chamber assay [Figure 1A]. Using conventional region-based behavioral metrics, we observed genotype differences consistent with previously reported social deficits in this model^[9]. Representative trajectory heatmaps further illustrated distinct spatial exploration patterns between genotypes [Figure 1B]. During the sociability phase, WT mice showed a clear preference for the chamber containing an unfamiliar mouse, whereas KO mice showed no significant preference, indicating reduced sociability [Figure 1C and D]. In the social novelty phase, WT mice preferentially investigated the newly introduced mouse, while KO mice failed to discriminate between familiar and novel social stimuli [Figure 1E and F].

Fine-grained behavioral quantification further resolved genotype differences [Figure 1G-L]. KO mice allocated less time to active social behaviors and more time to rigid (Stereotyped, repetitive postures with little variability) or inactive (No voluntary movement, including resting or freezing) states. Transition probability analysis showed increased switching from movement to rest or freezing and reduced transitions toward exploratory states, indicating reduced exploratory drive and altered social state dynamics. It is worth noting that total movement distance was quantified to aid interpretation of social interaction measures, as reduced locomotion could potentially influence time spent in social vs. empty chambers.

Machine learning-based classification reliably separated genotypes based on behavioral features alone [Figure 1I and L], supporting the presence of stable genotype-specific social behavioral signatures.

To explore latent behavioral structure beyond predefined categories, we performed dimensionality reduction and unsupervised clustering on high-dimensional behavioral features. This analysis identified distinct behavioral state clusters [Figure 2A-J]. Stationary or low-engagement (Minimal interaction with environmental or social stimuli) states were overrepresented in KO mice across both testing stages, whereas WT mice were more broadly distributed across exploratory clusters [Figure 2D, E, I, J]. These results indicate that KO mice exhibit reduced social engagement structure and restricted behavioral diversity, which are not fully captured by conventional summary metrics.

Consistent with prior reports of elevated repetitive behavior in Shank3B KO models^[17], KO mice displayed significantly increased total grooming duration compared with WT mice [Figure 3A and B]. Subtype analysis further showed selective increases in leg licking and reductions in body grooming, while other grooming components were unchanged [Figure 3C], indicating an altered grooming pattern rather than a uniform increase across all subtypes. Behavioral transition analysis supported this shift in grooming structure. KO mice showed higher transition probabilities toward leg-licking sequences and fewer transitions into body-grooming states [Figure 3D], suggesting biased and less diverse grooming organization. Importantly, SVM classifiers based on grooming-derived behavioral features achieved high accuracy in discriminating genotypes [Figure 3E].

To further resolve grooming microstructure, we performed dimensionality reduction and clustering on multidimensional grooming features, identifying five stable behavioral clusters (C0-C4) representing distinct grooming modules [Figure 3F]. Clusters C0, C1, and C3 were composed of mixed behavioral elements rather than single repetitive actions. C2 consisted almost exclusively of no-grooming states. C4 was dominated by body grooming. Cluster usage analysis revealed that C0, C1, and C2 showed comparable representation between genotypes. In contrast, cluster C3 was significantly enriched in KO mice, whereas cluster C4 was significantly enriched in WT mice [Figure 3G-J]. This cluster-level structure indicates that KO grooming behavior is more repetitive and less compositionally diverse, extending the conventional duration-based findings.

Using standard open-field metrics, we observed behavioral differences consistent with previously described anxiety-like phenotypes in Shank3B KO models^[17]. KO mice showed pronounced wall-following trajectories [Figure 4A], reduced center-zone time and distance, fewer center entries, and decreased total locomotion compared with WT mice [Figure 4B-G]. Movement velocity was also reduced across zones [Figure 4E and F], indicating lowered spontaneous activity.

Fine-scale trajectory analysis further revealed reduced path entropy and a shift toward small-angle turning in KO mice, together with fewer total turns [Figure 4H-L], indicating more predictable and stereotyped movement patterns. Detailed calculation method of the trajectory entropy is provided in Supplementary Figure 1A. Distance-to-wall measures confirmed strengthened thigmotaxis [Figure 4M and N]. Time-resolved indices showed persistently elevated anxiety scores and reduced exploration drive across the session in KO mice [Figure 4O and P]. Detailed calculation methods of the exploration index and anxiety index are provided in Supplementary Figure 1B and C.

To characterize latent behavioral structure, we clustered multidimensional behavioral segments into five stable states (C0-C4) [Figure 4Q]. Clusters associated with center exploration and active locomotion were broadly represented in WT mice, whereas KO mice were strongly enriched in the periphery-dominant (Preference for outer zones) activity cluster (C1) and underrepresented in exploratory clusters [Figure 4R-T]. This cluster distribution pattern refines conventional zone-based metrics by revealing restricted behavioral state diversity in KO mice. Locomotion-related measures (total distance, speed, and turn number) were quantified to aid interpretation of these outcomes, as reduced activity could influence center exploration and movement patterns.

Elevated plus maze testing revealed anxiety-like behavior in KO mice consistent with classical measures^[17]. KO mice showed reduced open-arm time, distance, and entries, together with decreased total locomotor distance [Figure 5A-G]. Movement velocity was reduced in closed arms and modestly increased in open arms [Figure 5H and I], consistent with altered risk-context locomotion.

Behavioral composition analysis indicated a higher proportion of risk-avoidant (Behaviors that minimize exposure to dangerous areas) behavioral states and reduced rearing in KO mice [Figure 5J]. D Transition probability mapping further showed that KO mice preferentially cycled among closed-arm states and were less likely to transition into locomotor and exploratory states [Figure 5K]. Behavioral-feature-based SVM classification again reliably separated genotypes [Figure 5L].

Clustering of multidimensional behavioral features identified five latent behavioral modules (C0-C4) with distinct behavioral associations [Figure 5M and N]. KO mice were enriched in clusters corresponding to risk-avoidant and closed-arm-dominant states (C3/C4), whereas WT mice showed greater representation in clusters associated with rest-exploration balance (C0/C2) [Figure 5O-Q]. These cluster-level differences extend traditional arm-based metrics by revealing genotype-dependent behavioral organization patterns. Locomotor activity measures (total distance and arm-specific speed) were quantified to facilitate interpretation, recognizing that reduced locomotion may partially account for decreased open-arm exploration.