Page 81 - Read Online

Wei et al. Art Int Surg 2024;4:187-98 Artificial Intelligence Surgery
DOI: 10.20517/ais.2024.12

Original Article Open Access

Scale-aware monocular reconstruction via
robot kinematics and visual data

in neural radiance fields

2
2
3
2
2
Ruofeng Wei 1 , Jiaxin Guo , Yiang Lu , Fangxun Zhong , Yunhui Liu , Dong Sun , Qi Dou 1
1 Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong 000000, China.
2 Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong 000000, China.
3 Department of Biomedical Engineering, City University of Hong Kong, Hong Kong 000000, China.
Correspondence to: Dr. Ruofeng Wei, Department of Computer Science and Engineering, Room115, Ho Sin-Hang Engineering
Building, The Chinese University of Hong Kong, Shatin, Hong Kong 000000, China. E-mail: ruofengwei@cuhk.edu.hk
Howto cite this article: Wei R, Guo J, Lu Y, Zhong F, Liu Y, Sun D, Dou Q. Scale-aware monocular reconstruction via robot kinematics
and visual data in neural radiance fields. Art Int Surg 2024;4:187-98. http://dx.doi.org/10.20517/ais.2024.12
Received: 6 Feb 2024 First Decision: 20 Jun 2024 Revised: 15 Jul 2024 Accepted: 31 Jul 2024 Published: 16 Aug 2024

Academic Editors: Andrew A. Gumbs, Eyad Elyan Copy Editor: Pei-Yun Wang Production Editor: Pei-Yun Wang

Abstract
Aim: Scale-aware 3D reconstruction of the surgical scene from a monocular endoscope is important for automatic
navigation systems in robot-assisted surgery. However, traditional multi-view stereo methods purely utilize monoc-
ular images, which can recover 3D structures arbitrarily scaled with the real world. Current deep learning-based ap-
proaches rely on large training data for relative depth estimation and further 3D reconstruction with no scale. Inspired
by recently proposed neural radiance fields (NeRF), we present a novel pipeline, KV-EndoNeRF, which explores limited
multi-modal data (i.e., robot kinematics, and monocular endoscope) for surgical scene reconstruction with absolute
scale.

Methods: We first extract scale information from robot kinematics data and then integrate it into sparse depth re-
covered from structure from motion (SfM). Based on the sparse depth supervision, we adapt a monocular depth
estimation network to the current surgical scene to obtain scene-specific coarse depth. After adjusting the scale of
coarse depth, we use it to guide the optimization of NeRF, resulting in absolute depth estimation. The 3D models of
the tissue surface with real scale are recovered by fusing fine depth maps.

Results: Experimental results on the Stereo Correspondence And Reconstruction of Endoscopic Data (SCARED)
demonstrate that KV-EndoNeRF excels in learning an absolute scale from robot kinematics and achieves 3D recon-
struction with rich details of surface texture and high accuracy, outperforming other existing reconstruction methods.

© The Author(s) 2024. Open Access This article is licensed under a Creative Commons Attribution 4.0
International License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, shar-
ing, adaptation, distribution and reproduction in any medium or format, for any purpose, even commercially, as long as you
give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate
if changes were made.

www.aisjournal.net