MIS-NeRF: neural radiance fields in minimally-invasive surgery.

Abstract

<h4>Purpose</h4>Minimally-invasive surgery (MIS) reduces the trauma compared to open surgery but is challenging for endophytic lesion localisation. Augmented reality (AR) is a promising assistance, which superimposes a preoperative 3D lesion model onto the MIS images. It requires solving the difficult problem of 3D model to MIS image registration. We propose MIS-NeRF, a neural radiance field (NeRF) which provides high-fidelity intraoperative 3D reconstruction, used to bootstrap iterative closest point (ICP) registration.<h4>Methods</h4>Existing NeRF methods break down in MIS because of the moving light source and specular highlights. The proposed MIS-NeRF is adapted to these conditions. First, it incorporates the camera centre as an additional input to the radiance function, which allows MIS-NeRF to handle the moving light source. Second, it uses a modified volume rendering which handles specular highlights. Third, it uses a regularised compound loss to enhance surface reconstruction.<h4>Results</h4>MIS-NeRF was tested on three synthetic datasets and retrospectively on four laparoscopic surgeries. It successfully reconstructed high-fidelity liver and uterus surfaces, reducing common artefacts including high-frequency noise and bumps caused by specular highlights. ICP registration achieved faithful alignment between the preoperative and intraoperative 3D models, with an average error of 3.25 mm, outperforming the second-best method by a 15% margin.<h4>Conclusion</h4>MIS-NeRF improves AR-based lesion localisation by facilitating accurate 3D model registration to multiple MIS images.