Full-tool registration in surgical navigation: Enhancing localization and visualization

Abstract

Instrument localization is widely used in computer-assisted surgery. Most existing navigation systems monitor the position of the surgical instrument’s stylus-tip in non-visible surgical areas. However, a gap still exists in meeting the surgeons’ requirements for instrument monitoring. It is difficult for them to monitor the instrument’s working direction and its contact with the lesion area without an intuitive silhouette or additional feature points on the instrument. This paper introduces a full-tool surgical navigation for enhancing instrument localization and visualization in surgery, providing silhouette monitoring and directional indications. The key to overcoming these challenges lies in reconstructing the surgical instrument, as well as calibrating the stylus-tip and direction of the key part on the instrument. Our method achieves this through a silhouette carving reconstruction method with a feature-dense position-sensing marker. In the experiment, the fiducial registration error of our method reaches 0.496 mm, and the relative orientation error is 1.401°. These localization and orientation accuracies demonstrate the potential of full-tool navigation system in surgeries.

Keywords

computer-assisted surgery surgical navigation instrument registration marker-based localization spatial model reconstruction

Introduction

Surgical navigation is a key component of the computer-assisted surgery (CAS), and can be categorized into electromagnetic navigation (Figure 1(a)),^1,2 optical navigation (Figure 1(b)),^3,4 and visual navigation (Figure 1(c) and (d)).^5–7 Enhancing accuracy, robustness, and visibility of surgical localization facilitates surgeons, especially inexperienced ones, in performing highly precise surgery. It can reduce the error and risk of surgery, and improve the efficiency of operation and therapeutic effect.

Figure 1.

Computer-aided technologies in surgical navigation: (a) electromagnetic navigation, (b) optical navigation, (c) visual navigation using feature array patterns (Claro Technology), (d) visual navigation using self-identifying markers (Inliant Technology and X-Nav Technology), and (e) visual navigation using position-sensing markers (HydraMarker).

Most existing navigation systems provide surgeons with precise localization of the surgical instrument’s stylus-tip in non-visible surgical areas.^8–10 Surgeons can monitor the real-time position of the stylus-tip on a screen and perform surgical procedures accordingly.^11,12 However, a gap still exists in meeting the requirements of instrument monitoring. In non-visible lesion area, surgeon’s spatial perception of instrument relies on the projected silhouette of instrument in the medical images.¹³ Thus, they require an intuitive silhouette of the instrument, or more feature points on instrument, to help them accurately monitor the instrument’s working direction and its contact with the lesion area.¹⁴

Outline-based localization methods enable contour monitoring and localization. However, they fail under conditions of large-area or complete occlusion, often leading to inaccurate position sensing of the surgical instrument.¹⁵ In contrast, marker-based localization methods provide high localization accuracy but offer limited capability for structural or shape monitoring.¹⁶ It should also include directional indications for the key part of the instrument, such as vectors. It is termed the full-tool navigation. With directional indication, surgeons can adjust the instrument’s pose in real-time and align it with the surgical path planned during preoperative planing. By monitoring the silhouette, surgeons can avoid nerves and tissues in the lesion area, reducing surgical errors and risks.

Presenting the silhouette and directional indications relies on a complete model of the surgical instrument. However, in practice, instrument models are usually not provided by the manufacturer, and instruments are sometimes modified to meet the requirement of surgeons and existing navigation systems. This results in navigation systems lacking effective models. Therefore, it is necessary to add the reconstruction of instrument to the preoperative registration, which is termed full-tool registration. Structured light-based and texture-based reconstruction methods cannot effectively reconstruct models because the instrument’s metallic surface reflects light and lacks texture.^17,18 Silhouette-based reconstruction method uses multi-view silhouettes of the object as boundary constraints to reconstruct the model,¹⁹ meeting the requirements of silhouette projection in full-tool navigation system. In addition, silhouette-based reconstruction process is simple to operate, suitable for surgeons.

However, state-of-the-art navigation systems lack the ability to achieve full-tool registration. The reconstruction process requires multiple viewpoints of observation and localization simultaneously. Ordinary marker-based navigation systems use optical spheres or a common checkerboard array marker, both of which have numerous observational blind spots.^20,21 Inliant Dental Technology and X-Nav Dental Technology design self-identifying markers to solve the problem of occlusion and achieve the multiple viewpoint observation.^22–26 However, they still have a low position-sensing ability in observation freedom. Therefore, a position-sensing marker is needed to cooperate with silhouette-based reconstruction method, and to address the observation incompleteness in registration and localization.

In this article, we propose a full-tool surgical navigation, including a full-tool registration method of the surgical instrument. Surgeons can complete the surgical instrument reconstruction by holding and swing it, and calibrate the key points and directions. This novel surgical navigation provides an easy-to-learn full-tool registration method and a position-sensing marker-based method in registration and localization. The major contributions of this paper are as follow:

1) Full-tool registration. The registration process is decomposed into the full-tool reconstruction and the full-tool calibration. A silhouette carving reconstruction method assists surgeons in reconstructing the complete model of the surgical instrument. In the full-tool calibration, the instrument’s stylus-tip and directional indication are calibrated based on instrument’s reconstruction model, achieving precise localization.

2) Marker-based localization. A feature-dense position-sensing marker can improve the accuracy and robustness of localization, and solve the problem of the incomplete observation. A trigonal checkerboard marker is affixed to the curved surface of the localization block (Figure 1(e)), reducing observational blind spots and enhancing robust observation from multi-viewpoints. In addition, a self-identifying marker-based spatial reconstruction method is proposed, which provides surgeons with an easy-to-learn marker registration method.

The rest of this paper is arranged as follows: Section Materials and Methods proposes full-tool navigation system and introduces a feature-dense position-sensing marker. Section Results presents the experiments to evaluate the accuracy. Section Discussion discusses the accuracy of the reconstruction, calibration, and localization. Section Conclusion gives conclusions.

Materials and methods

The hardware of the proposed reconstruction method includes two monocular color industrial cameras, a localization block with the fiducial marker, a workstation for data processing, and a display for visualization. The flow chart of full-tool registration method is shown in Figure 2. The method is divided into three main steps: (1) Calibration, including monocular-camera calibration and marker feature cloud reconstruction. (2) Extraction, the surgical instrument’s silhouette is obtain by image subpixel edge recognition and region growing criterion. (3) Reconstruction, the model of the surgical instrument is reconstructed, and the key points and vector are calibrated. The dental surgical instrument is used as an example to introduce our navigation system. The details are described in Section to Section.

Figure 2.

Full-tool registration method flow chart.

In addition, in our proposed method, the fiducial marker is generated using the HydraMarker toolkit provided by Zhu et al.²⁷ HydraMarker provides dense features and seamlessly overlapping self-identification units within a limited area, providing redundancy for pose estimation, as shown in Figure 3(a) and (b). Moreover, these patterns exhibit a certain resistance to splitting and bending, allowing them to be divided into multiple subpatterns and attached to any regular curved surface, as shown in Figure 3(c). Therefore, in this paper, these characteristics are used to address challenges associated with line-of-sight occlusion and multi-viewpoints pose estimation.

Figure 3.

Position-sensing marker: (a) HydraMarker has seamlessly overlapping self-identification units, (b) HydraMarker is detected with individual label on its every cross point, and (c) HydraMarker facing visibility problems: cutting, occlusion, and deformation.

Incremental marker point-cloud reconstruction

A complete feature point model of the marker is key to the marker-based localization method. Simultaneously, it simplifies the pose estimation of the silhouettes in reconstruction from different viewpoints. When a marker is attached to curved surfaces, it is necessary to identify and stitch feature points from multiple views. This paper proposes an adaptive incremental marker reconstruction method, including feature matching, reconstruction, incremental matching of new images, and iterative reconstruction, as shown in Figure 4.

Figure 4.

The marker reconstruction illustration and pipeline of our proposed method.

The method consists of two threads: the construction of the image pool and incremental stitching. In the first thread, the images are detected and judged: (1) If an image is blurry or of too poor quality, it will be removed; (2) If a detected image has high similarity to those in the pool, and is of higher quality or contains a greater number of feature points, it will be selected and added to the pool. Therefore, the image pool continuously optimizes its internal quality.

In the second thread, the feature points in the pool are matched and stitched. The image containing the most feature points is used as the primary-image. The sub-images which are images of less feature points than the primary-image, are stitched onto the primary-image. Then, the stitched sub-images are used to match other sub-images for further stitching. In this case, the accumulation of stitching errors can be reduced. Incremental reconstruction continuously adds images of new viewpoints to the image pool, or updates the image quality. The primary-image may be changed during this process, and it continues until the number of marker points reaches saturation.

Silhouette extraction

The proposed method combines the Canny edge detection algorithm²⁸ with the region growing criterion to extract clean instrument silhouettes. Image acquisition is carried out with a relatively clean background to enhance the silhouette information. Firstly, the image is smoothed using Gaussian Filtering, as shown in Figure 5(a). Then, the edge points of the surgical instrument are preliminarily extracted using the Canny edge detection algorithm.

Figure 5.

Illustration of silhouette extraction for the surgical instrument.

The surgical instrument silhouette is segmented from the background based on Canny, and its interior is filled using the region growing algorithm.²⁹ The pixel points at the four corners of the image are selected as seed points, as shown in Figure 5(b). Starting from the initial seeds, similar pixel points are matched from the neighboring pixels based on the intensity similarity criterion, as

Similarity (p, R_{i}) = | I (p) - μ_{R_{i}} | \leq T, {i | i \in [1, 4]},

(1)

where $I (p)$ is the pixel of the neighboring pixel points $p$ , $T$ is the intensity threshold, $R_{i}$ represents the growing region of the $i$ th initial seed point, and $μ_{R_{i}}$ is the updated seed intensity parameter, which can be described as

μ_{R_{i}} = \frac{1}{| R_{i} |} \sum_{(x, y) \in R_{i}} I (x, y) .

(2)

These similar pixels are then recorded and marked as new seeds. The old seeds are recorded as segmented points, and the seed intensity parameter $μ_{R_{i}}$ is updated. This iterative process continues until no new pixel points meet the seed intensity parameter, as shown in Figure 5(c). The preliminarily extracted edges by Canny prevent pixel seeds from entering the interior of the silhouette, thus avoiding overgrowth.

Full-tool reconstruction

The proposed method reconstructing the surgical instruments is based on the intersection of visual cones in camera’s field of view (FoV), as illustrated in Figure 6(a). In other words, the reconstructed model is formed by the intersection of the visual volumes generated from the silhouettes of each view. Voxel parts outside the silhouettes are carved away based on different silhouettes, gradually restoring the voxel model of the surgical instrument.

Figure 6.

Full-tool reconstruction and full-tool calibration: (a) the 2D illustration of reconstruction, (b) the color mapping of reconstructed model, (c) the 3D illustration of reconstruction, and (d) the illustration of full-tool calibration and geometry illustration of calibration.

Unlike ordinary shape-from-silhouette,¹⁹ we use the multi-level pyramid criterion to control the coarse-to-fine of models. A voxel block of size $m \times n \times k - d$ is constructed based on the marker point cloud model. The initial voxel point spacing density is $d$ , and the voxel level is 0. The surgical instrument is rigidly connected to the positioning block, thus the relative position between the camera and the voxel block can be computed by the identified markers. Then, the silhouette image of each direction of the voxel block is determined, as shown in Figure 6(c).

The 0 and 1 are used to represent whether a pixel point is inside the silhouette or not. For the silhouette of each view, each voxel point is determined whether inside the silhouette. If a voxel point is inside the silhouette, its level is increased by 1. The level is increased by adding more silhouette images. Different voxel points may be inside different silhouettes or multiple silhouettes, thus these voxels with different levels belong to different pyramid layers.

Selecting different layers within the pyramid can control the coarseness and fineness of the model. It can prevent some voxel points from being cut out due to over-segmented edges. Voxel points excluded by over-segmentation can be compensated by other edge features. The color of each pixel on the instrument images is mapped to the corresponding voxel points, as shown in Figure 6(b).

Full-tool calibration

The stylus-tip calibration of the surgical instrument using fiducial marker is typically performed with the pivot calibration method.³⁰ For point localization, the stylus-tip point of the instrument is projected onto medical images for surgeons monitor to perform operation. However, for irregular instruments, such as a surgical handpiece, surgeons cannot directly monitor the working direction of instrument without vector localization or a silhouette. The vector direction of implantation $α$ follows the drill, as shown in Figure 6(d). Therefore, it is not sufficient to only use point localization.

A vector calibration method is proposed based on the instrument’s point cloud, which is reconstructed from full reconstruction. The shell of the instrument point cloud model is extracted by the Convex Hull algorithm. The farthest point on shell, away from the center of the point cloud, is selected to determine the stylus-tip, as

{\begin{matrix} P_{tip} = \underset{P_{i}^{hull}}{argmax} ∥ P_{i}^{hull} - P_{center} ∥ \\ P_{center} = \frac{1}{n} \sum_{i = 1}^{n} P_{i} \end{matrix},

(3)

where $P_{tip}$ is the surgical instrument’s stylus-tip, $P_{i}^{hull}$ is the point on the shell of model, and $P_{center}$ is the center of the point cloud. Then, the center axis of the drill and the axial vector $α$ are calculated based on the stylus-tip and a part of point cloud of drill.

Results

Hardware configuration and experimental preparation

In this section, a series of experiments are conducted on the reconstruction, calibration, and localization methods for the surgical instrument. A control group of the pivot calibration method is set to compare with the proposed full-tool calibration. And a control group of the instrument with ArUco-based localization³¹ is used for comparation with HydraMarker in occlusion experiment.

A stereo camera is composed of two industrial cameras of Hikvision MV-CS023-10GM (mono, $1920 \times 1200$ pixels). The positioning block composed of a cylinder and two cone is 3D printed with 20 mm in radius, and is installed on the surgical handpiece. A $5 \times 28$ HydraMarker is divided into two sub-patterns, which are affixed on the surface of cylinder and cone. The computer for the navigation system to process data of visualization is equipped with Intel i9-10900 (2.80 GHz). Customized localization test block and orientation test hemisphere are 3D printed for experiments and accuracy evaluation. In addition, we prepare a raster ruler moving platform for the HydraMarker-based localization experiment.

Marker point-cloud reconstruction

In the preparatory work, the stereo camera is calibrated by stereoCameraCalibration in MATLAB toolbox. The reprojection error in calibration parameter is 0.052 pixels, meeting the accuracy requirement for triangulation. After camera calibration, we fix the surgical instrument on the rotating platform and rotate it arbitrarily, as shown in Figure 7(a). In the first thread, the camera adds new images to the image pool. Each set of left and right images is matched and triangulated, as shown in Figure 7(b). The new 3D point group is judged and selected for inclusion in the image pool based on the triangulation error. In the second thread, the marker point cloud is reconstruction based on the image pool. When the pool is updated, a new incremental reconstruction is carried out, until the marker point cloud model reaches saturation, as shown in Figure 7(c).

Figure 7.

Self-identifying marker spatial model under incremental reconstruction: (a) physical setup of the experiment, (b) feature points on marker under detection, identification, and matching, and (c) incremental reconstruction of different complete degrees.

Silhouette extraction of the surgical instrument

In this experiment, we extract silhouettes from 30 images of the surgical instrument obtained from different viewpoints. Four initial intensity seed parameters are set to compare the extraction efficiency ( $μ$ = 0.19, 0.22, 0.25, and 0.28). The results for the instrument’s silhouette extraction and reprojection are shown in Figure 8(a) and (b). The results show that the proposed method can reliably extract complete and clean silhouettes. The more detailed parts of the drill can be extracted without the phenomenon of over-segmentation. The results of the silhouette extraction efficiency under different initial intensity seed parameters are shown in Figure 8(c). It shows that the time cost for extraction efficiency remains stable at 30–40 s.

Figure 8.

Silhouette extraction: (a) Silhouette image is reprojected onto the instrument image, displaying different detailed parts, (b) the result of extraction, and (c) extraction time cost for different images.

Reconstruction of surgical instrument and silhouette monitoring

In this experiment, we construct a $3 D \times 2 D \times 3 D$ voxel block in the marker coordinate system based on the diameter $D$ of the marker model. Four voxel densities (d = 0.50, 0.75, 1.00, and 1.25 mm) are set to compare the reconstruction efficiency. Figure 9(a) shows the voxel model, reconstruction model, and texture mapping results of the surgical handpiece under d = 0.75 mm. The reconstruction result with d less than 0.75 mm shows better edge details of the instrument. Due to the pyramid criterion, each voxel needs to be determined within all silhouette images. Therefore, each image produces a certain time cost for voxel carving. The time cost increase with the increase in voxel density, as shown in Figure 9(b). The silhouette image with large pose estimation error $e_{rep}$ is removed if $e_{rep} > T_{rep}$ , where $T_{rep}$ is a predefined threshold ( $T_{rep} = \bar{e} + σ$ , where $\bar{e}$ is the average of reprojection error, $σ$ is the represents the corresponding standard deviation). When d is 0.75 mm, the cumulative reconstruction time is 100 s. The reconstruction model of the surgical handpiece is projected onto the medical images and virtual tooth by camera-to-oral registration to achieve visualization, as shown in Figure 9(c). Simultaneously, the directional indication of the surgical handpiece is projected to assist surgeons in correcting implant angle deviations.

Figure 9.

Marker-based surgical instrument reconstruction: (a) the results of voxel point cloud model, reconstructed model, texture mapping model, (b) the reconstructed time cost for different silhouette images under different voxel densities, and (c) the reconstruction model is projected onto medical images.

Instrument calibration and point localization accuracy evaluation

In this experiment, the fiducial registration error (FRE) is used to evaluate the localization accuracy after full-tool calibration,³² as

\begin{matrix} FRE = \sqrt{\frac{\sum_{i = 1}^{n} ∥ (R_{real}^{measured} {\bar{P}}_{measured}^{i} + t_{real}^{measured}) - P_{real}^{i} ∥_{2}^{2}}{n}}, \end{matrix}

(4)

where $R_{real}^{measured}$ and $t_{real}^{measured}$ are the transformation matrix from measured points to actual model coordinate, ${\bar{P}}_{measured}^{i}$ denotes the mean measured value with static measurement over 200 frames, $P_{real}^{i}$ is the actual points, and $n$ is the total number of the localization by the surgical handpiece.

We use the instrument’s stylus-tip to position the holes in the localization test block. A $5 \times 5$ array of circular holes are processed on the block’s inclined plane. Simultaneously, the pivot calibration method is set as the control group. As shown in Figure 10(c) to (h), the results are given, including the 25 holes’ center point projection after registration, the distance error between the center of each hole and the registered standard point, and the distance errors in X, Y, and Z axis. The results show that our proposed method has similar accuracy to the control group. The average FRE is 0.496 mm using our method, and 0.450 mm for the control group. The distance errors for the 25 localization points in Figure 10(d) and (g) show that only minor differences exist between the two methods. Figure 10(e) and (h) show that the reprojection errors in X (Y and Z) axis is 0.174 ± 0.136 mm (0.326 ± 0.203 mm and 0.253 ± 0.229 mm) using our method, and 0.261 ± 0.179 mm (0.178 ± 0.114 mm and 0.256 ± 0.243 mm) for the control group. These results show our localization accuracy is similar to the control group in different direction axis.

Figure 10.

Full-tool calibration of the surgical handpiece: (a) physical setup of the FRE experiment, (b) localization test block model, (c) the result of FRE using our calibration method (The red indicate “actual” coordinates while the blue points indicate the measured points), (d) distance error for each points using our method, (e) accuracy evaluation for X, Y, and Z axis using our method, and (f–h) are measured for the control group (The red indicate “actual” coordinates while the black points indicate the measured points).

Vector localization in orientation accuracy evaluation

In this experiment, the relative orientation error (ROE) is used to evaluate the orientation error in drill implantation.³³ The results of the normal vector of the drill in the hole on the orientation test hemisphere are recorded sequentially, as shown in Figure 11(a). The diameter of the hole $D^{'}$ is set based on to the diameter $D$ of the drill ( $D^{'} = D + 0.02$ mm). The hemisphere has 25 holes across four cross sections (0°, 45°, 90°, and 135°), as shown in Figure 11(c). The angle between the central axes of the holes measures 30°, as shown in Figure 11(b). For the case of two adjacent paths in one cross section, the equation is as follow

ROE = | ({\bar{α}}_{o_{i + 1}} - {\bar{α}}_{o_{i}}) - o_{std} |,

(5)

where $i \in [1, 6]$ , ${\bar{α}}_{o_{i}}$ denotes the mean orientation value with static measurement over 200 frames, and the standard angle $o_{std}$ is 30°. As shown in Figure 11(d), the comprehensive ROE for four directions on the hemisphere remains stable at 1.401° ± 0.304°.

Figure 11.

Orientation accuracy evaluation: (a) physical setup of the ROE experiment, (b) hemisphere containing 25 paths, which included angle between two adjacent paths in the same cross section is 30°, (c) hemisphere perspective view with four directions (0°, 45°, 90°, and 135°), and (d) accuracy evaluation in four cross sections and the average ROE.

HydraMarker-based localization and comprehensive effect evaluation

In this experiment, the raster ruler motion platform is used to evaluate the accuracy of HydraMarker-based localization, as shown in Figure 12(a). In the following experiment, different levels of occlusion were introduced, and the localization accuracy was evaluated based on the movement of the surgical handpiece. The occlusion rates were set to 0%, 25%, 50%, and 65%. The ArUco-based instrument localization method was used as the control group for comparative evaluation, as shown in Figure 12(a). Because surgeons typically perform slow and fine instrument manipulations during surgery. The surgical handpiece was fixed on a raster ruler motion platform. The platform was programmed to move in incremental displacements of 2 mm (4 or 8 mm) per step, as shown in Figure 12(b). When using HydraMarker, the localization movement error remained stable and below 0.250 mm under different occlusion conditions. In the absence of occlusion, the movement error reached as low as 0.070 mm at a displacement of 2 mm. Although the error exhibited a slight increase when the occlusion rate exceeded 50%, it remained within an acceptable range. In contrast, when using the ArUco marker, the movement error increased continuously with increasing occlusion levels.

Figure 12.

HydraMarker-based localization accuracy evaluation: (a) physical setup of the motion experiment, (b) the distance error result of different occlusion degrees (0%, 25%, 50%, and 65%) under different displacement tests (2, 4, and 8 mm), and (c) the results of dynamic motion tracking under different velocities.

Figure 12(b) shows the results of the two methods for motion error at different distances. In comparison to the control group, our method performs slightly better. The result of the dynamic motion path is shown in Figure 12(c). We set four platform motion velocities to test the tracking performance of HydraMarker-based localization (velocity = 5, 10, 15, and 20 mm/s). Figure 12(c) shows that the tracking performance is good when the velocity is less than 10 mm/s. A part of tracking results are lost at 15 mm/s. When the velocity exceeds 20 mm/s, it is only partially successful. Given that surgical instruments are typically manipulated at low speeds during procedures, our method is well-suited to accurately and reliably track their motion, thereby ensuring effective surgical localization and silhouette monitoring.

Discussion

As can be seen from Table 1, the position block is 88% lighter than the surgical handpiece in weight. HydraMarker on the positioning block can be detected and identified on a non-high-performance computer in 42 ms. In our marker reconstruction method, users only hold and turn the instrument to complete the marker spatial point cloud reconstruction. The image pool has an adaptive capability that continuously improves the accuracy of reconstruction. The second thread uses the incremental dendriform reconstruction, which reduces error accumulation caused by splicing in a fixed order.

Table 1.

The parameters of our system and comparison with control group.

Indicators	Physical parameters
Radius of the positioning block (mm)	20
Weight of the positioning block (g)	$\approx 21.5$
Weight of the surgical (g)	$\approx 170.5$
HydraMarker detector time (mhandpieces)	42
Indicators	Our system	Control group
Calibration	Full-tool calibration	Pivot calibration
Localization	Point + Vector	Point
Registration accuracy (FRE) (mm)	0.496	0.450
Orientation accuracy (ROE) (°)	1.401	-
Localization accuracy (mm)	$\leq 0.06$	$\leq 0.05$

Surgical instrument full-tool reconstruction is the key process for enhancing localization performance and the effectiveness of visualization in this paper. In our method, a complete and clean silhouette can be extracted in a non-complexity background. The extraction time for each image on a non-high-performance computer is 30–40 s. Silhouettes with large pose estimation errors are removed. And the reconstruction time on a non-high-performance computer is 100 s. Minor variations in the region-growing intensity threshold primarily affected fine edge details but did not result in catastrophic failures. In addition, the voxel density determines the spatial resolution of the reconstructed model and influences both geometric fidelity and computational cost. These results indicate a resolution–efficiency trade-off rather than a hard threshold. The surgical instrument registration can be performed offline, so the registration time does not occupy the time of the operation.

In experiment, we find that in the localization result of the stylus-tip using our method is similar to that of the control group. For static motion error, our method performs sightly better. In the accuracy evaluation of FRE, the FRE for ours is 0.496 mm, which is comparable to that of the control group. More importantly, we propose a novel calibration method for surgical instrument to calibrate the vector orientation. The average ROE is 1.401°. When using the same fiducial marker, the accuracy of the proposed calibration method is comparable to that of the control group. However, the control group is unable to provide the orientation localization. After camera-to-oral registration, the complete silouette of instrument can be intuitively displayed on the medical image to provide silhouette monitoring for surgeons. In dynamic tracking, HydraMarker-based localization performs better when instrument movements are at 10 mm/s or less. Frequent pose estimation failures were observed at speeds exceeding 20 mm/s. The localization accuracy achieved in the experiment was influenced by the type of marker employed. Owing to the redundancy of the feature-dense position-sensing marker, stable pose estimation was maintained under conditions of moderate partial occlusion. As the occlusion level increased, the localization error increased gradually rather than abruptly.

It should be mentioned that the proposed navigation system is highlighted with a surgical instrument full reconstruction and vector calibration method, but it has some potential limitations. The comprehensive performance is affected by certain configurations and errors, such as the camera performance, the camera calibration error, or computer performance. These are directions of optimization and improvement for the proposed method in the future.

Conclusion

In this paper, we propose a full-tool surgical navigation, including the full-tool registration and HydraMarker-based localization methods. A novel calibration method based on the instrument model is proposed to address the problem of calibrating both the stylus-tip point and directions of the key part on instrument, and we use a feature-dense position-sensing marker as the fiducial marker. Our method provides surgeons with complete localization of the surgical instrument, and a silhouette monitoring overlaid on medical images. Experiments demonstrate that the proposed method achieves localization accuracy similar to point localization using pivot calibration. The fiducial registration error (FRE) of ours is 0.496 mm, and the relative orientation error (ROE) is 1.401°. These results validate the potential of full-tool registration and HydraMarker-based localization. The localization accuracy remain stable using HydraMarker, and it can effectively track the surgical instrument in motion during surgery. After registration, the complete silhouette projected onto medical images provides surgeons with a silhouette monitoring. We plan to further optimize the reconstruction method and calibration methods in future works. We will also develop a novel non-contact registration method based on the proposed full-tool reconstruction method. Clinical experiments on patient are to be conducted in the future, given the consent of patient as well as the positive safety assessment to further evaluate the performances of the proposed method regarding its applicability. Furthermore, we plan to extend the application of full-tool navigation to other surgical instrument geometries in future work, particularly for procedures such as orthognathic osteotomy, where instrument designs differ substantially from dental handpieces.

Footnotes

ORCID iD

Hao Lin

Ethical considerations

No human participation, patient or clinical data, imagery or material was involved in this study.

Consent to participate

Informed consent was obtained from all individual participants included in this study.

Consent for publication

All participants provided informed consent for the publication of anonymized data and related materials included in this manuscript.

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by [Hao Lin], [Bingwei He], and [Mingzhu Zhu]. The first draft of the manuscript was written by [Hao Lin] and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (Grant 62373109), Major Scientific Research Projects in Fujian Province (Grant 2021ZD01003), Joint Funds for the innovation of science and Technology Fujian province (Grant 2021Y9190), Science and Technology Project of Fujian Province (Grant 2021Y0052).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The data generated and analyzed during this study, including registration parameters and quantitative evaluation results, are not publicly available as they form part of ongoing system development.

References

Sun

Lin

Chen

, et al. Robot-assisted mandibular angle osteotomy using electromagnetic navigation. Ann Transl Med 2021; 9(7): 567.

Brouwer de Koning

Geldof

van Veen

RLP

, et al. Electromagnetic surgical navigation in patients undergoing mandibular surgery. Sci Rep 2021; 11(1): 4657.

Reijnders

Coppes

Van Hulzen

, et al. Image guided surgery: new technology for surgery of soft tissue and bone sarcomas. Eur J Surg Oncol 2007; 33(3): 390–398.

Meng

Geng

Luo

, et al. An occlusion-free intraoperative active navigation system for orthopedic surgery. IEEE Trans Instrum Meas 2024; 73: 1–10.

Zhu

Wang

, et al. Robust oral localization based on multicamera tracking of self-identifying markers. IEEE Trans Instrum Meas 2024; 73: 1–11.

Zhang

Chan

, et al. Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker. Int J Comput Assist Radiol Surg 2017; 12: 921–930.

Zhu

Wang

, et al. A vision-based navigation system with markerless image registration and position-sensing localization for oral and maxillofacial surgery. IEEE Trans Instrum Meas 2023; 72: 1–11.

Cartucho

Wang

Huang

, et al. An enhanced marker pattern that achieves improved accuracy in surgical tool tracking. Comput Methods Biomech Biomed Eng Imaging Vis 2022; 10(4): 400–408.

Wang

Kobayashi

Sakuma

. Coarse-to-fine dot array marker detection with accurate edge localization for stereo visual tracking. Biomed Signal Process Control 2015; 15: 49–59.

10.

Deeb

Tran

Deeb

. Computer-aided planning and placement in implant surgery. Atlas Oral Maxillofac Surg Clin North Am 2020; 28(2): 53–58.

11.

Fan

Chen

Wang

, et al. Spatial position measurement system for surgical navigation using 3-d image marker-based tracking tools with compact volume. IEEE Trans Biomed Eng 2018; 65(2): 378–389.

12.

Xie

Zhu

, et al. Image-guided navigation system for minimally invasive total hip arthroplasty (mitha) using an improved position-sensing marker. Int J Comput Assist Radiol Surg 2023; 18(12): 2155–2166.

13.

Liu

Guo

Yuan

. Graph-based surgical instrument adaptive segmentation via domain-common knowledge. IEEE Trans Med Imaging 2022; 41(3): 715–726.

14.

Xue

Liu

, et al. Oriented localization of surgical tools by location encoding. IEEE Trans Biomed Eng 2022; 69(4): 1469–1480.

15.

Hein

Cavalcanti

Suter

, et al. Next-generation surgical navigation: marker-less multi-view 6dof pose estimation of surgical instruments. Med Image Anal 2025; 103: 103613.

16.

Cai

Zhu

, et al. Distributed visual positioning for surgical instrument tracking. Phys Eng Sci -med 2024; 47(1): 273–286.

17.

Deng

Ruan

Hao

, et al. Hand-eye calibration of line structured-light sensor by scanning and reconstruction of a free-placed standard cylindrical target. Measurement 2024; 229: 114487.

18.

Schonberger

Frahm

. Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.4104–4113.

19.

Yang

Fang

, et al. An improved sfs method for achieving fast, high-precision, and widely adaptable 3-d reconstruction. IEEE Trans Instrum Meas 2024; 73: 1–9.

20.

Morgan

Jayarathne

Rankin

, et al. Hand-eye calibration for surgical cameras: a procrustean perspective-n-point solution. Int J Comput Assist Radiol Surg 2017; 12: 1141–1149.

21.

Chong

Dhesi

Makdissi

. Computer-aided dynamic navigation: a novel method for guided endodontics. Quintessence Int 2019; 50(3): 196–202.

22.

Aminov

Regan

Giles

, et al. Targeting repeatability of a less obtrusive surgical navigation procedure for total shoulder arthroplasty. Int J Comput Assist Radiol Surg 2022; 17: 283–293.

23.

Block

Emery

. Static or dynamic navigation for implant placement—choosing the method of guidance. J Oral Maxillofac Surg 2016; 74(2): 269–277.

24.

Block

Emery

Lank

, et al. Implant placement accuracy using dynamic navigation. Int J Oral Maxillofac Implants 2017; 32(1): 92–99.

25.

Landaeta-Quinones

Hernandez

Zarroug

. Computer-assisted surgery: applications in dentistry and oral and maxillofacial surgery. Dent Clin North Am 2018; 62(3): 403–420.

26.

Panchal

Mahmood

Retana

, et al. Dynamic navigation for dental implant surgery. Oral Maxillofac Surg Clin 2019; 31(4): 539–547.

27.

Zhu

, et al. Hydramarker: efficient, flexible, and multifold marker field generation. IEEE Trans Pattern Anal Mach Intell 2023; 45(5): 5849–5861.

28.

Jiang

, et al. An improved canny edge detection algorithm with iteration gradient filter. In 2022 6th International conference on imaging, signal processing and communications (ICISPC), 2022, pp.16–21.

29.

Zhang

Zhou

Wang

, et al. Region growing based on 2-D–3-D mutual projections for visible point cloud segmentation. IEEE Trans Instrum Meas 2021; 70: 1–13.

30.

Min

Zhu

Meng

MQH

. Accuracy assessment of an n-ocular motion capture system for surgical tool tip tracking using pivot calibration. In: 2016 IEEE International conference on information and automation (ICIA), 2016, pp.1630–1634. IEEE.

31.

Garrido-Jurado

Muñoz-Salinas

Madrid-Cuevas

, et al. Generation of fiducial marker dictionaries using mixed integer linear programming. Pattern Recognit 2016; 51: 481–491.

32.

Fitzpatrick

West

Maurer

. Predicting error in rigid-body point-based registration. IEEE Trans Med Imaging 1998; 17(5): 694–702.

33.

Chi

Long

Zhang

, et al. Enhancing accuracy of surgical stylus-tip tracking: a comparative calibration study. Measurement 2025; 242: 115916.