Research Articles
HUANG Wei, LI Zhigang, HOU Xinyu, LIU Guangyao, WANG Lei, LAN Yanghui, LIU Jinhong, WANG Yi
3D-reconstructed relocation of spectral feature is a key technology in multi-spectral imaging fusion and 3D reconstruction. Here, the feasibility was to explore about relocating spectral data onto 3D-reconstructed dense point cloud through a new neural network HF-Net, one deep-learning-based AI technology and also a hierarchical localization approach based on a monolithic CNN that is able to simultaneously predict local features and global descriptors for spectral localization. Such an HF-Net was adopted to carry out the spectral relocation of heterogeneous localization. Specifically, the MobileNet and NetVLAD layers were taken to extract the global descriptors among the dataset of three-dimensional color point cloud from the spectral image so as to find the approximate position of spectral image in the three-dimensional point cloud. With the conjectural locations obtained through the prior global retrieval within those candidate places, the SuperPoint was utilized to get the local descriptors and key point scores of the spectral image so that the matching spectral points were found in the three-dimensional point cloud, therewith having mapped out the spectral information and three-dimensional reconstruction pattern. By leveraging the learned descriptors, this assay achieved remarkable localization robustness across large variations of appearance, demonstrating more robust and efficient than SSD algorithm. Due to limits with GPU (graphics processing unit) memory, the extracted spectral features were down-sampled from image to the largest resolution of 6016 × 4512 pixels. With HF-Net being trained through multi-task distillation in TensorFlow 1.12, a spectral image had been able to relocate into 3D reconstruction pattern in 12s under having run on the device of NVidia TESLA V100 with 32G memory and CPU of Intel (R) Xeon (R) Silver 4114 with 12G memory. The approach proposed here can realize the fine positioning of 3D spatial information and spectral features of 3D physical evidence. At present, there seems no such an exploration in forensic science home and abroad, revealing that the exploration tried here is a new application of artificial intelligence to the full-dimensional imaging data fusion technology of physical evidence. Such an HF-Net-based relocation is accurate, scalable, efficient, and a monolithic deep neural network choice for descriptor extraction, capable of achieving higher exactness on large-scale multi-spectral imaging fusion and 3D reconstruction.