RAMIS：Increasing Robustness and Accuracy in Medical Image Segmentation with Hybrid CNN-Transformer Synergy

Abstract

Hybrid architectures based on Convolutional Neural Networks (CNN) and Vision Transformer (ViT) have become an important research direction in medical image segmentation in recent years. However, the currently popular hybrid architectures weaken the decision making process within the transformer model, and the upsampling of convolution stacking is difficult to restore the blurred boundaries of the target area. To improve the feature learning capability by addressing these issues, we propose RAMIS, a novel hybrid architecture for general medical image segmentation. RAMIS develops implicit neural representation and self-distillation to simultaneously obtain the super-resolution details and core features of the image as input to the Transformer encoder. Meanwhile, RAMIS explores an unsupervised learning CNN to obtain the initial input to the Transformer decoder, which not only explicitly considers the correlation within different samples, reduces the constraints on small datasets, but also fully leverages the potential of Transformer’s cross-attention for optimizing segmentation results. RAMIS designs a multi-resolution interaction network to post-process the Transformer output and solves the problem of blurred segmentation boundaries by combining super-resolution image. We extensively evaluate RAMIS on five datasets from three typical publicly available medical image segmentation datasets. Extensive experimental results demonstrate the general applicability and superior performance of the proposed method.

(2) RAMIS intermediate output results

We show more intermediate results of the proposed method on three different datasets in this section.

[1/3]

Fig. 4 Interim results of RAMIS on the skin cancer lesion dataset. (a-c) The first row in each group of pictures from left to right is the input image, initial segmentation image, final segmentation image, prediction and Ground Truth comparison results (red indicates false negatives, blue indicates false positives). All the pictures in the second row are the output results of the self-distillation module. The last picture in each group shows the results after magnifying the implicit neural expression by twice the resolution.

[2/3]

Fig. 5 Intermediate results of RAMIS on breast tumor ultrasound image segmentation dataset. (a-c) The first row in each group of pictures from left to right is the input image, initial segmentation image, final segmentation image, prediction and Ground Truth comparison results (red indicates false negatives, blue indicates false positives). All the pictures in the second row are the output results of the self-distillation module. The last picture in each group shows the results after magnifying the implicit neural expression by twice the resolution.

[3/3]

Fig. 6 Intermediate results of RAMIS on retinal fundus vessel segmentation dataset. (a-c) The first row in each group of pictures from left to right is the input image, initial segmentation image, final segmentation image, prediction and Ground Truth comparison results (red indicates false negatives, blue indicates false positives). All the pictures in the second row are the output results of the self-distillation module. The last picture in each group shows the results after magnifying the implicit neural expression by twice the resolution.