Multilevel thresholding is a fundamental image segmentation technique widely used in medical image processing and computer vision applications. Although Otsu’s variance-based method provides effective segmentation results, its computational cost increases exponentially with the number of threshold levels, making it unsuitable for high-dimensional and real-time medical imaging tasks. To overcome this limitation, metaheuristic optimization algorithms have been extensively employed due to their ability to efficiently search complex solution spaces. However, many conventional metaheuristics suffer from premature convergence, slow convergence speed, and insufficient balance between exploration and exploitation, particularly at higher threshold levels. In this study, the Red Fox Optimization (RFO) algorithm and an enhanced version, termed Modified Red Fox Optimization (MRFO), are investigated for multilevel image thresholding applications. Inspired by the adaptive hunting behavior of red foxes, RFO offers an effective balance between global exploration and local exploitation. To further improve its performance, MRFO incorporates several strategic enhancements, including chaotic map integration, an adaptive control factor, a modified position update mechanism, and an adaptive mutation operator. These improvements are designed to enhance population diversity, accelerate convergence, and prevent entrapment in local optima. The proposed MRFO algorithm is evaluated on Brain MRI, Retina, Cells3D, and Mitosis image datasets for multilevel thresholding problems involving 2 to 5 threshold levels. Its performance is compared with Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Ant Colony Optimization (ACO), and Grey Wolf Optimization (GWO). Experimental results demonstrate that MRFO consistently achieves superior segmentation performance in terms of PSNR and SSIM while exhibiting faster computational efficiency and more stable convergence behavior than the compared algorithms. The findings confirm the effectiveness of MRFO as a robust and efficient optimization framework for complex medical image segmentation tasks.