Title : Novel REST block for gradient optimization in deep learning models

Deep learning surpasses traditional machine learning methods in image segmentation tasks by autonomously discerning pertinent features from the input, yielding remarkable outcomes. Optimizing algorithms are an important feature of training deep learning models which aid to minimize loss and modify the model’s parameters. In training deep learning models, unstable gradient poses a challenge. The optimizing algorithm mostly use gradient descent to learn and tune up model parameters. The problem of unstable optimizing algorithm in deep learning can lead to the issue of gradients becoming either extremely small (vanishing gradients) or very large (exploding gradients) during the training of deep neural networks. A very high or a very low gradient of the loss effects the ability of the optimizer algorithm to learn properly. Models based on fewer/shallow layers, such as Convolution Neural Network (CNN) as well as deeper/dense layered variants like U-Net have shown prominent segmentation results. Nevertheless, they frequently experience overfitting and unstable optimization algorithms during training, especially in the model’s early layers. In order to stabilize the optimizing algorithm, we propose a novel block which comprises of four sub-blocks collectively called the REST block; Residual Block, Efficient Channel Attention, 2D convolution layers, Stochastic Depth and a three layered Dense Block with three feature maps to reduce computational cost. The REST is embedded in the decoder layer of the U-Net model after the last up- sampling convolution. In the CNN model, REST is integrated after each maxpooling layer to recalibrate the feature maps before being sent in the subsequent stages of the network. Some features of REST are used after the first and fifth Conv2D layer, where overfitting is more likely to happen. The aforementioned models were successfully able to suppress the unstable gradient in order to improve optimizing algorithm and overcome overfitting of the model. Three imbalanced medical datasets were used to test the model including DRIVE, BUS2017 and CVC- Clinic. The outcomes of the experiment demonstrate that neural network models having deep as well as shallow layers, embedded with REST block helps to stabilize gradient. While being trained, the suggested model successfully decreased the convergence time through a drastic reduction in the loss up to 0.004% and increasing accuracy to 99.7% in comparison with other models. Comparison of models integrated with state-of-the-art REST block show promising results by reaching 0.982±0.002 in terms of testing accuracy while using 95% confidence interval on five fold cross validation. 

Dr. Haroon Haider is an accomplished academic with over 21 years of teaching experience across various universities. He has been affiliated with ROOTS IVY for the past four years and currently serves as Dean, Faculty of Computing, and Country Program Lead. He provides academic leadership while maintaining international standards through collaborations with UK partner institutions. Holding a PhD in Artificial Intelligence, his research focuses on emerging AI trends in medical health and autonomous vehicles. He actively mentors students in computer vision and NLP. Dr. Haider is an invited speaker at international conferences and has contributed multiple publications, including innovative neural network models and adaptive loss functions.