DynaGrad: A Novel Gradient Descent With Adaptive Dual Learning Rates & Momenta for Improved Optimization and Accelerated Convergence in Deep Neural Networks

Overview

Machine Learning (ML) optimizers are widely used to optimize the weights and biases of deep neural networks. However, current methods can suffer from issues like slow convergence and getting trapped in poor local minima. In this work, we propose a novel gradient descent algorithm with dual adaptive learning rates and momenta that aims to improve optimization and achieve faster convergence. The key idea is to dynamically adjust the learning rate and momentum for each weight and bias based on the magnitude and direction of its partial derivative while incorporating a slow and fast gradient, moving at different paces along the loss function. Weights and biases with larger gradient magnitudes are updated more aggressively, while those with noisy gradients are updated more conservatively. The momentum is also adapted to accelerate training along dimensions with consistent gradient directions and dampen oscillations in other dimensions. We evaluate our proposed algorithm on various deep neural network architectures and datasets. Results demonstrate faster decreases in the training loss and test error compared to the current state-of-the-art ML optimizers, and other adaptive methods. The improved optimization enables reduced training times and generally leads to better model performance. This adaptive gradient descent approach can enable more efficient training of deep neural networks across a variety of applications. Our approach, DynaGrad, is 15.6% faster than state-of-the-art algorithms and achieves a 2% lower loss. This leads to higher accuracy in deep neural networks, for medical imaging tasks, and any kind of research that involves advanced machine learning techniques.

Awards (4)

• Third Award of $1,000 $1,000
• Air Force Research Laboratory on behalf of the United States Air Force: Glass trophy and USAF medal for each recipient
• Air Force Research Laboratory on behalf of the United States Air Force: First Award of $750 in each Regeneron ISEF Category, $750
• National Security Agency Research Directorate : Third Place Award “Cybersecurity”

Competition history

• ISEF 2024 Systems Software · Entry SOFT046T · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.