Navigating Parameter Space: Mitigating Catastrophic Forgetting in Continual Learning

Overview

The most sought-after goal in machine learning is achieving artificial general intelligence, which requires stable memory and flexibility in learning. Existing models are dependent on all data being available during training and do not learn during inference. If the model is required to learn new samples continuously, performance on previous data will decrease sharply. This phenomenon, coined “Catastrophic Forgetting”, is the main challenge of Continual Learning. The most effective option to prevent Catastrophic Forgetting is retraining a model from scratch with new samples. However, retraining is extremely expensive and time consuming, especially for large models. This process is also impractical in real world scenarios, because the model cannot adapt. This work focuses on improving Parameter Regularization approaches for class incremental learning, the most challenging continual learning scenario. Elastic Weight Consolidation (EWC) is thoroughly studied through hyperparameter optimization and varying model architectures. The results reveal that the structure of the last Fully Connected (FC) Layer significantly affects accuracy, with Dynamic FC performing much better than Static FC. Furthermore, optimizing the Lambda hyperparameter improves Dynamic FC but not Static FC. To improve upon EWC, a novel Range Matrix method is introduced to further constrain all parameters through a custom-built optimizer. Static FC with Range Matrix allows for an improvement over EWC Static FC through hyperparameter tuning. In conclusion, this work shows that the FC layer with hyperparameter optimization is crucial for the success of Parameter Regularization methods, enabling improvement over previous Parameter Regularization works in CIL by almost doubling accuracy.

Competition history

ISEF 2025 Robotics and Intelligent Machines · Entry ROBO013 Affiliated fair in CA, United States of America

Resources

ISEF abstract page

Related projects

ISEF · 2019

Weight Friction: A Simple Method to Overcome Catastrophic Forgetting and Enable Continual Learning in Neural Networks

ISEF · 2025

Improving Generalizability in Exemplar-Free Class-Incremental Learning

ISEF · 2022

Neural Networks Learn Lazily: Improving Generalization and Adversarial Robustness via Learning Capacity-Complexity Constraints

JSHS · 2023

Utilizing the Learned Latent Structure from Dimensionality Reduction Algorithms to Prevent the Effects of Catastrophic Forgetting in Neural Networks

JSHS · 2023

Hybrid Plasticity: A Biologically Realistic Meta-Learning Algorithm to Increase the Adaptability and Efficiency of Artificial Intelligence Systems

ISEF · 2026