Phish: A Novel Hyper-Optimizable Activation Function

Overview

Deep-learning models estimate values using backpropagation. The activation function within hidden layers is a critical component to minimizing loss in deep neural-networks. Rectified Linear (ReLU) has been the dominant activation function for the past decade. Swish and Mish are newer activation functions that have shown to yield better results than ReLU given specific circumstances. Phish is a robust non-monotonic activation function proposed here. It is a composite function defined as f(x) = xTanH(GELU(x)), where no discontinuities are apparent in the differentiated graph on the domain observed. Generalized networks were constructed using different activation functions. SoftMax was the output function. Using images from heterogeneous medical imaging databanks, these networks were trained to minimize sparse categorical crossentropy. A large- scale cross-validation was simulated using stochastic Markov chains to account for the law of large numbers for the probability values. Statistical tests support the research hypothesis stating Phish could outperform other activation functions in image classification. In a first of its kind, Phish hybridizes Identity, Hyperbolic, and Gaussian mathematical relationships to create a unique transformation profile using continuity, non- monotonicity, and differentiability. When used disease classification models, the core math engine identified Coronavirus, Tuberculosis, Carcinoma, and Pneumonia from 96-99% accuracy. The models were also adept at identifying subtypes and malignancy of lung cancer in addition to viral and bacterial chest infections. The next generation activation function provides state-of-the-art training dynamics for expediting subvariants of stochastic gradient descent backpropagation. Future experiments could involve using Phish in generative adversarial networks training in an unsupervised two-player minimax framework.

Competition history

• JSHS 2022 Category not listed

Resources

• National JSHS abstract book ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.