Hardware Implementation of a Two-Layer Neural Network Based on FPGA: Analysis of the Efficiency of Using ReLU and LeakyReLU Activation Functions

Abstract

Methods for the efficient hardware implementation of neural networks on FPGA-type programm able logic integrated circuits are investigated. A key aspect is the influence of the choice of activation functions on the characteristics of the developed device. An approach to the use of activation functions that allow for efficient hardware implementation is proposed. In particular, the use of LeakyReLU as a compromise between computational simplicity and classification accuracy is justified. To test the approach, a two-layer feedforward network architecture was developed, its hyperparameters were optimized, and hardware implementation was carried out on a PYNQ Z2 board. An analysis of the impact of the bit depth of the coefficients in fixed-point format on the re cognition accuracy of the MNIST database and on hardware costs is conducted. The optimal bit depth of the fractional part (9 bits) was experimentally determined, ensuring an accuracy of 95.27 % while economically using the resources of programmable logic integrated circuits. Additionally, using the Kullback – Leibler divergence, the distortion of the weight distribution during quantization was estimated, on the basis of which a regression model was constructed to predict the accuracy of a neural network with quantized coefficients.