A new activation function Week 10

In our last blog post we discussed the idea of two activations functions RELU and tanh. RELU the more commonly used function written as max(x,0) would be possible to code in homomorphic encryption. However to calculate the derivative we need to check whether the value of max(x,0) == 0. Equality is impossible in homomorphic encryption, therefore we would have to decrypt the value of max(x,0) which itself makes calculating max(x,0) pointless and puts too much security risk. To be precise, we want to avoid running any decryption operations in loops. However calculating the RELU function will be done in a loop and provide massive security risk. Calculating the max itself is quite computationally intensive making the RELU activation function difficult to use.

The next option we discussed was tanh. However tanh has two problems. It is used less often than RELU due to its derivatives approximating 0 leading to less change in the values. However our approximate function is degree 5(already too big to be normally used) and lacks the same structure as the regular tanh function having certain negative values with a derivative of the wrong sign. This will cause a massive amount of problem and led to an extremely inaccurate and time-consuming process as expected.

This leads us to consider the point of an activation function in the first place(to introduce non-linearity to our system). If our activation function was just mx+b we would just be doing linear regression. Therefore one option is to try activation functions in the form of polynomials. the square activation function had been used briefly and is worth a try here. Although one might say that this is just polynomial regression, using a polynomial approximation for tanh accomplishes the same thing expect with bounded derivatives.  In our final blog post I will discuss my finding with this new activation function,