Maple Examples of Activation Functions

Identity Function

The identity function is given by

> f1 := proc(x) x; end;

> plot(f1(x),x=-10..10);

Step Function

The step function is if and if . This is also called the heaviside function. Another common variation is for it to take on values -1 and +1 as shown below.

> f2 := proc(x) 2*Heaviside(x) -1 end;

> plot(f2(x),x=-10..10);

Logistic Function (Sigmoid)

The logistic function has the form

> f3 := proc(x,a) 1/(1 + exp(-a*x)) end;

> plot(f3(x,1),x=-10..10);

> with(plots):

The parameter "a" in the logistic function determines how steep it is. The larger "a", the steeper it is.

> display([plot(f3(x,1),x=-10..10),

> plot(f3(x,2),x=-10..10, color = blue),

> plot(f3(x,.5),x=-10..10, color = green)]);

Symmetric Sigmoid

The symmetric sigmoid is simply the sigmoid that is stretched so that the y range is 2 and then shifted down by 1 so that it ranges between -1 and 1. If g(x) is the standard sigmoid then the symmetric sigmoid is

> f4 := proc(x) 2*f3(x,1)-1 end;

> plot(f4(x),x=-10..10);

The symmetric sigmoid differs from the hyperbolic tangent by a constant factor. As you can see, the graph below is identical to the graph for the symmetric sigmoid.

> plot(tanh(.5*x),x=-10..10);

Radial Basis Functions

A radial basis function is simply a gaussian, . It is called local because, unlike the previous functions, it is essentially zero everywhere except in a small region.

> f5 := proc(x,a) exp(-a * x^2); end;

> plot(f5(x,1),x=-10..10);

Derivatives

The derivative of the identity function is just 1. That is, if f(x) is the identity then

The derivative of the step function is not defined which is exactly why it isn't used.

The nice feature of sigmoids is that their derivatives are easy to compute. If f(x) is the logistic function above then .

> diff(f3(x,a),x);

> simplify(f3(x,a)*(1-f3(x,a)));

This is also true of hyperbolic tangent . If f(x) is tanh the

> diff(tanh(x),x);

>