Keywords: transfer function

Transfer Functions1

As we know, in a neuron there is an operation called threshold operation which acts as a switch, when the conditions it needs are reached it returns a state otherwise it will keep another state. A transfer function is a mathematical model for this mechanism. And this short post is a survey of the transfer functions used in neural network design.

Hard Limit

$$
f(x) = \begin{cases}
0& \text{ if } x<0\\
1& \text{ if } x\geq 0
\end{cases}
$$

Symmetrical Hard Limit

$$
f(x) = \begin{cases}
-1& \text{ if } x<0\\
1& \text{ if } x\geq 0
\end{cases}
$$

Linear

$$
f(x)=x
$$

Saturating Linear

$$
f(x) = \begin{cases}
0& \text{ if } x<0\\
x&\text{ if } 0\leq x\leq 1\\
1& \text{ if } x> 1
\end{cases}
$$

Symmetric Saturating Linear

$$
f(x) = \begin{cases}
-1& \text{ if } x<0\\
x&\text{ if } 0\leq x\leq 1\\
1& \text{ if } x> 1
\end{cases}
$$

Log-Sigmoid

$$
f(x)=\frac{1}{1+e^{-x}}
$$

Hyperbolic Tangent Sigmoid

$$
f(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}
$$

Positive Linear

$$
f(x) = \begin{cases}
0& \text{ if } x<0\\
x& \text{ if } x\geq 0
\end{cases}
$$

Competitive

$$
f(x) = \begin{cases}
1& \text{ if neuron with max x} \\
0& \text{ else }
\end{cases}
$$

References


1 Demuth, H.B., Beale, M.H., De Jess, O. and Hagan, M.T., 2014. Neural network design. Martin Hagan.
Last modified: March 24, 2020