Keywords: transfer function

## Transfer Functions1

As we know, in a neuron there is an operation called threshold operation which acts as a switch, when the conditions it needs are reached it returns a state otherwise it will keep another state. A transfer function is a mathematical model for this mechanism. And this short post is a survey of the transfer functions used in neural network design.

## Hard Limit

$$f(x) = \begin{cases} 0& \text{ if } x<0\\ 1& \text{ if } x\geq 0 \end{cases}$$

## Symmetrical Hard Limit

$$f(x) = \begin{cases} -1& \text{ if } x<0\\ 1& \text{ if } x\geq 0 \end{cases}$$

## Linear

$$f(x)=x$$

## Saturating Linear

$$f(x) = \begin{cases} 0& \text{ if } x<0\\ x&\text{ if } 0\leq x\leq 1\\ 1& \text{ if } x> 1 \end{cases}$$

## Symmetric Saturating Linear

$$f(x) = \begin{cases} -1& \text{ if } x<0\\ x&\text{ if } 0\leq x\leq 1\\ 1& \text{ if } x> 1 \end{cases}$$

## Log-Sigmoid

$$f(x)=\frac{1}{1+e^{-x}}$$

## Hyperbolic Tangent Sigmoid

$$f(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$$

## Positive Linear

$$f(x) = \begin{cases} 0& \text{ if } x<0\\ x& \text{ if } x\geq 0 \end{cases}$$

## Competitive

$$f(x) = \begin{cases} 1& \text{ if neuron with max x} \\ 0& \text{ else } \end{cases}$$

## References

1 Demuth, H.B., Beale, M.H., De Jess, O. and Hagan, M.T., 2014. Neural network design. Martin Hagan.