前馈神经网络（Feedforward Neural Network，FNN）

$$\begin{align} \end{align}$$

1 前馈神经网络结构及前向传播

训练数据集
\begin{align} \\& T = \left\{ \left( \mathbf{x}_{1}, y_{1} \right), \left( \mathbf{x}_{2}, y_{2} \right), \cdots, \left(\mathbf{x}_i,y_i\right),\cdots,\left( \mathbf{x}_{N}, y_{N} \right) \right\} \end{align}
其中，$\mathbf{x}_{i}$为第$i$个特征向量（实例），$\mathbf{x}_{i}=\left( x^{\left(1\right)}_i,x^{\left(2\right)}_i,\ldots ,x^{\left(j\right)}_i,\ldots ,x^{\left(n\right)}_i\right) ^{T} \in \mathcal{X} \subseteq \mathbb{R}^{n}$；$y_{i}$为$\mathbf{x}_{i}$的类别标记，类别标记表示为类别位置为1，其余位置为0的类别向量（one-hot编码）,$\mathbf{y}_i\in\{0,1\}^m$。

前馈神经网络输入层（层 1） \begin{align} & \mathbf{a}^{1}=\left( a_{1}^{1},a_{2}^{1},\ldots ,a_{j}^{1},\ldots ,a_{n}^{1}\right) ^{T}\\ & a_{j}^{1}=x^{\left(j\right)}_i\quad\left( j=1,2,\ldots ,n\right) \end{align}

前馈神经网络隐藏层（层 2） \begin{align} & \mathbf{a}^{2}=\left( a_{1}^{2},a_{2}^{2},\ldots ,a_{j}^{2},\ldots ,a_{p}^{2}\right) ^{T}\\ & a_{j}^{2}=\sigma \left( z_{j}^{2}\right) \\ & z_{j}^{2}= \sum _{k}w_{jk}^{2}\cdot a_{k}^{1}+b_{j}^{2}\quad\left( j=1,2,\ldots ,p\right) \\& \mathbf{z}^{2}=\left( z_{1}^{2},z_{2}^{2},\ldots ,z_{j}^{2},\ldots ,z_{p}^{2}\right) ^{T}\end{align}

前馈神经网络输出层（层 3） \begin{align} & \mathbf{a}^{3}=\left( a_{1}^{3},a_{2}^{3},\ldots ,a_{j}^{3},\ldots,a_{m}^{3}\right) ^{T}\\ & a_{j}^{3}=\sigma \left( z_{j}^{3}\right) \\ & z_{j}^{3}= \sum _{k}w_{jk}^{3}\cdot a_{k}^{2}+b_{j}^{3}\quad\left( j=1,2,\ldots ,m\right) \\& \mathbf{z}^{3}=\left( z_{1}^{3},z_{2}^{3},\ldots ,z_{j}^{3},\ldots ,z_{m}^{3}\right) ^{T}\end{align}

预测输出 \begin{align} & \hat{\mathbf{y}}=\left( \hat y_{1},\hat y_{2},\ldots ,\hat y_{j},\ldots ,\hat y_{m}\right) ^{T}\\ & \hat y_{j}=a_{j}^{3}\quad\left( j=1,2,\ldots ,m\right)\end{align}

实际输出 \begin{align} & \mathbf{y}=\left( y_{1},y_{2},\ldots ,y_{j},\ldots ,y_{m}\right) ^{T} \quad\left( j=1,2,\ldots ,m\right) \end{align}

其中，$\sigma\left(\cdot\right)$为激活函数。

2 参数学习

单个实例$\mathbf{x}$的损失函数$C_{\mathbf{x}}\left(\mathbf{y},\hat{\mathbf{y}}\right)$为平方损失函数、

$$\begin{align} & C_{\mathbf{x}}=\dfrac {1} {2}\left\| \mathbf{y}-\hat {\mathbf{y}}\right\| ^{2}=\dfrac {1} {2}\sum _{j}\left( y_{j}-\hat {y}_{j}\right) ^{2} \end{align}$$

目标函数（经验风险） \begin{align} & C=\dfrac {1} {N}\sum _{\mathbf{x}}C_{\mathbf{x}} \end{align}

第$l$层参数$w_{jk}^l$更新为

$$\begin{align} w_{jk}^l & \leftarrow w_{jk}^l-\alpha \dfrac {\partial C} {\partial w_{jk}^{l}} \\ &=w_{jk}^l-\alpha\frac{1}{N}\sum_{\mathbf{x}}\dfrac {\partial C_x} {\partial w_{jk}^{l}} \end{align}$$

第$l$层参数$b_j^l$更新为

$$\begin{align} b_j^l & \leftarrow b_j^l-\alpha \dfrac {\partial C} {\partial b_j^{l}} \\ &=b_j^l-\alpha\frac{1}{N}\sum_{\mathbf{x}}\dfrac {\partial C_x} {\partial b_j^{l}} \end{align}$$

3 误差反向传播

定义第$l$层的第$j$个神经元上的误差 \begin{align} & \delta _{j}^{l}\equiv \dfrac {\partial C_{\mathbf{x}}} {\partial z_{j}^{l}} \quad\left( l=2,3\right)\end{align}

输出层误差

$$\begin{align} \delta _{j}^{3}&=\dfrac {\partial C_{\mathbf{x}}} {\partial z_{j}^{3}} \\ & =\dfrac {\partial C_{\mathbf{x}}} {\partial a_{j}^{3}}\cdot\dfrac {\partial a_{j}^{3}} {\partial z_{j}^{3}} \\ & =\dfrac {\partial \left(\frac{1}{2}\sum_{j=1}^m \left(y_j-\hat{y}_j\right)^2\right)} {\partial \hat{y}_j}\cdot \sigma '\left( z_{j}^{3}\right) \\ & = \left(\hat{y}_j-y_j\right) \cdot \sigma'\left( z_{j}^{3} \right)\quad\left( j=1,2,\ldots ,m\right)\end{align}$$

隐藏层误差

$$\begin{align} \delta _{j}^{2}&=\dfrac {\partial C_{\mathbf{x}}} {\partial z_{j}^{2}}\\ & =\sum _{k}\dfrac {\partial C_{\mathbf{x}}} {\partial z_{k}^{3}}\cdot \dfrac {\partial z_{k}^{3}} {\partial z_{j}^{2}} \\ & = \sum _{k} \dfrac {\partial z_{k}^{3}} {\partial z_{j}^{2}}\cdot\delta _{k}^{3}\\ & = \sum _{k} \dfrac {\partial \left( \sum _{j}w_{kj}^{3}\cdot a_{j}^{2}+b_{k}^{3}\right)} {\partial z_{j}^{2}}\cdot\delta _{k}^{3}\\ & = \sum _{k} \dfrac {\partial \left( \sum _{j}w_{kj}^{3}\cdot \sigma \left( z_{j}^{2}\right)+b_{k}^{3}\right)} {\partial z_{j}^{2}}\cdot\delta _{k}^{3}\\ & = \sum _{k} w_{kj}^{3}\cdot \sigma '\left( z_{j}^{2}\right) \cdot\delta _{k}^{3} \\ & = \sigma '\left( z_{j}^{2}\right) \cdot\sum _{k} w_{kj}^{3} \delta _{k}^{3} \quad\left( j=1,2,\ldots ,p\right)\quad\left( k=1,2,\ldots ,m\right) \end{align}$$

4 梯度下降算法

损失函数在隐藏层（层2）／输出层（层3）关于偏置的梯度 \begin{align} & \dfrac {\partial C_{\mathbf{x}}} {\partial b_{j}^{l}}=\dfrac {\partial C_{\mathbf{x}}} {\partial z_{j}^{l}}\cdot \dfrac {\partial z_{j}^{l}} {\partial b_{j}^{l}}=\delta _{j}^{l}\cdot \dfrac {\partial \left( \sum _{k}w_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}\right) } {\partial b_{j}^{l}}=\delta _{j}^{l}\quad\left( l=2,3\right)\end{align}

损失函数在隐藏层（层2）／输出层（层3）关于权值的梯度\begin{align} & \dfrac {\partial C_{\mathbf{x}}} {\partial w_{jk}^{l}}=\dfrac {\partial C_{\mathbf{x}}} {\partial z_{j}^{l}}\cdot \dfrac {\partial z_{j}^{l}} {\partial w_{jk}^{l}}=\delta _{j}^{l}\cdot \dfrac {\partial \left( \sum _{k}w_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}\right) } {\partial w_{jk}^{l}}=\delta _{j}^{l}\cdot a_{k}^{l-1}\quad\left( l=2,3\right)\end{align}

算法4.1 偏置与权值的梯度计算算法
输入：实例$\mathbf{x}=\left(x^{\left(1\right)},\cdots,x^{\left(n\right)}\right)^\top$
输出：损失函数在隐藏层（层2）／输出层（层3）关于偏置及权值的梯度$\left(\dfrac {\partial C_{\mathbf{x}}} {\partial b_{j}^{l}}\right)$和$\left(\dfrac {\partial C_{\mathbf{x}}} {\partial w_{jk}^{l}}\right)$

1. 为输入层设置对应的激活值$\left(a_j^1\right)_{j=1}^n=\left(x^{\left(j\right)}\right)_{j=1}^n$
2. 前向传播：对每个$l（l=2,3）$计算\begin{align} &a_{j}^{l}=\sigma \left( z_{j}^{l}\right) \\ & z_{j}^{l}= \sum _{k}w_{jk}^{l}\cdot a_{k}^{l-1}+b_{j}^{l}\end{align}
3. 计算输出层误差$\left(\delta _{j}^{3}\right)_{j=1}^m$；
4. 计算误差反向传播：隐藏层误差$\left(\delta _{j}^{2}\right)_{j=1}^p$；
5. 计算损失函数在隐藏层（层2）／输出层（层3）关于偏置及权值的梯度$\left(\dfrac {\partial C_{\mathbf{x}}} {\partial b_{j}^{l}}\right)$和$\left(\dfrac {\partial C_{\mathbf{x}}} {\partial w_{jk}^{l}}\right)$。

算法4.2 梯度下降算法：
输入：训练实例集合$T$
输出：偏置和权值的更新

1. 对每个训练实例$\mathbf{x}$：设置对应的输入激活$\mathbf{a}^{\mathbf{x},1}$，并执行以下步骤：
   • 前向传播：计算$\mathbf{z}^{\mathbf{x},l}=\mathbf{W}^l\mathbf{a}^{\mathbf{x},l-1}+\mathbf{b}^l$及$\mathbf{a}^{\mathbf{x},l}=\sigma\left(\mathbf{z}^{\mathbf{x},l}\right)$，其中$l=2,3,\cdots,L$。
   • 输出层误差：$\delta^{\mathbf{x},L}=\nabla_{\mathbf{a}^L}C_\mathbf{x}\odot\sigma’\left(\mathbf{z}^{\mathbf{x},L}\right)$
   • 误差反向传播：对每个$l=L-1,L-2,\cdots,2$，计算$\delta^{\mathbf{x},l}=\left(\left(\mathbf{W}^{l+1}\right)^\top\delta^{\mathbf{x},l+1}\right)\odot\sigma’\left(\mathbf{z}^{\mathbf{x},l}\right)$。
2. 梯度下降：对每个$l=L-1,L-2,\cdots,2$，根据$\mathbf{W}^l\leftarrow\mathbf{W}^l-\frac{\eta}{N}\sum_\mathbf{x}\delta^{\mathbf{x},l}\left(\mathbf{a}^{\mathbf{x},l-1}\right)^\top$和$\mathbf{b}^l\leftarrow\mathbf{b}^l-\frac{\eta}{N}\sum_\mathbf{x}\delta^{\mathbf{x},l}$更新权重和偏置。

5 前馈神经网络的改进

交叉熵损失函数

单个实例$\mathbf{x}$的损失函数$C_{\mathbf{x}}$为交叉熵损失函数

$$C_{\mathbf{x}}=-\sum _{j}\left[y_j\log\hat{y}_j+\left(1-y_j\right)\log\left(1-\hat{y}_j\right)\right]$$

正则化目标函数

$L_2$正则化的目标函数（结构风险）

$$C=\frac{1}{N}\left(\sum _{\mathbf{x}}C_{\mathbf{x}}+ \frac{\lambda}{2}\sum_w w^2\right)$$

权值初始化

权值初始化
设对$l$层有有$n_{in}^l$个输入神经元，使用均值为$0$，标准差为$\frac{1}{\sqrt{n_{in}^l}}$的高斯分布初始化$l$层的权值

$$w^l\sim N\left(0,\frac{1}{n_{in}^l}\right)$$

早停止

当验证集上的错误率不再下降时，就停止迭代。停止迭代需要根据实际任务上进行优化。