type
Post
status
Published
date
Aug 18, 2018
slug
2018.08.18 curveball 优化算法推导
summary
2018.08.18 curveball 优化算法推导
tags
优化算法
推荐
算法
category
算天算地系列
icon
password
前段时间,VGG 的一拨人搞了个叫 curveball 的优化算法,能够将二阶信息用起来,同时避免了之前的传统方法要么去近似 Hessian 矩阵的逆,要么通过 conjugate-gradient 的方法去得到 Hessian 矩阵的逆,这些传统方法既耗时又对噪声敏感。而 curveball 算法并不需要直接算 Hessian 矩阵和它的逆,每次只是去估计梯度与 Hessian 矩阵的乘积即可,所付出的代价仅仅是额外的两次正向传播。 由于这篇文章写的极为简略,很多过程都直接略掉了,今天我们就来推导这个算法,将作者略去的部分补上。
文章链接:点我
文章代码 github 链接:点我
本文推导内容的 pdf 链接:点我
推导细节
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fafdd680a-876e-4511-91a1-e1f4fcf8337b%2Fabf43465-d2bf-4efc-a213-1dd3ff835e8e%2FUntitled.png%3Ftable%3Dblock%26id%3D16c7d9be-e452-4c49-8e2c-33f1c748ea09%26spaceId%3Dafdd680a-876e-4511-91a1-e1f4fcf8337b%26expirationTimestamp%3D1725753600000%26signature%3D4MJ6G-9MlhMbDgH3S4BECDmE7C1obwQLnOYCDgTVPfs?table=block&id=16c7d9be-e452-4c49-8e2c-33f1c748ea09&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fafdd680a-876e-4511-91a1-e1f4fcf8337b%2F96438fe1-df7f-405a-bc92-66308dc72ab9%2FUntitled.png%3Ftable%3Dblock%26id%3D0d619abb-6341-44f6-bf7b-6e5cdc825729%26spaceId%3Dafdd680a-876e-4511-91a1-e1f4fcf8337b%26expirationTimestamp%3D1725753600000%26signature%3Do-vmvmSxMOyhYlbHmPZMOkG5-ZlV3xyJ6vZOk-fefjQ?table=block&id=0d619abb-6341-44f6-bf7b-6e5cdc825729&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fafdd680a-876e-4511-91a1-e1f4fcf8337b%2F41d21470-4de2-4268-aec0-3c4016413836%2FUntitled.png%3Ftable%3Dblock%26id%3D62ac32ea-ef6d-4e61-b7c9-19ed5b0c937f%26spaceId%3Dafdd680a-876e-4511-91a1-e1f4fcf8337b%26expirationTimestamp%3D1725753600000%26signature%3Dzpl7r1Q8yJwfPoRdwEJ_0_ZmGUNGX3Zno2uGSpiOqMI?table=block&id=62ac32ea-ef6d-4e61-b7c9-19ed5b0c937f&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fafdd680a-876e-4511-91a1-e1f4fcf8337b%2F657b4d62-721f-4aa5-8e10-11b21704e24a%2FUntitled.png%3Ftable%3Dblock%26id%3D1c3d31cd-dbe6-41b2-982b-030a133a334c%26spaceId%3Dafdd680a-876e-4511-91a1-e1f4fcf8337b%26expirationTimestamp%3D1725753600000%26signature%3DLGCUhgZI5yIKy7zAxmdKB_PsAFeV5_BSpIFEwBOvrKo?table=block&id=1c3d31cd-dbe6-41b2-982b-030a133a334c&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fafdd680a-876e-4511-91a1-e1f4fcf8337b%2F0a14c31b-46c2-4d69-a8ca-8a9d8d56052a%2FUntitled.png%3Ftable%3Dblock%26id%3D41870c7c-13af-4eb0-8e4f-13adaf85501d%26spaceId%3Dafdd680a-876e-4511-91a1-e1f4fcf8337b%26expirationTimestamp%3D1725753600000%26signature%3DHfstM3oaT1ykqmriy5yjnCUoFf3eTU3FSZzOT-FxoVM?table=block&id=41870c7c-13af-4eb0-8e4f-13adaf85501d&cache=v2)