[WIP] First test auto adaptive - #16
Conversation
Keep positive threshold
Add params recommendation to exp3_params
| saved_var[tensor_name].add_(new_grad) | ||
| model.zero_grad() | ||
|
|
||
| gamma += (b/num_microbatches-gamma)/2 |
There was a problem hiding this comment.
I’ve been thinking about this more. Intuitively, we want to favor a clipping threshold at the beginning of training that has low bias, but higher variance. In this case, it’s ok that we add a higher clipping threshold in exchange for more noise because the low bias in the estimate helps the model converge towards a good initial solution. However, later on, its important that there is little noise added to the parameter estimate (low variance) so that the model can learn finer grained details.
See discussion by Andrew et. al, for example
There was a problem hiding this comment.
One idea is to taper the gamma value over training (akin to a learning rate scheduler). It would be cool if we could somehow estimate the bias / variance trade off during training. One way is to look at the delta between training and validation loss, which is normally taken as a measure of overfitting. I'll try the gamma tapering approach tomorrow morning
2 participants
No description provided.