Page 1 of 1
Loss briefly spiked, then recovered. What happened?
Posted: Tue Jul 16, 2019 10:45 pm
by Surrogator
I just started training for the first time. It was running for about 15 minutes, and I had stepped away from the computer for a moment. When I came back I saw this.
- collapse_and_recover.png (130.98 KiB) Viewed 3467 times
I think this means that my model collapsed, but it also recovered on its own. I'm pretty sure I would've terminated training if I was present during this 200 iterations.
So, what happened? Did my model collapse and recover? Is this a bad omen?
Re: Loss briefly spiked, then recovered. What happened?
Posted: Tue Jul 16, 2019 10:46 pm
by bryanlyon
This is usually a sign of an overclocked GPU. The other option is exploding gradients that recovered -- which is far more rare. The fact it was uniform for A and B tells me the GPU is more likely the culprit.
Re: Loss briefly spiked, then recovered. What happened?
Posted: Tue Jul 16, 2019 10:52 pm
by torzdf
This has happened to me before. I don't like it when I see it, but it should be ok.
To be honest, if it's early in the train I tend to kill my model and start again, because I'm super paranoid.
Peaks on the graph.
Posted: Wed Mar 04, 2020 1:49 pm
by jhsy1209021
I noticed a interesting phenomenon. When I was training the model, there were some peaks appeared on the graph. Although it just happened several times and had no impact on the model, but its is still wierd. Have anyone known or encountered the same situation before?
Train settings:
Model : Dlight
Mask : Unet-Dfl
Coverage : 62.5%
Conv Aware Init : true
Re: Peaks on the graph.
Posted: Wed Mar 04, 2020 7:37 pm
by bryanlyon
Perfectly normal and nothing to worry about.