Tensorflow error happens randomly while training
I am getting the following error
2023-10-15 12:28:59.400559: F tensorflow/core/common_runtime/device/device_event_mgr.cc:221] Unexpected Event status: 1
No idea how to fix it.
This happens randomly while training, sometimes an hour in, sometimes 20 min. However if I can get it to go for longer than 4/5 hours it seems to run fine.
My drivers are up to date, my cuda is up to date...
I am not the first person with this issue however I cannot find an actual solution
https://github.com/tensorflow/tensorflow/issues/46247
https://forum.faceswap.dev/viewtopic.php?t=2591