Memory Issues With Realface Trainer

[tomward16]

Hi All,

I've decided to try and use different trainers with the same datasets for comparison. I've used Original, IAE and Dfaker. I've just tried using Realface and I'm coming across issues and not sure how best to resolve!

I have 2 x GTX 1070's with 8GB RAM a piece.
I have an Intel i7-8700 running at 4.3GHz when training and
16Gb RAM

When I try running the training even with batch size set to 8 I am getting an error of CUDA_OUT_OF_MEMORY. When looking at the GPU monitor it's only showing GPU 1 as having 6.7/8GB in use and GPU 2 is not being utilised at all. (I have stated I have 2 GPU's in the training options). I get the same error when selecting Optimizer Savings.

Am I doing something wrong? Is there a reason it is not utilising the total 16GB of VRAM?

Any suggestions would be much appreciated!

[torzdf]

You'll most likely need to lower the batch size for Real Face. It's quite a heavy model. See https://faceswap.dev/forum/app.php/faqpage#f3r5 for more info.

[bryanlyon]

Also, you don't have "16gb of vram" but 2 cards with 8gb each. They don't share vram and the model has to be duplicated on each if you use multigpu.

The problem with multigpu is that if you have 2 cards, one card is almost definitely handling your monitors so loses some vram for that. On Windows this can represent a huge portion of your vram. To account for this you need to make sure that whatever BS you're using fits on the more constrained card alone, then you can enable multigpu and double the BS for the pair.

[tomward16]

Thanks for that. I went and tried this on a friend's rig (who has more money than sense!) which has 4 x RTX Titans ( I did show him your comment about them! ) Each is 24(?)GB VRAM and I was able to run it at batch 64 on Realface trainer fine. Although there were still a lot of warnings/errors saying it couldn't allocate 3.94GB of RAM which confused me a bit. I was surprised however how little the training speed was affected though compared to my 2 x GTX 1070's. So at least I know not to spend a lot of money on more graphics cards, but rather invest in one very good one with a large amount of RAM.

[bryanlyon]

I went and tried this on a friend's rig (who has more money than sense!) which has 4 x RTX Titans ( I did show him your comment about them! )...I was surprised however how little the training speed was affected though compared to my 2 x GTX 1070's. So at least I know not to spend a lot of money on more graphics cards, but rather invest in one very good one with a large amount of RAM.

There is a reason I say that the RTX Titan is ridiculous. It's just not a good value for FaceSwap. I personally have it for other projects so can BARELY justify it. For faceswap an RTX 2060 super is really the sweet spot. 1070s are really ideal too.