Page 1 of 1
Masking for training?
Posted: Sat Mar 30, 2024 4:13 am
by wulfrik
Apologies if this is a stupid set of questions or posted in the wrong place. I'm new and still learning.
So I extracted frames from various sources for both A and B. Some video sources and some from random photography. However, during the extraction phase I checked "custom" masker for both. I thought it didn't matter for the training, and all I needed was the faces aligned. I've run the resulting output images through 100k training iterations and the results are looking sharp! However, I noticed the masks in the preview thumbs are often...bad. For instance, in many frames the source is wearing a headband. It doesn't necessarily obscure the face, but the masker left some of that headband in and it's leaked into the model. Other masks are just not tightly constrained to the face. But the results look pretty good!
Is this an issue, or is it working as intended and the masking isn't a big deal?
Will the model learn OK regardless, or will the masking issues lead to downstream problems?
If the faulty masks are a problem, what's the remedy? For instance, how do I fix the headband issue for, say 1500 extracted images? Do I have to re-extract from the source and address each on manually? What tool do I use to see the masks before training?
Thanks for any input or feedback.
Re: Masking for training?
Posted: Sun Mar 31, 2024 11:28 am
by torzdf
wulfrik wrote: ↑Sat Mar 30, 2024 4:13 am
However, during the extraction phase I checked "custom" masker for both.
Custom masker doesn't really do anything. It just generates an empty mask for editing elsewhere. It doesn't hurt to have it, but it won't be helping you any.
wulfrik wrote: ↑Sat Mar 30, 2024 4:13 am
I thought it didn't matter for the training, and all I needed was the faces aligned. I've run the resulting output images through 100k training iterations and the results are looking sharp! However, I noticed the masks in the preview thumbs are often...bad. For instance, in many frames the source is wearing a headband. It doesn't necessarily obscure the face, but the masker left some of that headband in and it's leaked into the model. Other masks are just not tightly constrained to the face. But the results look pretty good!
Most likely you are using the default 'extended' mask. This is based entirely on the face landmark points.... Its fine, but it doesn't handle obstructions (as you've seen), it doesn't cover the forehead, and, if the alignments are wrong, it won't fit the face properly (as you have also seen).
wulfrik wrote: ↑Sat Mar 30, 2024 4:13 am
Is this an issue, or is it working as intended and the masking isn't a big deal?
It is working as intended, but it isn't working the best it could. You should use the BiSeNet-FP mask. You can generate additional masks for facesets/alignment files using the mask tool (no need to completely re-extract again)
wulfrik wrote: ↑Sat Mar 30, 2024 4:13 am
Will the model learn OK regardless, or will the masking issues lead to downstream problems?
It will be 'ok'. It won't be great. Unfortunately the masker is locked to the model, so you will need to restart training once the new mask has been generated, and ensure that you have selected the correct mask in training settings.
All of this is covered in the guides, so make sure you are following them:
Re: Masking for training?
Posted: Sun Mar 31, 2024 6:20 pm
by wulfrik
Thank you for the reply. I appreciate the attention and help.
So I re-ran the masker on the extracted files, in the Tools section (choosing BiSeNet-FP). It doesn't seem to have done a lot better. For instance, it's still picking up a lot of the headband fabric. I guess what I feel like I'm missing is where I can see these masks before I start training? And moreover, is there a guide for how to manually edit these masks? I'd love a grid of thumbnails where I can quickly scan for weird masks and fix them. I can't tell if there's something like that within the application or if it's another process in, say, Photoshop (or similar).
Re: Masking for training?
Posted: Sun Mar 31, 2024 6:22 pm
by torzdf
Most likely you are either continuing training the same model, in which case the mask you originally selected will be used (default is Extended), or you have started a new model but have neglected to select the BiSeNet-FP mask in training settings so the default is used (Extended).
A screengrab would be able to clear that up fairly quickly.
You can view/edit masks in the Manual tool.
BiSeNet will handle most obstructions (including hairbands) just fine:
Re: Masking for training?
Posted: Mon Apr 01, 2024 2:48 am
by wulfrik
or you have started a new model but have neglected to select the BiSeNet-FP mask in training settings so the default is used (Extended).
Oh wow. This was it!
Thank you so much for suggesting that. I didn't see that in the docs and I don't know if I would have thought to look in Settings -> Train -> Global -> Loss -> Mask Type to match the masker to the trainer.
So to reiterate and better describe my issue for the sake of posterity and any wayward travelers with a similar issue:
Thank you again for the expertise and timely assistance.