[Guide] Extraction - A Workflow

Extracting faces for training and converting
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Post Reply
User avatar
torzdf
Posts: 80
Joined: Fri Jul 12, 2019 12:53 am
Answers: 13
Has thanked: 7 times
Been thanked: 10 times

[Guide] Extraction - A Workflow

Post by torzdf » Sat Jul 20, 2019 7:46 pm

NB: This guide was correct at time of writing, but things do change. I will try to keep it updated.

Contents

Introduction

A lot of people get overwhelmed when they start out Faceswapping, and many mistakes are made. Mistakes are good. It's how we learn, but it can sometimes help to have a bit of an understanding of the processes involved before diving in.

In this post I will detail a workflow for Extraction. I'm not saying that this is the best workflow, or what you should do, but it works for me, and will hopefully give you a good jump off point for creating your own workflow.

I will be using the GUI for this guide, but the premise is exactly the same for the cli (all the options present in the GUI are availablle in the cli).

Why Extract?
At the highest level, extraction consists of two phases: detection and alignment. There are several plugins for detecting and aligning. Their pros and cons are detailed in the tooltips (for the GUI) or the help text (for the cli), so I won't be covering specifics. However, detection is the process of finding faces within a frame whilst alignment is finding the "landmarks" within a face and orienting the faces consistently.

Extracting serves 2 main purposes:
  1. To generate a set of faces, and optionally an alignments file, for training the model
  2. To generate an alignments file for converting your final frames
There is technically a 3rd purpose which is when you are extracting for converting and want some of these faces for training as well. I will also cover this off.

Whilst the extracted faces are not technically required for convert (only the alignments file is), it is useful to have them so that we can clean up our dataset for the conversion process.

The Alignments File
The alignments file holds information about all the faces it finds in each frame, specifically where the face is, and where the 68 point landmarks are:
Image

The purpose of the alignments file is two-fold:
  1. Training: It tells the process how to build a face mask if using a mask, and is required if "Warp to Landmarks" is enabled
  2. Converting: It tells the convert process where the face is in the original frame to be replaced
So now we know why we need to extract faces, how do we nail down a good workflow?

Extracting
The first step, whatever your reason for creating a faceset is to actually extract the faces from the frames.
  • Input Dir: First up we need a source. This can be set to either a video file (first folder icon) or to a folder containing a series of images (second folder icon).
  • Output Dir: Next we need to tell the process where to save the extracted faces to. Click the folder icon next to select an output location.
    Image 1.png
    Image 1.png (8.78 KiB) Viewed 4199 times
  • I ignore the next few options, so you probably should too, but the explanation of each of them is below:
    • Alignments: This option is in case you want to specify a different name/location for the alignments file. It's a lot easier to leave this blank, and let the process save it out to the default location (which will be in the same location as the source frames/video).
    • Ref Threshold, Filter/nFilter: These options are for filtering in/out unwanted faces. I don't use it. I prefer to do this myself at the next stage, as I will always be more effective at filtering out faces than a machine is, and it doesn't take long. Filtering faces also severely slows down the extraction process.
    • Serializer: This is the format that the alignments file will be saved in. Just leave this default.
  • Detector, Aligner: Select the ones you want to use. At the time of writing the best detector is S3FD and the best aligner is FAN (FAN-AMD for AMD users):
    Image 2.png
    Image 2.png (2.64 KiB) Viewed 4199 times
  • Normalization: This processes the image fed into the extractor to better find faces in difficult lighting conditions. Different normalization methods work with different datasets, but I find "hist" to be the best all-rounder. This will slightly slow extraction, but it can lead to better alignments:
    Image 3.png
    Image 3.png (1.95 KiB) Viewed 4199 times
  • The next few options can also be ignored. Again, I will detail what they do so you can make your own call:
    • Rotate Images: This is entirely unnecessary with all current detectors except for maybe CV2-DNN. The current detectors are perfectly capable of finding faces at any orientation, so this will just slow down your process for little or no gain.
    • Blur Threshold. I don't find this option useful at all. It is meant to filter out blurry faces, however if you are extracting for convert you will want all faces to be extracted. If you are extracting for training then some blurry faces may be useful to have in your training set. I prefer to leave this off and remove faces I'm not happy with myself from my extracted set
    • Size: This is the size of the image holding the extracted face. Currently no models support above 256px so leave this at default
  • Min-size: I normally set this at a fairly low value, but above zero, to filter out anything that is clearly too small to be a face. It will, however, depend on the size of your input frames, and then the size of the faces within your frames:
    Image 4.png
    Image 4.png (977 Bytes) Viewed 4199 times
  • Extract Every N: This is the first option that will change depending on the purpose of this extraction.
    • If you are extracting for convert, or you are extracting for convert AND will be using some of the faces for training, then leave this on 1 (i.e. extract from every frame)
    • If you are extracting for training only, then set a value that seems sane. This will depend on the number of frames per second your input is. However, for a 25fps video, sane values are between about 12 - 25 (i.e. every half second to a second). Anything less than that and you will probably end up with too many similar faces. It is worth bearing in mind how many sources you are extracting from for your training set, and how many faces you want to have in your final training set, as well as how long your source is. These will all vary on a case by case basis.
    Image 5.png
    Image 5.png (954 Bytes) Viewed 4199 times
  • Save interval: I don't bother with, but it's up to you.
  • If you are an AMD user, make sure you check the AMD checkbox at the bottom:
    Image 7.png
    Image 7.png (2.78 KiB) Viewed 4199 times
You should end up with a screen similar to below:
Image 6.png
Image 6.png (24.83 KiB) Viewed 4199 times
Hit Extract and wait for your faces to output and your alignments file to be generated.

Sorting
Now that we have extracted our faces, we need to clean up the dataset and the alignments file. The Extractor does a good job of getting faces, but it isn't perfect. It will have got some false positives, it will have failed to align some faces and it will have also extracted people who we don't want to swap. Most likely if you go to your faces folder you will have large sections of the output which look something like this:
Image-11.jpg
Image-11.jpg (24.87 KiB) Viewed 4197 times
Cleaning that up doesn't look like it will be fun! Fortunately we can make this easier. The quickest and easiest way to clean our dataset is to sort the faces into a meaningful order and then delete all the faces we don't want. The best sort method, by some distance, is "sort by face".

NB: Sorting by face is RAM intensive. It has to do a LOT of calculations. I have tested successfully sorting 30k faces on 8GB of RAM, but if you have more faces than this, or if you are RAM limited, then you may want to split your dataset into smaller subsets.
  • Head over to the "Tools" tab in the GUI and then to the "Sort" sub-tab:
    Image 8.png
    Image 8.png (3.58 KiB) Viewed 4198 times
  • Input: enter the folder which contains your extracted faces from the previous step.
  • Output: Clear any entry from this option, so that your faces are sorted in place.
  • Final Process: Make sure this is set to "Rename"
  • Sort By: Set this to "Face"
  • Backend: Optionally set this to GPU. Only do this if you have a GPU and you are not currently using your GPU for any other Machine Learning tasks (e.g. Training). It will make sorting go much faster.
You do not need to touch any of the other options. The screen should look something like this:
Image 31.png
Image 31.png (20.48 KiB) Viewed 4198 times
Hit Sort.

The process will start reading through the faces building up an identity for each. It will then cluster the faces together by similarity. The actual clustering process can take a long time as it has to compute a lot of data. Unfortunately there is no visual feedback as to it's progress, so please be patient.

Once complete you should find 99% of the faces sorted together:
Image-24.jpg
Image-24.jpg (24.68 KiB) Viewed 4197 times
And all the junk sorted together as well:
Image-25.jpg
Image-25.jpg (24.9 KiB) Viewed 4197 times
Now all you have to do is scroll through the faces folder and delete those faces out that you do not wish to keep.

Clean the Alignments File
Now we have removed all of the faces we don't want and just have a set that we do, we need to clean the alignments file. Why? Because all the information about the unwanted faces is still in the file, and this is likely to cause us problems in future. Cleaning the alignments file with the integrated tool also has the added advantage of renaming our faces back to their original filename, so it's a win-win.
  • Navigate to the "Tools" tab then the "Alignments" sub-tab, and select the "remove-faces" job:
    Image 12.png
    Image 12.png (3.66 KiB) Viewed 4198 times
    Image 14.png
    Image 14.png (6.85 KiB) Viewed 4198 times
  • Alignments File: select the alignments file that was generated during the extraction process It will be located in the same folder as the video you extracted from, or within the folder of the images you extracted from.
  • Faces Folder: select the faces folder that you outputted faces into as part of the extraction process (the same folder as you used for sort).
    Image 15.png
    Image 15.png (3.59 KiB) Viewed 4198 times
  • Leave all the other options at default
Your options should be similar to below:
Image 16.png
Image 16.png (24.14 KiB) Viewed 4198 times
Hit Alignments and let the process complete.

Once done your faces will have been renamed back to their default names, and all of the unwanted faces will have been removed from your alignments file.

The process will have backed up your old alignments file and put it next to your newly created file in it's original location. It will have the same name as your cleansed alignments file but with "backup_<timestamp>" appended on to the end of it. If you are happy that your new alignments file is correct, you can safely remove this backup file.

At this point, if you are extracting for convert (or the set is going to be used for convert and training) then you can entirely delete your faces folder. None of these faces are required any more. If you ever need to re-generate your face set, then this can be done with the Alignments Tool (extract job).

Manually Fix Alignments
Ok, we've extracted our faces, we've cleaned out all the trash, surely we're done now? Well hold on there fella. Sure, you could move on now, but do you want an ok swap or do you want a great swap?

Manual is useful for the following tasks:
  • Removing any left over "multiple faces" in a frame
  • Adding any alignments to those frames that are missing them
  • Fixing misaligned frames
Depending on what we are extracting our dataset for will dictate what we want to do here. If we are purely extracting for a training set, we could feasibly skip this step altogether, although it is a good idea to review the existing alignments to make sure any masks get built correctly.

If extracting for convert, then at the absolute minimum we want to fix any multiple faces in frames, and any frames missing alignments. This alone will improve the final swap. Depending on how thorough you want to be, you can then fix any poor alignments

I won't go into full details on how to use the Manual Tool. That is a guide in itself, but it is quite intuitive, with instructions written in a pop-up window.

To launch the Manual tool:
  • Navigate to the "Tools" tab then the "Alignments" sub-tab and select the "manual" job:
    Image 12.png
    Image 12.png (3.66 KiB) Viewed 4198 times
    Image 17.png
    Image 17.png (6.84 KiB) Viewed 4198 times
  • Alignments File: select the alignments file that was generated after cleaning in the last step. It will be located in the same folder as the video you extracted from, or within the folder of the images you extracted from.
  • Frames Folder: select the video or the frames folder that you used as an input for the extraction process
    Image 19.png
    Image 19.png (4.59 KiB) Viewed 4198 times
  • Leave all the other options at default
Your options should be similar to below:
Image 20.png
Image 20.png (24.15 KiB) Viewed 4198 times
Hit the Alignments button to launch the tool.

Manually Fixing for Converting

NB: Press the "m" key to switch the mode from view and edit

For fixing for convert we want to do the following:
  • Multi-Faces
    • Press "s" until the skip-mode is set to "multi-faces"
    • Press "x" to skip to the next frame with multiple faces in the frame
    • Click on any faces which you do not want and press "d" to delete them
    • Continue until you have reached the last frame
    • Press "Space" to save out your amended alignments file
  • Frames without faces
    • Press "{" to return to the first frame
    • Press "S" until the skip-mode is set to "no-faces"
    • Press "x" to skip to the next frame which has no faces in the frame
    • If you hit a frame where no face was detected but should have been, you can either click on the face and adjust the bounding box with the mouse, or you can copy the previous/next frame's alignments with the "c" or "v" keys and edit those.
    • Repeat until your reach the last frame
    • Press "Space" to save out your amended alignments file
  • Fixing alignments
    • Press "{" to return to the first frame
    • Press "s" until the skip-mode is set to "has-faces"
    • Press "x" to skip to the next frame which has faces
    • If the alignments look incorrect, adjust the bounding box until it's fixed
    • Repeat until you reach the last frame
    • Press "Space" to save out your amended alignments file
The process will have created a backup old alignments file each time you saved your alignments into the same folder as the video you extracted from, or within the folder of the images you extracted from. It will have the same name as your cleansed alignments file but with "backup_<timestamp>" appended on to the end of it. If you are happy that your new alignments file is correct, you can safely remove these backup files.

Manually Fixing for a Training Set

NB: Press the "m" key to switch the mode from view and edit

For fixing for training we want to do the following:
  • Fixing alignments
    • Press "s" until the skip-mode is set to "has-faces"
    • Press "x" to skip to the next frame which has faces
    • If the alignments look incorrect, adjust the bounding box until it's fixed
    • Repeat until you reach the last frame
    • Press "Space" to save out your amended alignments file
The process will have created a backup old alignments file each time you saved your alignments into the same folder as the video you extracted from, or within the folder of the images you extracted from. It will have the same name as your cleansed alignments file but with "backup_<timestamp>" appended on to the end of it. If you are happy that your new alignments file is correct, you can safely remove these backup files.

Once the alignments are fixed, you will need to regenerate your faceset. This is because the face image will have changed due to moving the landmarks, so the training process will not be able to find the faces you amended.

Firstly delete your output faces folder. You don't want this around any more.
  • Navigate to the "Tools" tab then the "Alignments" sub-tab and select the "extract" job:
    Image 21.png
    Image 21.png (10.32 KiB) Viewed 4198 times
  • Alignments File: select the alignments file that was generated when after cleaning in the last step. It will be located in the same folder as the video you extracted from, or within the folder of the images you extracted from.
  • Faces Folder: select an empty folder where you want the outputted faces to go.
  • Frames Folder: select the video or the frames folder that you used as an input for the extraction process
    Image 22.png
    Image 22.png (5.14 KiB) Viewed 4198 times
  • Leave all the other options at default
Your options should be similar to below:
Image 23.png
Image 23.png (25.18 KiB) Viewed 4198 times
Hit the Alignments button to re-extract the faces.
If you intend to train with a mask or use Warp to Landmarks, then copy the alignments file from your source frames location into your newly created faces folder.

Extracting a training set from an alignments file
Now you've cleaned up your alignments file, you may want to pull some of these faces out to use for a traning set. This is a simple task
  • Navigate to the "Tools" tab then the "Alignments" sub-tab and select the "extract" job:
    Image 21.png
    Image 21.png (10.32 KiB) Viewed 4198 times
  • Alignments File: select the alignments file that was generated after cleaning in the last step. It will be located in the same folder as the video you extracted from, or within the folder of the images you extracted from.
  • Faces Folder: select an empty folder where you want the outputted faces to go.
  • Frames Folder: select the video or the frames folder that you used as an input for the extraction process
    Image 22.png
    Image 22.png (5.14 KiB) Viewed 4198 times
  • Extract Every N: This will depend on the number of frames per second your input is. However, for a 25fps video, sane values are between about 12 - 25 (i.e. every half second to a second). Anything less than that and you will probably end up with too many similar faces. It is worth bearing in mind how many sources you are extracting from for your training set, and how many faces you want to have in your final training set, as well as how long your source is. These will all vary on a case by case basis.
    Image 29.png
    Image 29.png (1.08 KiB) Viewed 4198 times
  • Leave all the other options at default
Your options should be similar to below:
Image 30.png
Image 30.png (25.36 KiB) Viewed 4198 times
Hit the Alignments button to extract a training set from your cleansed alignments file to your selected folder.
If you intend to train with a mask or use Warp to Landmarks, then copy the alignments file from your source frames location into your newly created faces folder.

Merging facesets for training
So you've got a load of faces ready for training, and you've got a load of alignments files for the frames that the faces come from. Can these be combined into one source for training? Sure they can!
  • Firstly copy all of the faces you want to train with from their respective folders into one folder (we'll call it "training" for reference).
  • Copy all of the alignments files that relate to the faces you've just selected from their source frames location into the new "training" folder.
  • Navigate to the "Tools" tab then the "Alignments" sub-tab and select the "merge" job:
    Image 12.png
    Image 12.png (3.66 KiB) Viewed 4198 times
    Image 26.png
    Image 26.png (6.84 KiB) Viewed 4198 times
  • Alignments File: Navigate to your "training" folder, hold shift and highlight all of the alignments files you just copied to select them.
  • Faces Folder: select the "training" folder that you placed all your faces in
    Image 28.png
    Image 28.png (3.5 KiB) Viewed 4198 times
  • Leave all the other options at default
Your options should be similar to below:
Image 27.png
Image 27.png (24.1 KiB) Viewed 4198 times
Hit the Alignments button to merge the alignments files.

When the process has completed:
  • Delete all of the alignments files that you previously copied into the "training" folder from the "training" folder.
  • Rename the generated alignment file inside your "training" folder to "alignments.json"
Your training set is ready
My word is final

User avatar
JiangshanLee
Posts: 3
Joined: Sun Jul 28, 2019 6:08 am

Re: [Guide] Extraction - A Workflow

Post by JiangshanLee » Sun Jul 28, 2019 8:17 am

can not see photo in your direction

User avatar
torzdf
Posts: 80
Joined: Fri Jul 12, 2019 12:53 am
Answers: 13
Has thanked: 7 times
Been thanked: 10 times

Re: [Guide] Extraction - A Workflow

Post by torzdf » Sun Jul 28, 2019 11:48 am

They are hosted on imgur. Is that blocked where you are?

I have updated the hosting, so hopefully it works now.
My word is final

User avatar
JiangshanLee
Posts: 3
Joined: Sun Jul 28, 2019 6:08 am

Re: [Guide] Extraction - A Workflow

Post by JiangshanLee » Sat Aug 03, 2019 4:52 am

It's works, thank you very much

User avatar
JiangshanLee
Posts: 3
Joined: Sun Jul 28, 2019 6:08 am

Re: [Guide] Extraction - A Workflow

Post by JiangshanLee » Sat Aug 03, 2019 5:19 am

Thank you very much for your great tutorial. I have not been able to run successfully because of my poor English. Can you post a video tutorial? thank you very much.

User avatar
torzdf
Posts: 80
Joined: Fri Jul 12, 2019 12:53 am
Answers: 13
Has thanked: 7 times
Been thanked: 10 times

Re: [Guide] Extraction - A Workflow

Post by torzdf » Fri Aug 09, 2019 12:30 am

I'm unlikely to post a video, sorry. I just don't have the time to make it at the moment.
My word is final

User avatar
Vaca
Posts: 1
Joined: Thu Jul 18, 2019 10:42 pm

Re: [Guide] Extraction - A Workflow

Post by Vaca » Sat Aug 24, 2019 4:54 pm

When I go to the sort section my Backend shows options for Cpu and Opencl, but no GPU. Is Opencl the same as GPU or do I not have something setup correctly?

User avatar
kilroythethird
Posts: 2
Joined: Fri Jul 12, 2019 11:35 pm
Has thanked: 1 time

Re: [Guide] Extraction - A Workflow

Post by kilroythethird » Sat Aug 24, 2019 7:35 pm

@Vaca: If you see opencl and CPU options in sort you are using a old faceswap version. Update via "tools" -> "check for updates. Should also be a good bit faster after the update,
that amd guy

Post Reply