I have now improved my system and I extended folders to 49 (7x7). It's necessary because folders 1-14 and 36-49 have usually so few images so there must be more folders in the middle where most of the images is grouped.
I have made script that groups all extracted images to those 49 directories. Then it checks every folder how many images it have and sorts them by image count to 'keep', 'mid' and 'big' folders. In keep folder is all directories less than 500 images, in mid folder all folders with 500-5000 images and in big folder all with over 5000 images.
After that it starts to reduce imagecount of big folder. Let's say there's folders 25 and 26 in big directory. All the big directories is grouped by face and after that all face folders is sorted by face-cnn (In different tasks because it's faster that way). Then it takes every 1 to every 15 pictures depending of current folder image count to keep. After all folder 25 face directories is done, all those images is copied to mid\25 directory. And then same thing to folder 26 and so on. After reducing big directories those have about 2000-6000 images left.
After all big directories is reduced and all the reduced big directories is moved to mid directory, it starts to reduce mid directories. First all the mid directories is sorted by face-cnn and then it takes every 1 to every 15 pictures depending of current folder image count to keep. Then all reduced mid directories is moved to keep directory and now there's all what we want to keep. Usually there's some of the folders 15-35 with each 1-500 files and after quicks check every folder is ok folders is ready to move where we wanna save all different b-face image folders.
This whole process is automatic and all I have to do, is make quick cleanup of extracted faces before using this script to remove all pictures I don't wanna be sorted. For example hand obstructed faces, blurry faces and so on...
I also found a good way to sort blurry images. I found little python script what uses OpenCV to convert images to grayscale and then uses laplacian to determine if image is blurry or not with adjustable treshold. Only problem was that some of the images was determined blurry because of blurry backround even when actual face is sharp. So I made different script that uses ImageMagick to crop middle part of every image where the face is and save it to different folder. Then that blur script sorts those cropped images to blurry and not blurry and after that the script check what files is found in not blurry folder and copies all equivalent original files to keep.
I use that blur detection script only with B-side faces before using that sorting script. Sorting script works also with A-side faces but there I leave blurry faces because that script reduces count of them also and leaves a some of them what probably is necessary.
Third script is to compare A-side and B-side image folders. If using multiple different videos to B-side set, all B-side folders (1-49) must be sorted by face-cnn before using this compare script. This third script checks what folders is used with A-side and then it takes necessary amount of B-side images of those needed folders. And because all the directories is sorted by face-cnn it takes every possible face expression available in that current directory.
I hope you understard because my english is far from perfect and this is little bit hard to explain. But this is the short version of my system and so far it have been working good when doing basic faceswaps. Professionals do their own way