Hello,
I am trying to extract faces from images. My input folder contains 25,000 images. Faceswap processes around 3,000 images and then the OOM killer kills the Faceswap python process because it uses too much RAM (Not VRAM). I read in "Hardware Requirements" topic that Faceswap does not need much RAM because it uses VRAM for calculations. I am trying to understand why the extraction processes is using 32GB of RAM. Can you guys please help me?
My PC specs are:
OS: Ubuntu 22.04 LTS
CPU: Intel Xeon Silver 4310 CPU @ 2.10GHz (24 cores)
RAM: 32GB
VRAM: 6GB (NVIDIA GTX 1060)
I checked the syslog and it logged following messages after OOM killer killed the Faceswap process:
Jan 12 00:49:21 Ubuntu kernel: [ 9904.476519] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=python,pid=2017,uid=1000
Jan 12 00:49:21 Ubuntu kernel: [ 9904.476757] Out of memory: Killed process 2017 (python) total-vm:62482864kB, anon-rss:30951064kB, file-rss:66384kB, shmem-rss:23332kB, UID:1000 pgtables:66812kB oom_score_adj:0
I wrote a bash script that saves the RAM and VRAM usage info in a text file. It logged the following usage information right before the process was killed:
RAM:
total used free shared buff/cache available
Mem: 31713 31334 224 39 154 19
Swap: 511 511 0
VRAM:
Code: Select all
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11 Driver Version: 525.60.11 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:51:00.0 On | N/A |
| 0% 49C P5 9W / 180W | 5814MiB / 6144MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1288 G /usr/lib/xorg/Xorg 95MiB |
| 0 N/A N/A 4401 C .../envs/faceswap/bin/python 5712MiB |
| 0 N/A N/A 3168537 G flurry 2MiB |
+-----------------------------------------------------------------------------+
As you can above, Faceswap was using all available RAM when it was killed (Only 19MB of 32GB was free).
Following is the Faceswap log:
Code: Select all
01/12/2023 22:10:14 MainProcess MainThread logger log_setup INFO Log level set to: INFO
01/12/2023 22:10:16 MainProcess MainThread plugin_loader _import INFO Loading Detect from S3Fd plugin...
01/12/2023 22:10:16 MainProcess MainThread plugin_loader _import INFO Loading Align from Fan plugin...
01/12/2023 22:10:16 MainProcess MainThread plugin_loader _import INFO Loading Recognition from Vgg_Face2 plugin...
01/12/2023 22:10:16 MainProcess MainThread plugin_loader _import INFO Loading Mask from Components plugin...
01/12/2023 22:10:16 MainProcess MainThread plugin_loader _import INFO Loading Mask from Extended plugin...
01/12/2023 22:10:16 MainProcess MainThread plugin_loader _import INFO Loading Mask from Bisenet_Fp plugin...
01/12/2023 22:10:16 MainProcess MainThread pipeline _set_plugin_batchsize INFO Reset batch sizes due to available VRAM: Detect: 1, Align: 1, Recognition: 1, Mask: 1
01/12/2023 22:10:16 MainProcess MainThread extract process INFO Starting, this may take a while...
01/12/2023 22:10:16 MainProcess MainThread extract __init__ INFO Output Directory: /media/guest/fee1c673-3f7f-4529-a986-ade8019f64b4/Extracted/B/dir_001
01/12/2023 22:10:16 MainProcess MainThread _base initialize INFO Initializing S3FD (Detect)...
01/12/2023 22:10:19 MainProcess MainThread _base initialize INFO Initialized S3FD (Detect) with batchsize of 1
01/12/2023 22:10:19 MainProcess MainThread _base initialize INFO Initializing FAN (Align)...
01/12/2023 22:10:32 MainProcess MainThread _base initialize INFO Initialized FAN (Align) with batchsize of 1
01/12/2023 22:10:32 MainProcess MainThread _base initialize INFO Initializing VGGFace2 (Recognition)...
01/12/2023 22:10:39 MainProcess MainThread _base initialize INFO Initialized VGGFace2 (Recognition) with batchsize of 1
01/12/2023 22:10:39 MainProcess MainThread _base initialize INFO Initializing Components (Mask)...
01/12/2023 22:10:39 MainProcess MainThread _base initialize INFO Initialized Components (Mask) with batchsize of 1
01/12/2023 22:10:39 MainProcess MainThread _base initialize INFO Initializing Extended (Mask)...
01/12/2023 22:10:39 MainProcess MainThread _base initialize INFO Initialized Extended (Mask) with batchsize of 1
01/12/2023 22:10:39 MainProcess MainThread _base initialize INFO Initializing BiSeNet - Face Parsing (Mask)...
01/12/2023 22:10:43 MainProcess MainThread _base initialize INFO Initialized BiSeNet - Face Parsing (Mask) with batchsize of 1
Following are the screenshots of global settings:
Following are the screenshots of extract settings:
Thanks