This should be fine, assuming your data quality is good.
You don't train a faceswap model to swap a scene. You train a model to swap 2 identities.
Once you have a trained model on 2 identities, you can use that model to swap the faces in any scene where the A face appears.