Talking Face - Data Processing Pipeline

This is a ready-to-use data processing pipeline for talking avatar videos.

vis-data.mp4

This pipeline include:	By default, set task=-1 to run all subtasks
task == 1	Extract audio from video, saved as audio.wav
task == 2	Extract audio features Supported feature type: [hubert]
task == 3	Extract original images from video
task == 4	① Extract Face Mesh (substitute to landmarks) ② Track Face poses and camera poses
task == 5	Parse Face by semantics
task == 6	Calculate background images from video, based on semantics
task == 7	Extract torso images, based on semantics
task == 8	Predict depth for each frame of image
task == 9	Split the dataset into train & eval sets pytorch dataset file provided.

Quick Start

① Install dependency

$ python3 -m pip install -r requirements.txt
$ sudo apt-get install ffmpeg

② Download necessary models:

Download Depth Checkpoints of DaGAN, put the folder depth_face_model_Voxceleb2_10w under folder data_process/face_depth/, like this:

└── data_process
    └── face_depth
        └── depth_face_model_Voxceleb2_10w
            ├── depth.pth
            └── encoder.pth
        ├── extract_face_depth.py
        └── ...
    ├── face_mesh
    └── ...

③ Run the processing:

python data_process/porcess_video.py  

By default, the model uses GPU if available. If you want it not to use GPU, set CUDA_VISIBLE_DEVICES="".

Pytorch Dataset for use after data processing

After the data processing, the data is in a structure of:

└── base_dir
    ├── video.mp4
    ├── audio.wav
    └── ori_imgs
        ├── i.jpg
        ├── ...
        ├── camera_poses.json
        ├── face_mesh.json
        ├── face_poses.json
        └── nodes_points.json
    └── parsing
        ├── i.npy
        └── ...
    ├── bc.jpg
    └── gt_imgs
        ├── i.jpg
        └── ...
    └── torso_imgs
        ├── i.png
        └── ...
    └── depth
        ├── i.npy
        └── ...
    ├── transforms_train.json
    └── transforms_val.json

TODO a dataset.py for late use

Implementation Details

Audio Processing

Reference: HuBert TODO Support more features

Face Landmarks and Mesh

Reference: Mediapipe Face Landmark [Doc] [Github]

Mediapipe works better than face-alignment.

Face Parsing

Reference: face-parsing.PyTorch

Face Depth

Reference: DaGAN

Reference

• AD-NeRF [Project Link]
• RAD-NeRF [Project Link]