图文介绍部分转自公众号：计算机视觉Daily

GitHub标星3.6k | 给AI一张高清照片，分分钟还你细节满满的3D人体模型！

传送门

GitHub地址：
https://github.com/facebookresearch/pifuhd

Demo地址：
https://colab.research.google.com/drive/11z58bl3meSzo6kFqkahMa35G5jmh2Wgt?sp=sharing#scrollTo=afwL_-ROCmDf

图文介绍

手动对人体进行3D建模并非易事。

但现在，只给AI一张高清照片，它还真就能分分钟搞定这件事。

甚至还挺高清，衣服褶皱、面部表情，细节一点不少。

这项新研究来自南加州大学和Facebook，中选CVPR 2020。

并且已经在GitHub上开源，标星3.6k，还在一天内就涨了207颗星，登上GitHub热榜。

一起来看看，这究竟是如何实现的。

多级像素对齐隐式函数

这只AI名叫PIFuHD，其基础框架是ICCV 2019上已经登场的像素对齐隐式函数PIFu。不过，PIFu以分辨率为512×512的图像作为输入，输出的3D模型分辨率不高

为了得到高分辨率的输出，在这项研究中，研究人员在PIFu的基础之上，额外叠加了一个像素对齐的预测模块。

如图所示，顶部粗层次像素对齐预测器捕捉全局的3D结构。高分辨率的细节则由下面的Fine模块添加。

具体而言，fine模块将1024×1024的图像作为输入，并将其编码成高分辨率的图像特征（512×512）。

此后，高分辨率特征嵌入和第一个模块中得到的3D嵌入被结合起来，用以预测占位概率场。

为了进一步提高重建的质量和保真度，该方法还会在图像空间中预测正反两面的法线图，并将其作为额外的输入反馈给网络。

Demo可玩

论文代码已经开源，并且，研究团队还在Colab上提供了在线试玩。

输入一张你自己的照片，几分钟之内就能收获一个数字3D的你。

真·3D建模师福音。

结合可以让3D模型动起来的Mixamo食用，网友们都玩嗨了。

运行代码

Colab链接：
https://colab.research.google.com/drive/11z58bl3meSzo6kFqkahMa35G5jmh2Wgt?sp=sharing#scrollTo=afwL_-ROCmDf

进入Colab，将其拷贝至自己的云端硬盘里。

进入自己的云端硬盘，找到刚才拷贝过来的副本，点击进去。

因为Colab的Pytorch版本不对，需要将pytorch版本更新为1.6.0

先运行代码：

!pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html

依次运行代码框：

Clone PIFuHD repository

!git clone https://github.com/facebookresearch/pifuhd

Configure input data

cd /content/pifuhd/sample_images


from google.colab import files

filename = list(files.upload().keys())[0]


这里需要选择本地图片，限制512*512像素的图片才可以，如果尺寸不对，就先裁剪或resize一下。

import os

try:
  image_path = '/content/pifuhd/sample_images/%s' % filename
except:
  image_path = '/content/pifuhd/sample_images/test.png' # example image
image_dir = os.path.dirname(image_path)
file_name = os.path.splitext(os.path.basename(image_path))[0]

# output pathes
obj_path = '/content/pifuhd/results/pifuhd_final/recon/result_%s_256.obj' % file_name
out_img_path = '/content/pifuhd/results/pifuhd_final/recon/result_%s_256.png' % file_name
video_path = '/content/pifuhd/results/pifuhd_final/recon/result_%s_256.mp4' % file_name
video_display_path = '/content/pifuhd/results/pifuhd_final/result_%s_256_display.mp4' % file_name


cd /content

Preprocess (for cropping image)

!git clone https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch.git


cd /content/lightweight-human-pose-estimation.pytorch/


!wget https://download.01.org/opencv/openvino_training_extensions/models/human_pose_estimation/checkpoint_iter_370000.pth


import torch
import cv2
import numpy as np
from models.with_mobilenet import PoseEstimationWithMobileNet
from modules.keypoints import extract_keypoints, group_keypoints
from modules.load_state import load_state
from modules.pose import Pose, track_poses
import demo

def get_rect(net, images, height_size):
    net = net.eval()

    stride = 8
    upsample_ratio = 4
    num_keypoints = Pose.num_kpts
    previous_poses = []
    delay = 33
    for image in images:
        rect_path = image.replace('.%s' % (image.split('.')[-1]), '_rect.txt')
        img = cv2.imread(image, cv2.IMREAD_COLOR)
        orig_img = img.copy()
        orig_img = img.copy()
        heatmaps, pafs, scale, pad = demo.infer_fast(net, img, height_size, stride, upsample_ratio, cpu=False)

        total_keypoints_num = 0
        all_keypoints_by_type = []
        for kpt_idx in range(num_keypoints):  # 19th for bg
            total_keypoints_num += extract_keypoints(heatmaps[:, :, kpt_idx], all_keypoints_by_type, total_keypoints_num)

        pose_entries, all_keypoints = group_keypoints(all_keypoints_by_type, pafs, demo=True)
        for kpt_id in range(all_keypoints.shape[0]):
            all_keypoints[kpt_id, 0] = (all_keypoints[kpt_id, 0] * stride / upsample_ratio - pad[1]) / scale
            all_keypoints[kpt_id, 1] = (all_keypoints[kpt_id, 1] * stride / upsample_ratio - pad[0]) / scale
        current_poses = []

        rects = []
        for n in range(len(pose_entries)):
            if len(pose_entries[n]) == 0:
                continue
            pose_keypoints = np.ones((num_keypoints, 2), dtype=np.int32) * -1
            valid_keypoints = []
            for kpt_id in range(num_keypoints):
                if pose_entries[n][kpt_id] != -1.0:  # keypoint was found
                    pose_keypoints[kpt_id, 0] = int(all_keypoints[int(pose_entries[n][kpt_id]), 0])
                    pose_keypoints[kpt_id, 1] = int(all_keypoints[int(pose_entries[n][kpt_id]), 1])
                    valid_keypoints.append([pose_keypoints[kpt_id, 0], pose_keypoints[kpt_id, 1]])
            valid_keypoints = np.array(valid_keypoints)
            
            if pose_entries[n][10] != -1.0 or pose_entries[n][13] != -1.0:
              pmin = valid_keypoints.min(0)
              pmax = valid_keypoints.max(0)

              center = (0.5 * (pmax[:2] + pmin[:2])).astype(np.int)
              radius = int(0.65 * max(pmax[0]-pmin[0], pmax[1]-pmin[1]))
            elif pose_entries[n][10] == -1.0 and pose_entries[n][13] == -1.0 and pose_entries[n][8] != -1.0 and pose_entries[n][11] != -1.0:
              # if leg is missing, use pelvis to get cropping
              center = (0.5 * (pose_keypoints[8] + pose_keypoints[11])).astype(np.int)
              radius = int(1.45*np.sqrt(((center[None,:] - valid_keypoints)**2).sum(1)).max(0))
              center[1] += int(0.05*radius)
            else:
              center = np.array([img.shape[1]//2,img.shape[0]//2])
              radius = max(img.shape[1]//2,img.shape[0]//2)

            x1 = center[0] - radius
            y1 = center[1] - radius

            rects.append([x1, y1, 2*radius, 2*radius])

        np.savetxt(rect_path, np.array(rects), fmt='%d')
 

net = PoseEstimationWithMobileNet()
checkpoint = torch.load('checkpoint_iter_370000.pth', map_location='cpu')
load_state(net, checkpoint)

get_rect(net.cuda(), [image_path], 512)

Preprocess (for cropping image)

cd /content/pifuhd/


!sh ./scripts/download_trained_model.sh

Run PIFuHD!

# Warning: all images with the corresponding rectangle files under -i will be processed. 
!python -m apps.simple_test -r 256 --use_rect -i $image_dir

# seems that 256 is the maximum resolution that can fit into Google Colab. 
# If you want to reconstruct a higher-resolution mesh, please try with your own machine. Render the result
# This command takes a few minutes
!pip install pytorch3d


!pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html


from lib.colab_util import generate_video_from_obj, set_renderer, video

renderer = set_renderer()
generate_video_from_obj(obj_path, out_img_path, video_path, renderer)

# we cannot play a mp4 video generated by cv2
!ffmpeg -i $video_path -vcodec libx264 $video_display_path -y -loglevel quiet
video(video_display_path)

我的测试效果