/Learning-Notes

Personal records, notes and resources on Machine Learning.

Study Plan

Research_Material - Paper_List | Prog - Programming | QF-Quantitative Finance

2021-5

  • Mesh deformation (cage based)

  • Mesh control (bone based)

    • Skeleton-Aware Networks for Deep Motion Retargeting (SIGGRAPH'20) (code) (project)
    • Learning Skeletal Articulations with Neural Blend Shapes (SIGGRAPH'21) (code)
    • HeterSkinNet: A Heterogeneous Network for Skin Weights Prediction (CGIT)

  • Physical control

2020-11

  • Basic

    • Motion builder
    • Deepmind-research (code)

  • 3D human

    • Inference Stage Optimization for Cross-scenario 3D Human Pose Estimation (Arxiv) (shows generalization of 3D pose estimation)
    • View-Invariant Probabilistic Embedding for Human Pose (eccv'20) (code) (pose invariant)
    • Multi-Scale Networks for 3D Human Pose Estimation with Inference Stage Optimization (Arxiv) (shows generalization)
    • CAPE: Clothed Auto-Person Encoding (CVPR'20) (code) (interesting, just invariant of SMPL)
    • XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera (SIGGRAPH'20)

  • Retargeting

    • Skeleton-Aware Networks for Deep Motion Retargeting (SIGGRAPH'20) (code) (project)
    • Unpaired Motion Style Transfer from Video to Animation (SIGGRAPH'20) (code) (Application for skeleton-aware task)

  • Control

    • Character Controllers using Motion VAEs (SIGGRAPH'20) (prject) (exciting direction, worth reading)
    • Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction (SIGGRAPH'20) (related work may be useful)
    • A scalable Approach to Control Diverse Behaviors for Physically Simulated Characters (SIGGRAPH'20) (code) (obtain scalable and diverse character)

  • Human Character

    • Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks (SIGGRAPH'20) (deepmind; need testing for platform)
    • Example-driven Virtual Cinematography by Learning Camera Behaviors (SIGGRAPH'20) (code) (exciting and promising; very good work to follow)
    • Robust Motion In-betweening (SIGGRAPH'20) (character between frames)

  • Robot

    • Fast and Flexible Multilegged Locomotion Using Learned Centroidal Dynamics (SIGGRAPH'20) (code) (How to establish platform for extend research)
    • Learned Motion Matching (SIGGRAPH'20) (interactive robot and ground)
    • Model Predictive Control with a Visuomotor System for Physics-based Character Animation (SIGGRAPH'20) (project)

  • Virtual Reality

    • Holographic Optics for Thin and Lightweight Virtual Reality (SIGGRAPH'20)

  • Render

    • Enlighten Me: Importance of Brightness and Shadow for Character Emotion and Appeal (SIGGRAPH'20) (more details)

  • Flow

    • Constraint Bubbles and Affine Regions: Reduced Fluid Models for Efficient Immersed Bubbles and Flexible Spatial Coarsening (SIGGRAPH'20) (code)
    • Fast and Scalable Turbulent Flow Simulation with Two-Way Coupling (SIGGRAPH'20) (project)
    • Lagrangian Neural Style Transfer for Fluids (SIGGRAPH'20) (image to 3D fluid transfer)
    • Wave Curves: Simulating Lagrangian water waves on dynamically deforming surfaces (SIGGRAPH'20) (enhance detail of a water surface simulation)

  • Non-rigid

    • Homogenized Yarn-Level Cloth (SIGGRAPH'20) (project) (platform for clothing + skeleton)
    • Incremental Potential Contact: Intersection- and Inversion-free, Large-Deformation Dynamics (SIGGRAPH'20) (project) (code) (worth doing this + make person or object soft)
    • Interface Quadrature Material Point Method for Non-sticky Strongly Two-Way Coupled Nonlinear Solids and Fluids (SIGGRAPH'20) (project) (traditional)
    • Phong Deformation: A Better CO Interpolant for Embedded Deformation (SIGGRAPH'20) (make mesh soft + math)
    • Projective Dynamics with Dry Frictional Contact (project) (hair + cloth based on one author)
    • Robust Eulerian-On-Lagrangian Rods (project) (cloth)

2020-10

  • 3D human
    • Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (ECCV'20) (code)

2020-7

  • 3D human
    • DeepCap: Monocular Human Performance Capture Using Weak Supervision (cvpr'20)(project)
    • TexMesh: Reconstructing Human Texture and Geometry from Monocular Video([eccv'20])(project)
  • Inplicit field
    • Learning Implicit Fields for Generative Shape Modeling (cvpr'19)
    • BAE-NET: Branched Autoencoder for Shape Co-Segmentation (iccv'19)
    • BSP-Net: Generating Compact Meshes via Binary Space Partitioning (cvpr'20)

2020-6

I start one new task about fluid construction and robot control in Huawei. My research have little progress, i need focus.

Reading

  • 3D control
    • TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting (cvpr'20)(code)
    • Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild (cvpr'20)
  • 3D face
    • Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild (cvpr'20)(code)
    • Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (cvpr'20)
    • RetinaFace: Single-shot Multi-level Face Localisation in the Wild (cvpr'20)

2020-5

I submitted one paper for NIPS.

Reading

  • 3D Matching and Control
    • Human Motion Mapping to a Robot arm with Redundancy Resolution (paper'14)
    • Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets (3DV'19)(code)
    • RigNet: Neural Rigging for Articulated Characters (SIGGRAPH'20)(code)

2020-4

Reading

  • 3D Mesh Reconstruction
    • C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (arXiv)(code)
  • Dense Pose
    • DensePose: Dense Human Pose Estimation In The Wild (CVPR'18)(code)
    • Canonical Surface Mapping via Geometric Cycle Consistency (ICCV'19)(code)
    • SCOPS: Self-Supervised Co-Part Segmentation (CVPR'19)
    • Slim DensePose: Thrifty Learning from Sparse Annotations and Motion Cues (CVPR'19)
  • 3D Pose Estimation
    • Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimation (arXiv)(code)

2020-3

Work hard for NIPS and TPAMI. Preparing supplementary material for ECCV.

Study

Reading

  • 3D Mesh Reconstruction from image
    • HoloPose: Holistic 3D Human Reconstruction In-The-Wild (CVPR'19)(proj)
    • Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning (CVPR'19)(code)
  • 3D Mesh Reconstruction from video
    • Expressive Body Capture: 3D Hands, Face, and Body from a Single Image (CVPR'19)(proj)
    • Exploiting temporal context for 3D human pose estimation in the wild (CVPR'19)(code)
    • VIBE: Video Inference for Human Body Pose and Shape Estimation (arXiv)(code)
    • HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose and Shape Estimation (ICCV'19)
  • Unsupervised 3D Mesh Reconstruction
    • PoseNet3D: Unsupervised 3D Human Shape and Pose Estimation (arXiv)
    • Self-supervised Learning of Motion Capture(NIPS'17)(code)
    • TexturePose: Supervising Human Mesh Estimation with Texture Consistency (ICCV'19) (code)
    • Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop (ICCV'19) (code)
  • 3D reconstrunction
    • On the Continuity of Rotation Representations in Neural Networks (CVPR'19)
    • Learning to Estimate 3D Human Pose and Shape from a Single Color Image (CVPR'18)
    • Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (NIPS'19)(code)
    • Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning (CVPR'19)(code)
    • VIBE: Video Inference for Human Body Pose and Shape Estimation (arXiv)(code)
    • Neural 3D Mesh Renderer (CVPR'18)(code)
    • learning 3d human dynamics from video (CVPR'19)(code)
    • Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image (ECCV'16) sm
    • MobilePose: Real-Time Pose Estimation for Unseen Objects with Weak Shape Supervision (CVPR'20)
    • Meta3D: Single-View 3D Object Reconstruction from Shape Priors in Memory (ECCV'20)
  • 3D Detection
    • Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud (CVPR'20)(code)
  • 3D Pose Estimation
    • Metric-Scale Truncation-Robust Heatmaps for 3D Human Pose Estimation (FG)
    • Cross-View Tracking for Multi-Human 3D Pose Estimation at over 100 FPS (CVPR'20)

2020-2

A very busy month for submitting ECCV and CVPR rebuttal.

Luckily, our paper "Deep Kinematics Analysis for Monocular 3D Pose Estimation" has been accepted by CVPR.

2020-1

Super busy but less productive. Chinese new year is comming, I wish all Chinese could be safe under the threat of 2019-nCoV, hope everything works well.

Reading

  • Graph

    • Multi-Stage Self-Supervised Learning for Graph Convolutional Networks (arXiv)

  • 3D Mesh Reconstruction

    • Self-supervised Learning of Motion Capture(NIPS'17)(code)
    • VIBE: Video Inference for Human Body Pose and Shape Estimation (arXiv)(code)

  • 2D Pose Estimation

  • Loss

    • Bayesian Loss for Crowd Count Estimation with Point Supervision (iccv19)(code)

  • Unsupervised

    • Object landmark discovery through unsupervised adaptation (NIPS'19)(code)

  • 3D Pose Estimation

    • On Boosting Single-Frame 3D Human Pose Estimation via Monocular Videos (iccv19)
    • XNect: Real-time Multi-person 3D Human Pose Estimation with a Single RGB Camera (arXiv)
    • RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation(CVPR'19)(code)
    • Bottom-up Higher-Resolution Networks for Multi-Person Pose Estimation (arXiv)(code)
    • The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation (arXiv)(zhihu)
    • MaskedFusion: Mask-based 6D Object Pose Detection (arXiv)(code)

2019-12

Fortunately, i am invited to the award ceremony of IJSAI 2019.

A very busy month for preparing papers and final exams.

Study

Reading

  • Detection
    • EfficientDet: Scalable and Efficient Object Detection (arXiv)(zhihu)
    • Detectron2 (project)
    • ThunderNet: Towards Real-time Generic Object Detection (ICCV19)
    • ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design(arXiv)
    • Light-Head R-CNN: In Defense of Two-Stage Object Detector(arXiv)
  • NAS
    • RC-DARTS: Resource Constrained Differentiable Architecture Search (arXiv)
    • Understanding and Robustifying Differentiable Architecture Search (ICLR'20 oral)(code)(review)
    • EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (arXiv)(code)
    • Blockwisely Supervised Neural Architecture Search with Knowledge Distillation (arXiv)
    • Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (arXiv)(code)
  • Pruning
    • Global Sparse Momentum SGD for Pruning Very Deep Neural Networks (NIPS'19)(code)
  • Render
    • Few-shot Video-to-Video Synthesis (NIPS'19)(code)
    • Fashion++: Minimal Edits for Outfit Improvement (ICCV19)(code) : Borrow from BicycGAN and pix2pixHD
    • DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression using Learned Statistics of Natural Videos (Facebook Reality Labs)
    • Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis (TOG'19)(project)(code)
  • RL
    • Neural Painters: A learned differentiable constraint for generating brushstroke paintings (arXiv)(code)
  • Tracking
    • You Only Look Once: Unified, Real-Time Object Detection (arXiv)

2019-11

Main focus: preparing for ICML and ECCV.

Prepare CVPR submissions and supplementary materials.

Study

Reading

  • Unsupervised 3D Pose Estimation
    • Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation (CVPR'19)
    • Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction (NIPS'19)
    • Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning (NIPS'18)
    • Domes to Drones: Self-Supervised Active Triangulation for 3D Human Pose Reconstruction(NIPS'19)
    • Unsupervised 3D Pose Estimation with Geometric Self-Supervision
  • Graph
    • Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation(IJCAI'18)
    • Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition(AAAI'18)
    • Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (CVPR'19)
  • NAS
    • ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (ICLR19)(code)
    • FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search (facebook)(code)
    • Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (arXiv)

2019-10

Work hard for CVPR2020 and PRCV2019 Challenge workshop (Rank 8th).

2019-9

Start for PHD in Vision and Learning Lab supervised by Bingbing Ni and Wenjun Zhang.

Before 2019

  • ILSVRC 2015: Classification+localization with additional training data (Rank 1st).
  • ILSVRC 2016: Object detection/tracking from video with additional training data (Rank 1st).
  • ILSVRC 2016: Object detection from video with provided/additional training data (Rank 1st).
  • ILSVRC 2017: Object detection with provided/additional training data (Rank 1st).
  • DAVIS Challenge 2016（just in experiments）: Unsupervised Video Segmentation (When i was intern in MSRA supervised by Yan Lv and Xiulian Peng in 2018) (Rank 1st)