Haiyang Liu

liuhaiyangkmj.iis.u-tokyo.ac.jp

Hi! I'm currently a final-year PhD student in Information Science and Technology, The University of Tokyo, supervised by Shunsuke Kamijo, since 2020.10. I'm working on Generative AI for 2D&3D Character Animation, such as generating full-body talking videos or 3D skeleton gestures. I received my M.E. from Waseda University, in 2020.9, and B.E. from Southeast University, in 2019.9.

I've been interning for most of my PhD journey. I'm interning at Adobe Research in Summer 2024, working on video gereration. Previously I spent a good time at CyberAgent AI Lab and Huawei Research Tokyo, for co-speech 2D video generation and co-speech 3D gestures generation, respectively.

I prefer and respect simple yet effective ideas like Associative Embedding for research. And I'm interested in research or startup project for developing the API of human-related animations - the API to support Automatic Live Streaming, Game NPC, etc.

I'm seeking full-time positions starting in 2025 fall :)

Research interests

  • Computer Graphic: Animation
  • 3D Character Animation: Co-Speech Gesture Generation
  • 2D Character Animation: Co-Speech Video Generation

Selected Publications

TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio-Motion Embedding and Diffusion Interpolation

Haiyang Liu Xingchao Yang Tomoya Akiyama Yuantian Huang Qiaoge Li Shigeru Kuriyama Takafumi Taketomi

ArXiv Preprint 2410.04221 (ArXiv) 2024

Huggingface Space | ArXiv | Project Page
GitHub stars GitHub issues

EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Mask Audio Gesture Modeling

Haiyang Liu* Zihao Zhu* Giorgio Becherini Yichen Peng Mingyang Su You Zhou Xuefei Zhe Naoya Iwamoto Bo Zheng Michael J. Black

Computer Vision and Pattern Recognition (CVPR) 2024

Huggingface Space | Replicate | Colab | Blender Addon | Media | Paper | Project Page
GitHub stars GitHub issues

BEAT: A Large Scale Semantic and Emotional Multi Modal Dataset for Conversational Gesture Synthesis

Haiyang Liu Zihao Zhu Naoya Iwamoto Yichen Peng Zhengqing Li You Zhou Elif Bozkurt Bo Zheng

European Conference on Computer Vision (ECCV) 2022

Colab | Meida | Paper | Project Page
GitHub stars GitHub issues

DisCo: Disentangled Implicit Content and Rhythm Learning for Diverse Co-Speech Gesture Synthesis

Haiyang Liu Naoya Iwamoto Zihao Zhu Zhengqing Li You Zhou Elif Bozkurt Bo Zheng

ACM Multimedia (ACMMM) 2022

Code | Paper | Project Page
GitHub stars GitHub issues

Intern Experience

Academic Services

  • Reviewer: ICLR (25) Eurographics (25), CVPR (24), ECCV (24), SIGGRAPH Asia (23, 24), ACMMM (23, 24)
  • Talk: TechBeat (Beijing, 2022); Virtual Computing (Kyoto, 2022), HuaWei (Tokyo, 2024)

Thanks

  • “I haven't been out of my room for the past ten days, but I am not sure it's could catch CVPR or postpone for Siggraph.”
    “I’m perfectly fine either way — you know best.” from Michael J. Black.
    - It was an almost give up, pressure and frustrated moment before one week of DDL.
  • “May be this time I still cant go for the registeration for CVPR.”
    “I could personally support you $200 and will aslo ask others for support you as well.” from Naoya Iwamoto.
    - Happend when company do not have the budget for conference in the third time.
  • “It requires 3-5 days training, I will only try to keep necessary experiments.”
    “If you need more 8*A100, you can use google cloud and I approval that.” from Takafumi Taketomi.
    - Under the limited budget and numbers of GPUs in AI Lab, he still gives the maximum flexibility for research.