MetaHuman to VRChat Avatar | Webcam Face Tracking

Introduction

I used to think MetaHuman only worked in Unreal. Then I saw people bringing it into Blender, and I started thinking: if it can get into Blender, it should be able to get into Unity too. It could become a VRChat avatar, with its expressions driven live by a laptop webcam. And if that works, maybe I could even try putting a MetaHuman into standalone Quest 3 VR content.

For now, I've finished the PC-only version. The avatar's face has 172 blendshapes baked from Epic's RigLogic rig, and its expressions are controlled live through a free webcam tracker and OSC.

The tools and more complete step-by-step instructions are on GitHub: metahuman-to-vrchat. This post is mostly about how I did it, and the parts that had me stuck for a while.

Unreal to VRChat

MetaHuman used to be tightly tied to Unreal Engine. Starting with UE 5.6, it's covered by the standard Unreal Engine EULA, so it can be used for free in other engines and platforms as long as revenue stays under US$1 million. That's why I can now upload a MetaHuman avatar I made to VRChat. Publicly distributing the original assets is a different matter, so this repo only includes the tools. You'll need to use your own DNA files, textures, and meshes.

The other hard part was the face. As far as I know, VRChat mostly relies on blendshapes for expressions, but MetaHuman uses RigLogic, which drives a large number of facial joints and corrective shapes. If you export an FBX directly, none of those expressions come with it. So in the end, I had the rig make each expression one by one and baked the results into blendshapes.

Pipeline

Unreal Engine 5.6   MetaHuman Creator -> head.dna, body.dna, textures, hair cards
Blender 5.1         Poly Hammer Character DNA runs RigLogic -> bake 172 shape keys -> Avatar.fbx
Python              Repack textures, flip normal maps, draw hair strand atlas
Unity 2022.3.22f1   Import settings, Poiyomi materials, Jerry's ARKit face-tracking template
VRChat              webcam -> FoxyFace -> VRCFaceTracking -> OSC -> avatar

The Blender stage runs as a headless script. On my laptop, regenerating the FBX from head.dna takes under two minutes. Later I reran the public version on my archived data, and the FBX it produced was bit-for-bit identical to the file I originally uploaded. So I could finally confirm I hadn't broken the pipeline while cleaning up the repo.

Blender

The final bake has 172 shapes: the 52 ARKit shapes, VRChat visemes, extra expressions from Unified Expressions, MetaHuman-specific controls, and emotion presets. Compared with live RigLogic, the maximum error is 0.00036 mm.

What surprised me most here was that the shape-key import option in the free version of the Poly Hammer addon doesn't actually do anything. Without the corrective shapes, the small details around the lips and eyelids disappear, so I ended up reading them out of the DNA myself. The first bake was also misleading. The script finished without problems and produced 169 shapes, but only the first one had the correct correctives. The only sign of the problem was in stderr.

I also tried Blender's "New Shape From Mix", but it ignores the armature, and most of MetaHuman's expressions are driven by the armature. Some controls also don't move at all on the neutral face. Lips-together is one of them, so I had to use an open jaw as the base and bake the difference. All of these details are built into the pipeline in the repo.

Textures and Hair

At first I assumed the normal maps didn't need flipping. The lighting on the pores ended up coming from the wrong direction, and nothing reported an error. I made a small controlled test with the skin maps and confirmed that Unreal and Unity use opposite green channel directions.

Hair can't be exported as a single texture either. MetaHuman uses a procedural strand shader, so instead I drew strand atlases for the hair, eyelashes, and eyebrows, and mapped the hair cards onto them. Surprisingly, the eyebrows were the hardest part: 936 cards stacked on top of each other, so if each one is even a little too opaque, the whole thing turns into a solid black patch. (.m.)

Unity & VRChat

On the surface, the Unity stage is just adjusting settings, but it's easy to trip over errors that give no warning. Once I turned off "Convert Units" on the FBX, and the avatar became 170 meters tall. Read/Write and Streaming Mip Maps also have to be on, or the SDK won't let you build. So I made a small editor package that applies and checks these settings.

For materials, Poiyomi's defaults are better suited to toon avatars. Because this character has a realistic face, it didn't really pick up the ambient light in different worlds. After I changed four values, the face started following the warm and cool tones of each scene.

Before uploading, you also need to lock the Poiyomi materials into generated shaders. Once I missed the eye material, and it showed up magenta in VRChat. Now the package blocks any material that hasn't been locked.

While cleaning up the repo, I found another awkward problem. The skin scattering LUT looked fine in the editor, but after locking the shader, it tinted the cheeks red in shadow. The avatar I'd already uploaded has this issue. For now, I've removed the LUT from the tools.

Driving Expressions with a Webcam

Right now, face tracking works like this: FoxyFace picks up MediaPipe landmarks from the webcam feed and sends them through VRCFaceTracking and OSC into VRChat. Then Jerry's ARKit template maps them to the 52 ARKit blendshapes.

The first time I connected everything, three parts happened to be broken in three different ways. OSC was off in VRChat, I had installed the VRCFaceTracking module by copying the DLL (it actually needs to be installed from the module registry), and FoxyFace itself wasn't running. Once I checked which process was using each UDP port, I found the cause in a few seconds. That check is in the repo too.

The whole pipeline runs now, and the avatar's face follows mine. But FoxyFace isn't calibrated yet, so the eyes currently look half-closed. I did record a set of eyelid parameter values, but the OSC parser had a bug at the time, and the values at the end look like I'd moved out of the camera's view. So I'm not treating them as a conclusion yet. Once calibration is done, I'll add a real before-and-after comparison.

Current Limitations

On PC, the Performance Rank is Very Poor, mainly because the model has 110,616 triangles. The head also carries the blendshapes, so I can't cut the polygon count much without losing facial detail. The other metrics are Good or Excellent, but many players hide Very Poor avatars by default.

This is also a Desktop mode setup. Once you put on a headset, the webcam can't see your face. To track expressions in VR, you'd need face tracking and eye tracking hardware, plus a different tracking module (which I don't have yet XD). So far, I've only tested this version on my own Windows 11 laptop with the specific versions listed. I haven't tested newer versions of the Blender addon or FoxyFace.

What's Next

Next, I might try hair physics bones and automatic blinking when tracking is off. The shape needed for auto-blink is already an option in the tools. If I want to use it with a headset later, that's probably when I'll look into VR tracking setups.

Repo and step-by-step guide: github.com/Ducklexander/metahuman-to-vrchat