168 lines
7.2 KiB
Markdown
168 lines
7.2 KiB
Markdown
# Rigging System Improvements
|
||
|
||
## Problem
|
||
The original rigging system had a **huge mess** in coordinate mapping between:
|
||
- Avatar image coordinates (from rigging editor)
|
||
- MediaPipe face tracking coordinates (from webcam)
|
||
|
||
This caused avatar features to not align properly with the user's face movements.
|
||
|
||
## Solution Overview
|
||
|
||
### 1. **Face Reference System** (`src/shared/types.ts`)
|
||
Added `riggingReference` to `AvatarConfig`:
|
||
```typescript
|
||
riggingReference?: {
|
||
faceCenter: { x: number; y: number }; // Center point between eyes
|
||
faceWidth: number; // Normalized width of face at eye level
|
||
faceHeight: number; // Normalized height from brow to chin
|
||
};
|
||
```
|
||
|
||
### 2. **Rigging Editor Calculations** (`src/renderer/components/RiggingEditor.tsx`)
|
||
|
||
The editor now calculates face reference points when rigging is complete:
|
||
|
||
```typescript
|
||
const calculateRiggingReference = () => {
|
||
// Face center is midpoint between eyes
|
||
const faceCenterX = (leftEye.x + leftEye.w / 2 + rightEye.x + rightEye.w / 2) / 2;
|
||
const faceCenterY = (leftEye.y + leftEye.h / 2 + rightEye.y + rightEye.h / 2) / 2;
|
||
|
||
// Face width is distance between eye centers (normalized)
|
||
const faceWidth = Math.abs(rightEyeCenter - leftEyeCenter) * 2.5;
|
||
|
||
// Face height from brow to chin
|
||
const faceHeight = chinY - browY;
|
||
|
||
return { faceCenter, faceWidth, faceHeight };
|
||
};
|
||
```
|
||
|
||
**Visual Guide**: A cyan dashed box shows the calculated "Face Reference Area" during rigging.
|
||
|
||
### 3. **Auto-Calibration** (`src/renderer/components/Studio.tsx`)
|
||
|
||
On first face detection, the system:
|
||
1. Waits 1 second for stable tracking
|
||
2. Stores initial face position as `calibrationOffset`
|
||
3. All subsequent movements are **relative** to this offset
|
||
|
||
```typescript
|
||
const relX = trackingData.translationX - calibrationOffset.x;
|
||
const relY = trackingData.translationY - calibrationOffset.y;
|
||
```
|
||
|
||
### 4. **Feature Position Mapping** (`src/renderer/components/Studio.tsx`)
|
||
|
||
Features are now positioned relative to the face center:
|
||
|
||
```typescript
|
||
const calculateFeaturePosition = (featureRect: Rect, featureType: 'eye' | 'mouth') => {
|
||
const { faceCenter, faceWidth, faceHeight } = avatar.riggingReference;
|
||
|
||
// Calculate feature position relative to face center in rigging space
|
||
const relX = featureCenterX - faceCenter.x;
|
||
const relY = featureCenterY - faceCenter.y;
|
||
|
||
// Scale relative positions by face width/height to match tracking scale
|
||
const scaledX = relX * faceWidth * avatarPosition.scale;
|
||
const scaledY = relY * faceHeight * avatarPosition.scale;
|
||
|
||
return { x: scaledX, y: scaledY };
|
||
};
|
||
```
|
||
|
||
### 5. **Exponential Smoothing** (`src/renderer/hooks/useFaceTracking.ts`)
|
||
|
||
Added smooth interpolation to prevent jittery movements:
|
||
|
||
```typescript
|
||
const smoothingFactor = 0.15; // Lower = smoother but more lag
|
||
|
||
const smooth = (current: number, target: number) => {
|
||
return current + (target - current) * smoothingFactor;
|
||
};
|
||
|
||
// Apply to all continuous values
|
||
const smoothedData = {
|
||
rotationX: smooth(prevDataRef.current.rotationX, newData.rotationX),
|
||
rotationY: smooth(prevDataRef.current.rotationY, newData.rotationY),
|
||
// ... etc
|
||
};
|
||
```
|
||
|
||
Also improved blink detection threshold from `0.5` to `0.6` for more reliable blinks.
|
||
|
||
## Coordinate Flow
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ RIGGING PHASE │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Avatar Image │ User places boxes on: │
|
||
│ │ (Normalized) │ - Left/Right Eye (Red/Blue) │
|
||
│ │ 0-1 coords │ - Mouth (Green) │
|
||
│ └────────┬────────┘ - Main Body (Yellow) │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ Calculate riggingReference: │
|
||
│ - faceCenter (between eyes) │
|
||
│ - faceWidth (eye distance × 2.5) │
|
||
│ - faceHeight (brow to chin) │
|
||
└───────────┬─────────────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ STUDIO PHASE │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Webcam Feed │ MediaPipe detects: │
|
||
│ │ (Real-time) │ - translationX/Y (-1 to 1) │
|
||
│ │ │ - rotationX/Y/Z │
|
||
│ └────────┬────────┘ - mouthOpen, blink │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ 1. Auto-calibrate (store initial offset) │
|
||
│ 2. Calculate relative movement │
|
||
│ 3. Apply smoothing (EMA with α=0.15) │
|
||
│ 4. Map rigging coords to tracking scale │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Render Avatar │ - Position from tracking │
|
||
│ │ (Composited) │ - Features from riggingReference │
|
||
│ └─────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Key Benefits
|
||
|
||
| Before | After |
|
||
|--------|-------|
|
||
| ❌ Fixed positions | ✅ Dynamic face-relative positioning |
|
||
| ❌ No calibration | ✅ Auto-calibration on startup |
|
||
| ❌ Jittery movement | ✅ Smooth exponential interpolation |
|
||
| ❌ No visual feedback | ✅ Face reference guide during rigging |
|
||
| ❌ Unreliable blinks | ✅ Improved blink threshold (0.6) |
|
||
| ❌ Scale mismatches | ✅ Proper scale mapping via faceWidth/Height |
|
||
|
||
## Testing Tips
|
||
|
||
1. **Rigging Phase**:
|
||
- Ensure the cyan "Face Reference Area" encompasses the entire face
|
||
- Eye boxes should be centered on pupils
|
||
- Mouth box should cover the lip area
|
||
|
||
2. **Studio Phase**:
|
||
- Wait for "Calibrating..." indicator to disappear
|
||
- Start with face centered in camera
|
||
- Move head slowly to test tracking range
|
||
|
||
## Future Improvements
|
||
|
||
- [ ] Manual calibration button for re-centering
|
||
- [ ] Adjustable smoothing factor (UI slider)
|
||
- [ ] Face outline overlay for alignment verification
|
||
- [ ] Multiple face support
|
||
- [ ] Save/load rigging presets
|