How to update animation data in real-time for webgl

Shinhu

Hello, I'm trying to do some sort of motion capture with spine web player.
My script without spine web player script estimates pose of a person standing in front of web cam, and generates rotation values for all bones, every 1/30 seconds (30 fps).

All left is updating the bones' rotation data in Spine web player, but i have no idea how to update json file which contains data of bone tra/rot/sca data, every 1/30 seconds.

Expected result is to see my character moving exactly same with me, standing in front of web cam.
Sort of like motion capture or real time animating.

How can I do this?

Nate

Likely you would not want to try to modify JSON data, then parse the JSON data to create the skeleton objects so you can render them. Instead, you parse the JSON data to get skeleton objects once when your app starts, then you can manipulate those objects to match the rotation generated by your webcam. To do that, you'd set the Bone rotation. To get a list of all the bones there's Skeleton bones. To get a specific bone there is Skeleton findBone, but this compares the name of each bone, so you probably don't want to call it every frame. Instead you can use it to find the bones when the app starts and store the references somewhere for use later, when you need to set their rotation.

Note that Bone rotation is the bone's local rotation. If your rotations are world rotations, there is Bone worldToLocalRotation.

Be sure to see the API reference and especially the Spine Runtimes Guide.

Shinhu

Hello, Nate! Sorry for late reply. I already saw yours 2 weeks ago, but I wanted to reply after seeing the result by myself.
Your advices helped me a lot, first letting me know this :

Nate yazdı
To get a specific bone there is Skeleton findBone, but this compares the name of each bone, so you probably don't want to call it every frame. Instead you can use it to find the bones when the app starts and store the references somewhere for use later, when you need to set their rotation.

I also checked API reference page you linked, and later checked spine-webgl.js code to see methods that were not introduced well in API reference(like, camera.viewportWidth/Height/position, resize(fit/expand))
You are genius, Nate! Thanks a lot! :grinteeth:

And... Here's a video that shows what I have done so far..if you can notice.. there are 'some' issues, and I want to get advice from you(everyone in this forum) again! Please watch it carefully.

it seems to detects arm movement and leg movement well, but almost all bones are too much shaky.

This javascript codes first detects my pose, returns coords of 12 body part keypoints. Then a function i wrote will transform each spine bones. its parameters are :
-a Spine character's bone
-two body part keypoints that correspond to character bone
-Spine skeleton object to update
Here's the code for the function :

//Webcam canvan size : 600 x 500, spine runtime canvas : 1200 x 1000
// Get two keypoints, a character bone and minConfidence
// Animate bone if minimum confidence condition is met
function animateDetectedBone(chaBone, initKeyPt, termKeyPt, minConfidence, skeleton) {
  if (initKeyPt.score > minConfidence && termKeyPt.score > minConfidence) {
    var chaScale;
    chaBone.rotation = chaBone.worldToLocalRotation(getAngle(initKeyPt.position, termKeyPt.position));
    

if (["head", "torso1", "torso2"].includes(chaBone.data.name)) {
  if(initKeyPt.position.x < termKeyPt.position.x){
    chaBone.rotation += 90;
  }
  else {
    chaBone.rotation -= 90;
  }
}

if (["torso1"].includes(chaBone.data.name)) {
  var distanceKeys = Math.sqrt((initKeyPt.position.x - termKeyPt.position.x) ** 2 + (initKeyPt.position.y - termKeyPt.position.y) ** 2);
  chaScale = (distanceKeys * 2 / 100);
  debug.camera.viewportWidth = 1200 / chaScale;
  debug.camera.viewportHeight = 1000 / chaScale;
  debug.camera.position.set(debug.camera.viewportWidth / 2, debug.camera.viewportHeight / 2,0);
}

if ("torso1" == chaBone.data.name) {
  var posCha = {x : (initKeyPt.position.x + termKeyPt.position.x), y : 1000 - (initKeyPt.position.y + termKeyPt.position.y)};
  chaBone.x = (posCha.x * (1 / chaScale));
  chaBone.y = (posCha.y * (1 / chaScale));
}

skeleton.updateWorldTransform();
  }

}

Note that webcam canvas has 600 x 500px size, and spine runtime canvas has 1200 x 1000px size.

Another issue i found that is, I'm not sure if I'm precisely translating my character upon my position in webcam. Second and Third if condition in code above is the part that deal with this, but i'm doubful if this is done correctly.. 🙁

Damn. I still have no idea what causes this character moving around so randomly :think:

My next step is to fix those shaky viewport camera movement, fixing the ratio differences between two canvases, improving pre-trained PoseNet model, and adding more features like detecting turn-around, facial bones.. whatever that sounds awesome.

But I have to solve those randomly moving bones first. Do you have any idea how to solve this...?