3D Joint positions from SMPL Model (ref: [19])

[J-M-pixel]

Hi,

I am trying to acquire mesh vertices and 3D joint positions by providing pose and shape parameters (provided as ground truth in SURREAL) to the SMPL model given in ref. [19]. However, the 24 3D joint positions which I get from SMPL model, are different from what is provided as the ground truth 3D joint positions in SURREAL.

Could you please guide me so that I could acquire same 3D joint positions from SMPL model as provided by SURREAL in the ground truth ?

-- Thanks

[gulvarol]

Hi, did you rotate the SMPL model by zrot, i.e. multiply zrot (convert euler to rotation matrix) by pose[0:3] (convert axis-angle to rotation matrix)? The global rotation around the z-axis was saved in zrot variable.

[xiangdonglai]

Hi Gül,

Do you mean that from the output of SMPL (rotation in pose[0:3] is already taken into account) we just need an additional rotation around the z-axis?

I do this but the result seems still inconsistent. Could you please clarify?

Best,
Donglai

[buaacyh]

When I test the gt2d joints of the surreal dataset, I find the problem that left and right joints are exchanged(compare to smpl ground truth). I draw the picture and draw the 23rd joints of surreal which shows the point is left fingers. However, when I find in smpl ground truth, the 23rd joints means right fingers. Is my problem?

[ypflll]

@jameel-malik Also meet this problem. Do u find a solution?

[ypflll]

@gulvarol Hi, Gül. Thank you for sharing your good work.
I also met a problem when I try to project SMPL 3d vertices to 2d. I use the code below:

## Load SMPL model (female model or male model)
m = load_model(model_name)

## for the first frame
m.pose[:] = info['pose'][:,0]
m.betas[:] = info['shape'][:,0]

### matrix get from the matlab code, here I use files in 01_01
intrinsic = np.array([[600,0,160],[0,600,120],[0,0,1]])
extrinsic = np.array([[0,0,-1.,-1.9292990],[0,1.,0,0.98388302],[-1.,0,0,6.7456222]])
M = np.dot(intrinsic, extrinsic)

vertices = np.concatenate((m.r, np.ones((m.r.shape[0],1))), axis=1)
coord_projected = np.dot(M, vertices.T)
coord_projected[0,:] /= coord_projected[2,:]
coord_projected[1,:] /= coord_projected[2,:]

Result (green points are the projected joints3D which seems right):

I tried to multiply a rotation matrix provided by zrot as you mentioned above, but still not right:

theta = info['zrot'][0]   #*180/math.pi
Rot_z = np.array([[math.cos(theta), -math.sin(theta), 0], [math.sin(theta), math.cos(theta), 0], [0, 0, 1]])
m.pose[0:3] = np.dot(Rot_z, m.pose[0:3])

Result:

This really confuses me. What's the righ way to get compatibility between SMPL and SURREAL?

[ypflll]

@gulvarol Still not very clear about this. Can you take a look at this?

[gulvarol]

Sorry for the delay. I have added some sample code to visualize the projection from SMPL vertices and from SMPL 3D joints to 2D here: https:/gulvarol/surreal/tree/master/datageneration/misc/smpl_relations

Indeed, there is a left/right swap that needs to be done. Sorry for inconvenience.

I also added some rotation and translation. Now the projected vertices match the image as below:

However, note the slight mismatch between the 3D joints, which happens especially in male bodies (almost negligible). It almost never happens in female bodies. It's because the joint_regressor [24 x 232] is a subset of the J_regressor [24 x 6890] from the female pkl model.

surreal/datageneration/main_part1.py

Lines 329 to 331 in 6a80037

	def reset_joint_positions(orig_trans, shape, ob, arm_ob, obname, scene, cam_ob, reg_ivs, joint_reg):
	# since the regression is sparse, only the relevant vertex
	# elements (joint_reg) and their indices (reg_ivs) are loaded

[ypflll]

@gulvarol Thank for clearifying this.
I've solved this by refering another repo, which uses the relationship between 2d and 3d joints, solving a linear equation and geting the intrinsic matrix.
A little complex and time costing, but works.

[J-M-pixel]

Hello Gul Varol,
Thanks for the clarification and the additional code to support.
Thanks again !

[ypflll]

@gulvarol Tested this code. Find a bug: for train/run2/132_50/132_50_c0002_info.mat, frame 49, get this:

Eps for rotmat2rotvec(R) is too small. Change 1e-4 to 1e-3 can avoid this. Not sure what is the critical value.

[gulvarol]

Thanks for reporting, I removed the dependency to the rotations.py file and replaced with the functions from transforms3d package instead. Hopefully it's more robust.

[chaitanya100100]

@gulvarol Thanks for explanation and code. I would just like to ask some doubts.

Using your code, I am able to transform mesh and joint locations from the world coord system to the camera coord system by multiplying extrinsic matrix. But whenever we are trying to train any model for SMPL parameters estimation, we treat global rotation with respect to the camera. How can we convert SMPL global rotation parameters (which are according to some world coord system) to make it in the camera coord system?

[wangsen1312]

@gulvarol Thanks for your great work.
I have one question: I ran your smpl_relations.py code, The smpl_joints3D and joints3D are not exact the same (with little difference)? Does it correct?

[gulvarol]

Yes, see above my answer about the joint_regressor (to go from vertices to joints) in my Blender implementation not using all 6890 vertices, but a subset of 232; and the female joint_regressor for both genders.

[wolterlw]

@gulvarol Thank for clearifying this.
I've solved this by refering another repo, which uses the relationship between 2d and 3d joints, solving a linear equation and geting the intrinsic matrix.
A little complex and time costing, but works.

@ypflll, could you link the repo that has code solving for the intrinsic matrix?

[RhythmJnh]

@gulvarol Thanks for explanation and code. I would just like to ask some doubts.

Using your code, I am able to transform mesh and joint locations from the world coord system to the camera coord system by multiplying extrinsic matrix. But whenever we are trying to train any model for SMPL parameters estimation, we treat global rotation with respect to the camera. How can we convert SMPL global rotation parameters (which are according to some world coord system) to make it in the camera coord system?

@chaitanya100100, Did you solve this problem？

[wine3603]

@gulvarol Thanks for explanation and code. I would just like to ask some doubts.
Using your code, I am able to transform mesh and joint locations from the world coord system to the camera coord system by multiplying extrinsic matrix. But whenever we are trying to train any model for SMPL parameters estimation, we treat global rotation with respect to the camera. How can we convert SMPL global rotation parameters (which are according to some world coord system) to make it in the camera coord system?

@chaitanya100100, Did you solve this problem？

same question, did you find the way to output 3D coordinates of smpl vertex?

[gulvarol]

I am not sure I understand the problem. The vertices of the model can be accessed with model.r.
Please have a look at the sample script included in the smpl_relations folder (see above) to understand the relationships between different coordinate systems.

[gsygsy96]

How to correct the mirror problem?

[thancaocuong]

@gulvarol How I we solve the swap left-right problem??