0

How to placed the virtual object on the real object by using pixel coordinate and raycast. I stuck on this problem for a month. T^T

Based on the question from here https://forums.hololens.com/discussion/3117/pixel-coordinates-to-world-coordinates

I do a similar thing, but I try to do live camera on Hololens to detect the object on the camera instead of taking a picture. I've been able to get the pixel coordinate (x,y) of the center of the object which appears on the camera by using OpenCV. I have tried to use screenToWorldPoint in Unity and used that vector to do raycasting, but the virtual object wasn't placed on the real object. After I found the website above, I think screenToWorldPoint may not work to achieve my target.

From the answer on the website, he/she suggested to draw the ray from the camera COP through the image object by using the pixel coordinates, the focal length of the camera, and projectionMatrix but I have no idea how should I do it. Could you please help me by explaining in more details or give me some suggestion where I should have to understand before I will do it?

Here is some of my code. X and Y are pixels that I got from the camera. The camera resolution is 896x504. I set the camera projection in Unity to orthographic. The size is 0.14. The distance of near plane is 0.8 and The distance of far plane is 1000.

    Vector3 PosFar = new Vector3(896-X, Y, Camera.main.farClipPlane);
    Vector3 PosNear = new Vector3(896-X, Y, Camera.main.nearClipPlane);
    Vector3 PointPosFar = Camera.main.ScreenToWorldPoint(PosFar);
    Vector3 PointPosNear = Camera.main.ScreenToWorldPoint(PosNear);
    var headPos = PointPosNear;
    var gazeDirection = PointPosFar - PointPosNear;
    RaycastHit hit;
    if (Physics.Raycast(headPos, gazeDirection, out hit, 30.0f, SpatialMappingManager.Instance.LayerMask)){......Generate object....}
  • If the question is about drawing the line for debugging checkout Debug.DrawLine to draw the line in the SceneView – derHugo Jun 7 at 11:58
  • 1
    @Bmychocolate Can you explain a little more what you mean by "the virtual object wasn't placed on the real object"? Do you mean to say it didn't place anywhere? Or do you mean that it placed somewhere unexpected? If it was placed somewhere unexpected, where was it placed? Make sure you have a Spatial Mapping Collider in the scene and it is enabled, and that its layer matches one of the selected layers in the SpatialMappingManager.Instance.LayerMask. – Ruzihm Jun 7 at 14:06
  • @derHugo Sorry to make you confused. Maybe I used the wrong question. Actually I would like to know how to place the virtual object on the real object by using pixel coordinate and raycast. I will chage my topic and question. – B mychocolate Jun 7 at 14:14
  • @Ruzihm It placed on unexpected area. I already had a Spatial Mapping Collider in the scence. I can shoots the ray to the same direction with the object appear on the camera from the code above, but the ray didn't point to the real object. – B mychocolate Jun 7 at 14:38
  • 1
    @Ruzihm The result is the same. I will explain what I saw on the display of Hololens. The line was around the real object. - Ray shoot to the right-hand side of the object when the object appears on the right-hand side of the center of the camera. - Ray shoot to the left-hand side of the object when the object appears on the left-hand side of the center of the camera. - When the object appears below the center of the camera. The ray shoot to below of the object or near the center of the object. -When the object appears above the center of the camera. The ray shoot to above of the object. – B mychocolate Jun 10 at 2:01
1

Are you using MRTK? If yes, FocusManager has a bool for debugDrawPointingRays. If you set this true, you should see the ray in scene mode. Maybe the way its implemented can help you...

What I also found: https://gamedev.stackexchange.com/questions/150602/how-can-i-draw-a-line-of-the-raycast-hit-in-game-view-while-the-game-is-running/150616

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.