Using `Enter AR` Button Causes Renderering Issues With Markers Resulting In Offset Positions, And AR Markers Move When Headset/Camera Changes Position
loganknecht opened this issue · 1 comments
Hello!
Really love your library! Thank you so much for making it
Environment
- Magic Leap 2
- Helio Browser
- Remote Debugging via Chrome
Expected Behaviour
I expect AR Markers to render correctly where the AR Marker is.
The Problem
I am using a Magic Leap 2 headset to test WebXR development.
I am encountering an issue where AR Markers are being incorrectly rendered when I am in AR Mode
after selecting the Enter AR Mode
button. This is when the web browser is expanded into a full screen mode.
The issue is that when I am rendering my scene outside of AR mode, the markers are correctly translating and placed correctly in world space. It's perfect. There are no complaints.
However when I enter AR Mode my markers are incorrectly placed and offset in the opposite direction of my camera movement. I have no idea why this is.
Demo Video
I have created a video here to demonstrate the issue
bad_performance_small.mp4
The Code
package.json
{
"name": "@artcom/react-three-arjs-example",
"version": "0.1.0",
"license": "MIT",
"scripts": {
"watch": "webpack serve --hot --https --mode=development",
"build": "webpack --mode=production",
"start": "http-server -S -C cert.pem dist/"
},
"engines": {
"node": ">= 16.15"
},
"repository": {
"type": "git",
"url": "git+https://github.com/artcom/react-three-arjs.git"
},
"devDependencies": {
"@babel/cli": "^7.22.6",
"@babel/core": "^7.22.8",
"@babel/preset-env": "^7.22.7",
"@babel/preset-react": "^7.22.5",
"@pmmmwh/react-refresh-webpack-plugin": "^0.5.10",
"babel-loader": "^9.1.3",
"copy-webpack-plugin": "^11.0.0",
"eslint": "^8.44.0",
"eslint-config-prettier": "^8.8.0",
"eslint-plugin-compat": "^4.1.4",
"eslint-plugin-import": "^2.27.5",
"eslint-plugin-prettier": "^4.2.1",
"eslint-plugin-react": "^7.32.2",
"eslint-plugin-react-hooks": "^4.6.0",
"html-webpack-plugin": "^5.5.3",
"http-server": "^14.1.1",
"react-refresh": "^0.14.0",
"webpack": "^5.88.1",
"webpack-cli": "^5.1.4",
"webpack-dev-server": "^4.15.1"
},
"dependencies": {
"@ar-js-org/ar.js": "^3.4.5",
"@artcom/react-three-arjs": "0.5.6",
"@react-three/drei": "^9.88.2",
"@react-three/fiber": "^8.14.5",
"@react-three/xr": "^5.7.1",
"buffer": "^6.0.3",
"core-js": "^3.31.1",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"three": "^0.154.0"
},
"browserslist": [
"defaults"
]
}
index.html
<html>
<head>
<title>@artcom/react-three-arjs example</title>
<meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0" />
<style>
body {
/* width: 100%;*/
/* height: 100%;*/
/* margin: 0px;*/
/* padding: 0px;*/
/* overflow: hidden;*/
}
</style>
</head>
<body>
<div id="root"></div>
</body>
</html>
index.js
// Third-Party
import { ARCanvas, ARMarker } from "@artcom/react-three-arjs";
import {
Box,
Environment,
OrbitControls,
// PerspectiveCamera,
Stage
} from "@react-three/drei";
import {
ARButton,
// Controllers,
Hands,
// Interactive,
XR,
// useHitTest,
useXR
} from "@react-three/xr";
import React from "react";
import { createRoot } from "react-dom/client";
// Custom
// N/A
function ConstellationStage(props) {
// -------------------------------------------------------------------------
// State
// -------------------------------------------------------------------------
const current_xr_state = useXR();
// -------------------------------------------------------------------------
// Rendering
// -------------------------------------------------------------------------
const placeholder_view = (
<Stage
// for formatting
adjustCamera={true}
intensity={1}
preset="rembrandt"
shadows="contact"
>
<Environment preset="forest" background={true} />
<OrbitControls />
<ambientLight />
{/*{current_xr_state.isPresenting === false ? null : null}*/}
</Stage>
);
const ar_view = (
<>
<Hands />
<ambientLight />
{/*<pointLight position={[10, 10, 0]} intensity={10.0} />*/}
<ARMarker
params={{ smooth: true }}
type={"pattern"}
patternUrl={"data/patt.hiro"}
onMarkerFound={() => {
console.log("Marker Found");
}}
>
<Box
args={[1, 1, 1]}
material-color="hotpink"
// position={[0, 1.5, -1]}
/>
</ARMarker>
<Box args={[0.1, 0.1, 0.1]} material-color="red" position={[0, 1.5, -1]} />
<Box args={[0.1, 0.1, 0.1]} material-color="blue" position={[0, 1.6, -1.2]} />
<Box args={[0.1, 0.1, 0.1]} material-color="green" position={[0, 1.7, -1.3]} />
</>
);
const final_render_element = current_xr_state.isPresenting === true ? ar_view : placeholder_view;
// return final_render_element;
return ar_view;
}
function ConstellationScene(props) {
const final_render_element = (
<div
// style={{ height: "100vh", width: "100vw" }}
>
<ARButton />
<ARCanvas
// gl={{ antialias: false, powerPreference: "default", physicallyCorrectLights: true }}
onCameraStreamReady={() => console.log("Camera stream ready")}
onCameraStreamError={() => console.error("Camera stream error")}
onCreated={({ gl }) => {
gl.setSize(window.innerWidth, window.innerHeight);
}}
>
<XR>
<ConstellationStage />
</XR>
</ARCanvas>
</div>
);
return final_render_element;
}
createRoot(document.getElementById("root")).render(<ConstellationScene />);
Code Sandbox
It is available here to reproduce
https://codesandbox.io/s/example-react-ar-js-vx2dj4
Speculation
I have reason to believe that there is something going on in AR Mode that is causing the rendering to get messed up with the camera itself.
I don't know why, or how.
Are there settings that I need to be mindful of when entering AR Mode?
Work Around Request
Is there a way to emulate AR Mode without using your library? I have a sneaking suspicion if I were able to do AR Mode in a different approach I might see the correct behaviour.
You are mixing ARjs and WebXR, which are different approaches to the same problem, with partially non overlapping capabilities. I recommend checking if one of both approaches can be used to implement your use case. You probably want to check if the WebXR implementation of the device you are using supports Image Tracking