Vision-inertial sensor fusion for precise augmentation on display devices

Augmented reality (AR) is a technology that overlays virtual 3D content in the real world to enhance a user’s perception. These techniques are increasingly adopted in different fields, ranging from games to extreme environments. This thesis focuses on AR for a maintenance application in an ATLAS particle physics detector (in Large Hadron Collider (LHC)). An AR application greatly depends on the pose estimation algorithms to know the desired object location in a real world scene. It is also very important to have a jitter-less and smooth virtual content rendering on the AR display devices. Sensors like inertial measurement units (IMUs) and cameras are used for the pose estimation purpose. Although, the pose data from vision algorithms are accurate, they often require longer computational time and they suffer during the fast movements. On the other hand, an IMU can provide fast data rates from which an absolute pose can be determined with fewer calculations. However, the fundamental problem with an IMU is the drifts over time which greatly depends on the cost, size and weight of an IMU. The commercially available head mounted displays (HMDs) have in-built IMUs that are hobbyist (low cost) grade with high frequency noise and low frequency drift. The objective of this thesis is to analyze and efficiently reduce the registration error caused by the IMU (high and low cost) drift in an AR application. This drift can be substantially reduced by fusing periodic pose updates from a vision algorithm. Keeping the intent of accuracy and fast pose estimation in mind, we propose a hybrid coupled fusion of the camera and the IMU using Extended Kalman filter (EKF) with two state-space models, 1) constant velocity-constant angular velocity process (CVCAV) model and 2) constant acceleration-constant angular velocity (CACAV) model. The experimental results obtained under static and simulated head (pan and tilt) movements show that the CACAV model is very well suited for both the low and high cost IMU. In an extreme environment, the AR virtual information should be relevant without distracting the operator’s attention. For this reason, we carefully designed and developed different visualization schemes to test and evaluate the best suited virtual content for each maintenance procedure. Developing this AR visualization content requires programming skills and it is a time consuming task. This problem is addressed by an authoring tool that relates the real object with the virtual AR counterparts. In this thesis, we propose an authoring tool using Unity3D game engine with user-friendliness and re-usability as its two fundamental objectives. The usability evaluation study illustrates that our tool is easy to learn and use.