Project 4 - Augmented Reality Due: March 19th 2021 noon

Description: Augmented Reality projects virtual objects onto the real world. In this project, you will implement a basic AR application that can place objects and annotations within the real world. This application will consist of a simple UI that allows users to define their own annotations. For this project, it is recommended that you build and run the following scenes within ARCore and understand how they work:

1. HelloAR 2. ObjectManipulation 3. AugmentedImage (Note: To get this working you will have to use one of the images

within the Example Database, or add your own image to the database). 4. ComputerVision

Basic Functionality Requirements: Note: Graduates have an additional requirement compared to the undergraduate students.

1. (Undergraduates: 33%, Graduates 25%) Image Tracking: Use ARCore to track an image on a table. When ARCore detects an image, a model should appear at the location of the image. ARCore should be setup to allow for movement of the image.

a. For this task, it is recommended that you look at the Augmented Images example scene located in Assets > GoogleARCore > Examples > AugmentedImage. In the example scene, ARCore will try to recognize images within the example database. You will need to add your own images to that database (or alternatively create your own database for images). By default, when an image is detected, an image frame appears. You will have to modify/replace AugmentedImageVisualizer to allow the placement of your own prefab within the scene.

b. For best results for moving images, make sure both ArTrackingState and ArAugmentedImageTrackingMethod are used to determine whether the image has been detected and whether its position is known.

2. (Undergraduates: 33%, Graduates: 25%) Allow the user to place objects on planes

tracked by ARCore: ARCore Describes the current best knowledge of a real-world planar surface. Open the HelloAR example and place the the PlaneVisualizer and PawnGenerator prefab from that scene into your project. You should see that ARCore will create planes within the real world.

Placing objects onto ARCore detected planes (plane visualization is optional).

3. (Graduates only: 25%) Allow the user to define annotations: Allow users to place

text within the 3D environment. This requirement will allow users to label objects from within the room.

a. Use the built-in 3D Text primitive to add text to the scene. b. You will need to add an Input Field in order to allow the user to input text. c. Make sure there is some generic label if no label is specified. For example,

“object” or “annotation”.

d. ARCore tracks both planes and feature points. ARCore has examples of how to raycast to planes. However, you want to be able to place annotations on other surfaces or areas. You will need to determine a way to place objects based on the location of nearby feature points in screen space in order to get the depth to place the annotation.

4. (Undergraduates: 34%, Graduates: 25%) Basic UI: Allow the user to switch between placing object on plane (part 2 above) and adding text label to scene object (part 3 above). The UI should have an undo button to delete previous objects / text labels. For 490 students in place of having a button to place annotation you should have a second object the user can place in the scene.

5. Extra Credit (up to 20%): a. Give the application the ability to add annotations to objects that are very far

away with only rotational tracking, e.g. a building that is 50+ yards away. b. Guide user to take complete set of pictures of environment using 2D grid c. Guide user to find a small object hidden in the real world, e.g. by drawing a line

on the floor for the user to follow. d. Guide user to take a picture with a given pose. e. AR telementoring, with remote mentor providing instructions to local mentee via

AR. f. Get a virtual object to cast shadow on a real world table. g. Simulate a transparent display for distant geometry and a fixed user position with

respect to tablet. h. Diminished reality--erase a real world object, by painting the background on top

of it. i. Any feature you think adds to the application (tell us about it first).

Resources: 1. https://developers.google.com/ar/develop/c/augmented-images/guide 2. AR Tutorial Lab:

https://drive.google.com/open?id=1TSNMHPUsV5eDbmERsA5P516N-6fefpEm9GPivjhSA0c

Turnin instructions: In addition to all normal requirements, ensure that you give instructions on how to access all of the features of the demo (if you believe anything would be unclear to a user with no experience). Also, please submit the deliverables all as one zip file. Also name your .zip file with your name followed by Project 4. For example, VoicuPopescu_Project4.zip. All projects must be turned in through BrightSpace.. You will be required to create a video that displays the functionality and features of your project. The video should demonstrate ALL required elements of the project. The video should also demonstrate any of the “extras” have been incorporated into the project. Deliverables:

1. A video demonstrating features of the project

2. The entire Unity project. 3. The Android package (APK) build. This archive file (should have the .apk extension) is

created when the application is built and contains all the resources necessary to run the application on the Android device.

4. A Readme file which contains any extras you incorporated into the project.

Project Setup If you prefer, you can instead use the AR Tutorial Project as your starter for the project: https://drive.google.com/open?id=1_x7quzwyxljmut94vdOJPAI6snMS1eqh If you want to follow the instructions for setting up the AR Tutorial from an empty Unity Project check here: https://drive.google.com/open?id=1TSNMHPUsV5eDbmERsA5P516N-6fefpEm9GPivjhSA0c Make sure you enable Developer Mode for the tablet before you build.

Get the ARCore SDK for Unity 1. Download ARCore SDK for Unity 1.15.0 or later.

The SDK is downloaded as arcore-unity-sdk-1.15.0.unitypackage.

Create a new project and import the SDK 1. Open Unity and create a new 3D project. 2. Unity 2019 only: Select Window > Package Manager and install the following

packages:

1. Multiplayer HLAPI (required by the CloudAnchors sample) 2. XR Legacy Input Helpers (required by Instant Preview, which uses the

TrackedPoseDriver) 3. Import the ARCore SDK for Unity:

1. Select Assets > Import Package > Custom Package. 2. Select the arcore-unity-sdk-1.13.0.unitypackage that you downloaded. 3. In the Importing Package dialog, make sure that all package options are

selected and click Import.

Open the sample scene 1. In the Unity Project window, you can find the HelloAR sample in:

Assets/GoogleARCore/Examples/HelloAR/Scenes/.

Configure project settings 1. Go to File > Build Settings to open the Build Settings window. 2. Select Android and click Switch Platform. 3. In the Build Settings window, click Player Settings. 4. In the Settings window, configure the following:

Build and run the sample app 1. Enable developer options and USB debugging on your device. To enable developer

options, tap the Build Number option 7 times. You can find this option in one of the following locations, depending on your Android version:

● Android 9 (API level 28) and higher: Settings > About Phone > Build Number ● Android 8.0.0 (API level 26) and Android 8.1.0 (API level 26): Settings > System

> About Phone > Build Number ● Android 7.1 (API level 25) and lower: Settings > About Phone > Build Number

2. Connect your device to your development machine.

Setting Value

Player Settings > Other Settings > Rendering

Uncheck Auto Graphics API If Vulkan is listed under Graphics APIs, remove it.

Player Settings > Other Settings > Package Name

Create a unique app ID using a Java package name format. For example, use com.example.helloAR

Player Settings > Other Settings > Minimum API Level

Android 7.0 'Nougat' (API Level 24) or higher (For AR Optional apps, the Minimum API level is 14.)

Player Settings > XR Settings > ARCore Supported

Enable

3. In the Unity Build Settings window, click Build and Run. Unity builds your project into an Android APK, installs it on your device, and launches it.

4. Move your device around until ARCore starts detecting and visualizing planes. 5. Tap a plane to put an ARCore object on it.