Set up Augmenta Hardware
INTRODUCTION
This guide might be useful for anyone that must deploy an Augmenta installation or design an interactive space using Augmenta. The topics discussed in this guide will be the following:
Hardware setup for an installation
Calibration process
Technical study and simulation
Example use cases
For details about how to use the different softwares of the solution, see the User Guide.
The first part of this guide discusses the main installation guidelines. You will learn how to correctly set up the nodes physically as well as the calibration process.
The second part is about planning and designing an interactive area. It might help you to answer questions like “how many nodes do I need ?” and what you will be able to do with your setup. This part will answer the following questions: will you be able to track people’s height ? What is the setup used to track position ? Will I have occlusion issues ?).
The goals of this guide are:
Help you understand the whole technical set up process of the Augmenta solution
Give you hints about the ideal setup for you and the potential tradeoffs you are making
Outline the possibilities of the solution
SETUP GUIDELINES
Hardware setup constraints
The Augmenta nodes should be placed in order to cover the entire desired area. Where the nodes should be placed in space depends on the use case, constraints of the space you are setting up in and the sensor constraints. Constraints of the sensors can be found in the User Guide. Multiple use cases can be found in the Interactive Area Design section. An example of setup for tracking people, using Astra nodes is given below.
In order to use several nodes with a server running AugmentaFusion, the nodes and the server should be on the same network in order to communicate via OSC.
Some general installation guidelines for camera nodes are given below :
Camera height should not be below 1m or above 6m.
The ideal angle for the cameras is 0° (facing the ground directly). You can give an angle to cover a larger surface. The camera angle should not go above 45°.
The main plane seen by the camera must be the ground. If this is not the case, you will have to select manually the ground plane or disable the reprojection of the data. Otherwise the data will make no sense.
Ideally the ground should be light and mat. The darker and reflective the ground is, the closer the camera should be to the ground. A perfect reflective or transparent surface will not be seen by the cameras.
If the ground cannot be seen (e.g. it is black in the Input view in the node’s web interface), the background can be learned using another temporary material on the ground surface (paper, sheet, cardboard, etc…). Once the background is computed, the background plane can be recomputed.
If the ground cannot be seen by any means, Augmenta will work without reprojection and the camera's angles should be 0°.
Cameras overlap should be between 0.5 and 2 meters depending on the application. A high overlap will improve the tracking but increase the number of cameras required for the same area.
For long term installations, we advise to disable the background once the system has been calibrated (set background to Off) in the web interface on each node. This is because data coming from the camera may shift depending on the temperature, and the background is quite sensitive to this kind of variation.
Setup process
Augmenta Setup
Setting up Augmenta differs slightly depending on the desired use case (video, sound, light). For video projection, the setup is detailed below. For the other use cases, the calibration steps and prerequisites may vary slightly.
Prerequisites depending on the technology used to display the content
Video projection: A video-mapped projection surface with a video software able to use a Spout or Syphon video input. You also need to know the resolution expected by the media server (or video projector) as this is the resolution you need to put in AugmentaFusion for the Spout texture size.
LED or TV screen: You need to know the resolution expected by the media server (or the screen) as this is the resolution you need to put in AugmentaFusion.
Others: Check the Use Case Section.
For a complete Augmenta setup, follow these steps:
Set up your nodes in the physical space, according to the node specifications and desired area coverage.
Once you have provided power and networking to your node, use AugmentaFusion to see if all your nodes are detected. You can access the web interface of each node through AugmentaFusion (see the User Guide to find out how to use AugmentaFusion, or the step 8 below).
Ensure that your nodes have sufficient overlaps. This can be visualized through each node interface. To set up the overlap visually, open the web interface for both nodes, be sure to display the input stream. Place an object in the overlap area (a bar stool works fine for this). If you can see the object in both interfaces’ input panel all along the centerline of the overlap it should be fine. Reproduce these steps for all overlap areas. You should at least see the knees of someone standing in the middle of the overlap area, higher is better but you should not go under or you might have tracking issues.
Note : you can use some Mylar metalized polyester film tape to set up marks in the scene that can be seen by the depth cameras.
Configure each node with the following steps :
Note : Due to the sensitivity of Astra Pro camera to temperature, the following steps must be performed in the closest heat conditions as possible as the performance conditions.
Connect to the node’s web interface with a laptop, tablet or smartphone.
Configure the Augmenta parameters of the node in each panel successively (Input -> Background -> Processing -> Output) according to your scene. A good calibration should result in stable blobs with a colored bounding box in the Output, and no noise appearing with a bounding box.
Save the node configuration. Note that Augmenta has an auto-save and will regularly save the last configuration.
Open AugmentaFusion on your media server/main computer.
Enter the whole area size in meters (in the Inspector, when clicking on the scene you want to configure). This size is the physical size of your interactive area (and also in many cases your projection).
Enter your desired output resolution in the Inspector panel. The desired output resolution is the projection resolution expected by the display device. This is necessary for video calibration and/or if your display device is a video projector, a screen or a LED panel. This is not needed if you do not have a display device (lights or sound). Note that Fusion requires that you enter a ratio that corresponds to the ratio of the scene size (in meters), else it will change either the resolution or the scene size to keep the ratio (see the User Guide for more details).
Add your nodes (if you have not already added them) in AugmentaFusion sources. Click on the green “plus”, then you can automatically add all detected nodes, add a specific one, or manually add one. When adding a node manually you will have to specify its IP address.
Calibrate each node in AugmentaFusion (cf AugmentaFusion calibration below)
AugmentaFusion calibration
The calibration of each Augmenta node in AugmentaFusion is done with a two-point calibration. This is done by following these steps:
Connect the Augmenta node(s) to have it(them) as a source(s) in AugmentaFusion and enable the source(s) if needed.
Click on the scene you want to calibrate (in the Scenes tab on the top of the app). In the inspector, enable ‘Video Output (Spout)’ in order to have a spout output with the Merger panel content.
Display the spout output called AugmentaFusion on your projection display (using your video projection software) in order to see the Merger window on your projection area.
Check that the background squares are actual squares on the projection surface. If this is not the case, either the output resolution in AugmentaFusion is not correct, or the video-mapping is not correct.
Calibrate each source with the following steps:
The IRL (=In Real Life) and Fusion in parenthesis in the following step are there to help understand which part of the instructions are to be performed physically on the interaction area, or into AugmentaFusion’s interface.
Enable only the current source
(IRL) Put a single object in your interaction area. You should see a single point in AugmentaFusion.
(Fusion) Translate the source transformer by dragging and holding the left click in the Merger window until the detected point you see in the projection matches the physical position of the object (IRL).
(Fusion) Move the transformer anchor (the orange cross) to the center of the detected object by right clicking near the detected object, and select ‘set anchor to closest object’ or by holding Alt and dragging the anchor with the left click.
(IRL) Move the object to the opposite side of the interaction area. The detected point should also move in AugmentaFusion.
(Fusion) Rotate and scale the transformer (hold Maj key and drag the source) around its anchor until the detected point you see in the projection matches the new object's physical position (IRL).
You can check that your position is correctly detected when moving around the detection area.
Enable all your nodes and check that you are correctly detected everywhere, and especially in the overlap areas.
When your whole area is calibrated, Disable Draw Enabled and Out Texture in AugmentaFusion to improve its performances.
Save your AugmentaFusion configuration.
You are done 🤘
Note that since Spout is used by AugmentaFusion to calibrate with video projection, AugmentaFusion should be on the machine that generates the video projection for the video calibration. Once the calibration is done, Spout is no longer used and AugmentaFusion can then be deported on another machine. Once AugmentaFusion supports an NDI output, it will no longer need to be on the same machine for calibration. You could also use a software such as “Spout To NDI'' to send Spout textures over the network to another computer if you cannot do it this way.
Calibration without video
You do not need a video output to calibrate Augmenta. In that case, you can manually do the same process without seeing the projected result. You just need to measure the coordinates and report it in fusion.
The calibration of an Augmenta node can then be done by following those steps:
Get your whole area size in meters. Enter it in AugmentaFusion.
Put an object in your area and measure its coordinates in meters from one corner of the area.
Place the transformer of the source in fusion so the object is at the same coordinate in the scene as in real life (the squares in Fusion are 1m x 1m). For example if you place your object at (1m,1m) from the top left corner in real life, you should place the source so that the detected object is in the bottom right corner of the top-left square from the grid.
Place the red cross on the detected object
Move the object to another place in your area and measure its new coordinates.
Rotate and scale the transformer of the source to match the detected object to the coordinates
Managing Nodes
Using the source controls in AugmentaFusion, you can easily wake (All), shutdown (All) or restart (All) each (all) node(s) on the network..
Tips : If the node was never detected before, you can manually add a new source and enter the node mac address in the properties, then wake it.
INTERACTIVE AREA DESIGN
Layout design method
When you need to turn an area interactive, the first question to answer is: how many nodes do you need ? To answer this, you need to know which 3D camera / Lidar you will use.
It mainly depends on the environment (height of the ceiling, light pollution - especially the infrared wavelength -, occlusion, FOV of the camera). For a first approach of the sensor placement, we recommend the simple use case described below, the zenital point of view of the sensors.
Method:
Gather the dimensions of the area to track
Gather the max / min height to place the camera
Know what you will track (people, object, finger, hand, quick movement of the object/hand to track …)
The table below is a reminder of the FOV for each kind of camera (more details in 1.3 section).
model
FOV
Resolution - ratio
Astra pro
0.87
640 x 480 - 4:3
Kinect azure
NFOV : 0.652 / WFOV : 0.29
(NFOV) 320x288 10:9 / (WFOV) 512x512 1:1
You can simulate the FOV of the camera in a 3D software. You can use sketchup which is free, Cinema4D or a tool to perform video projection studies like LightAct, Fly Elise Immersive designer, Mapping Matter, Smode etc …
Here is an example of 3D with an orbbec Astra pro camera
Simple use case using cameras
Description : Cameras have zenital point of view (e.g. looking straight down).
When you simulate the coverage of the camera, you have to make sure you see at least above the knees of people over the whole area. The surface covered on the ground is not an important piece of information. The goal is to track what is above the ground, not on it. Thus, if the cameras have a spec at 6m max but you have to place it at 6.5m, it can work. You will detect all objects of at least 50cm high above the ground. This can work if you track humans for example. Note that you will need to place looking straight down to the ground in this case, because they will not be able to compensate for any angle without seeing the floor.
Use case with camera overlap
Description: cameras have zenital point of view
Each camera must be able to see above the knees which is around 60 cm high. For a rough simulation, you can apply 30% of overlap to be sure. Depending on the interactive content, if you need to use the height of the bounding box, you probably need to detect the entire body. You need to place the camera quite high for this (5m approx) or using more nodes for this to cover the entire area.
around 30% overlap
Example overlap above the knee
Top view with quad cameras
Side view detecting the entire body
Use case with camera overlap - low ceiling
Description: cameras have oblic point of view
When you must hang your cameras at a low height (2,5m - 4m), you can tilt or pan the camera in order to cover more area. It works well but you will face a potential occlusion problem when one person stands behind another from the camera’s point of view. Here is an example of how to set it up.
In the following picture, the height of the cameras is 2,5m. We want to project video on the ground. The projected interactive content only needs x-y position (no need for the height info), thus if we can place the camera in order to detect the knees of the people, it’s enough.
The trick to cover a maximum surface is to place one row of cameras tilted in one way and another row tilted in the other way (as in the picture below).
Use case with lidar for wall
Description: choice of lidar
Depending on the angular resolution of the lidar sensor, there is a minimum size of object that can be detected, depending on the distance between the object and the lidar.
The schematic below illustrates the main idea to keep in mind. If the content needs to be very accurate (finger detection 1-2 cm), you can do it at 4m with 0,250° degree/point or at 6m with a 0,125° degree/point. In the first case, if you are at 3m, you will probably be able to properly detect one finger.
Place the LIDAR where people can’t touch it and also avoid occlusions. On the top middle of a wall for example.
Lidar standard = Hokuyo UST-10LX
Lidar high resolution = Hokuyo UST-10LX-H01
Beware of the position of the lidar in height, it must be high enough over the surface to avoid the rays hitting the surface.
EXAMPLE USE CASES
Detecting people from above with two overlapping cameras
This use case represents a setup with two cameras looking at the floor on which people are walking. The floor is a flat white surface of around 40m² on which video projection is used to display visuals that people can interact with.
Note that the interactive area is a bit smaller than the actual field of view of the camera. It's because we want to be able the see at least up to the people’s knees inside the interactive area for an accurate detection as soon as they enter the area. Otherwise the interactive area could extend a bit further but the detection might not work well on the sides.
Interactive wall with lidar
You can create interactive walls using lidars with Augmenta. When using lidar you should always try to keep the sensor out of reach of the people. The best place to put your lidar is in the center of the area.
You could also use multiple lidar to solve occlusion issues if there are a lot of users. Put them at different places (for example, two corners of the wall) avec place one over the other in Fusion, the points belonging to both sources should be fused together and you should also get the point that only one lidar sees.
Touch table with lidar
A touch table is really similar to an interactive wall, except that you might also detect the people outside of the interactive zone.
There are parameters in the node to handle those cases. You can set a low noise threshold that would solve the case shown above.
Apart from the unwanted object rejection, it is the same workflow as an interactive wall.
Touch table with camera
You can also set up interactive tables using cameras.
As always, the camera looking straight down at the interactive surface is the best scenario. However, often in the case of interactive tables you will have to put the camera close to one side. If you can choose, set it up on the opposite side of the users. The ideal position is as low as possible while still seeing the whole table. Once the camera is set up we advise that you mask everything outside of the table (in the input tab of the node interface). Then you will have to set a small depth threshold range (about 5cm, still in the node interface) , just above the table.
Airtouch
A workflow kind of similar to the “touch table with camera” is the airtouch. They are similar in the sense that you will also set up a small range of depth threshold. You can do this two ways:
Setting plane to off in the background tab of your node. You will have to set up the camera perfectly perpendicular to the floor, otherwise you will have a inclined
Setting plane to auto, but you will have to have a wall in the sight of the camera, or to set up a temporary plane perfectly perpendicular to the floor for calibration.
Curved walls
Beta feature
Curved walls can also be turned into interactive surfaces. There are more constraints for this setup than on others. Cameras should be as close as possible to the curved wall. The camera also has to be high enough, depending on your camera’s FOV to provide a high enough interactive surface. If the camera is further from the wall, occlusions will occur between a user and its hands. Also note that this setup needs a more advanced projection setup. The cameras need to see a plane parallel to the floor or the floor itself to be able to calibrate properly.
Tracking people with lidars
It is possible to track people over a space using lidars. This can be useful if the camera can not be set up correctly (ceiling too high, not possible to attach them to a ceiling…). The big disadvantage of this technique is that you will need multiple lidars to avoid occlusions and that lidars are quite pricey.
Nevertheless, to set up such a system you will have to put multiple Lidar around the interactive area. They should be at the people's waist height. You might have to filter the object size quite high to avoid as much false-positive due to arms or legs, that will create additional unwanted points. Also if someone is smaller than what you did set that person will not be detected.
Network features of Augmenta nodes
Annex - Networking crash course
Switch vs router
An important thing to understand is the difference between switches and routers.
The default choice for most users is to use a router. If you cannot find your nodes, make sure that you are using a router (if you do not have a specific reason to use a switch).
Who’s who
If networking is kind of a black magic to you, here’s a hint: Routers, most of the time, have antennas, because they also provide a wifi network. Switches will never have antennas, so if you are unsure, use the thing with antennas.
Note : Sometimes routers will not have antennas, as in the picture above.
Networking crash course
The network example above can summarize the majority of things to know about networking in the context of Augmenta. A switch connects devices on a network. A router connects multiple networks together.
The router A connects the 192.168.1.0/24 network to the Internet (which is a network of networks) through its Internet Service Provider. The router B connects the 192.168.1.0/24 network to the 192.168.2.0/24 network. The whole network could be viewed as a small venue, with an office network (192.168.1.0/24) that contains office and desk devices. The 192.168.2.0/24 could be the performance room network.
We can see that the switches are like power strips, they only connect the devices together. That’s why they do not have an IP address, they only let traffic pass through them.
All the network devices have an IP address.There is two way to configure them :
automatically : the network device will ask the router to give it a network address.
manually : in the settings of your computer, you enter the desired IP address to use.
That’s why by default, you won’t be able to connect to a node by hooking everything to a switch, the node will try to ask the router foran IP address, but there is no router on the network to answer.
If for a long period of time a node cannot get an IP address from a router, it will fallback to a link-local address. This is a random address taken in the 169.254.0.0/16 range. It will then advertise its presence on the network and you should be able to use a zeroconf browser or Augmenta Fusion. This also means that you can connect a computer directly to a node, without a router or a switch. Your computer must have an IP address in the range 169.254.0.0/16, on Windows, when the computer does not detect a router, it is the default option and it also should be on Mac OS.
Conclusions
You should have a better understanding of how a network is set up. If you are still unsure of how to set up a minimal network for Augmenta, follow those instructions:
Use a router, power it.
Connect you computer and the nodes to the router.
If there are not enough network plugs on your router to connect everything, plug a switch to the router, and then plug your remaining network devices to the switch.
Last updated