The latest version of Scaniverse can resolve small objects and fine details better than ever before. Here’s how.
A Scaniverse model, cut away to reveal the underlying mesh on the right half of the screen. You can view this model on Sketchfab further down on this page.
It’s a true feat of engineering that Apple managed to pack a LiDAR sensor into the latest generation of iPhones and iPads. As with any new technology, however, there are certain areas where the hardware still has room for improvement. One such area is resolution.
Teardowns have shown that Apple’s LiDAR sensor works by projecting a 24x24 grid of infrared dots into the world and measuring the time it takes each of these to bounce off objects and return to the sensor. These timings are converted to distances, then post-processed to produce a depth map.
This is great for large objects and flat surfaces. But there are many things you might want to scan that require a higher level of detail: small objects, plants and flowers, furniture, or architectural elements. Once you start scanning, you realize these are everywhere, and it would be nice to model them more accurately. But how?
Tricks from the graphics world
3D rendering and reconstruction are similar problems — they just operate in opposite directions. 3D rendering takes geometry in world space and projects it into an image. 3D reconstruction takes images and depth maps and projects them back out into the world. When faced with a reconstruction problem, you can often find inspiration from rendering.
One technique we can lean on is antialiasing. When rendering geometry (like the letter “A” in the figure below) with one sample per pixel, each pixel is either inside or outside the geometry, which leads to stair-step artifacts along the edges. The process of eliminating these artifacts is called antialiasing.
Aliased image on the left, antialiased on the right. (by Mwyann — CC BY-SA)
Early antialiasing techniques worked by taking multiple samples for every pixel and averaging them together. However, this could be computationally expensive — if you’re taking 4 samples per pixel, you’re doing 4 times the work.
One way to avoid this is with temporal antialiasing. The key observation is that you’re generally not rendering a single frame in isolation. Instead, you’re rendering a sequence of frames, and you can reuse samples from previous frames by projecting them into the current frame. This gives you the benefit of having multiple samples per pixel but without additional cost because you’ve already computed them in earlier frames.
It turns out we can use a similar trick in reconstruction. We can take samples across multiple frames, project them, and combine them to get a higher effective resolution than we would have from any single frame.
This allows us to do a better job at resolving individual leaves on this plant,
the stem, handlebars, and platform on this electric scooter,
and the legs and slats on this chair.
As you can see, things are still far from perfect. The scooter’s kickstand is missing, and the back of the chair looks pretty rough. But it’s another step along the road to realism.
Tips for scanning fine details
You can try out these improvements in the latest version of Scaniverse. In addition to the general scanning tips on our website, here are a few things that can help for objects with fine detail:
- Reduce the LiDAR range to be about the size of the object that you’re scanning. This prevents Scaniverse from picking up extraneous detail and allows it to devote more resources to the parts of the scene you care about. You can set the range before or during your scan by pressing the Range button.
- Don’t be afraid to get close. You can collect useful data as near as 10 cm (4 inches) away from the object.
- For objects with holes or gaps, like the back of the chair above, try moving your device slowly in circles parallel to the surface. The different viewpoints allow the LiDAR pulses to hit or pass through different parts of the object and allow Scaniverse to build up additional detail over time.
We have many more improvements planned for the coming weeks and months as we continue along the path to realism. Thank you for joining us on this journey!
Scaniverse is available on the App Store for the iPhone 12 Pro, iPhone 12 Pro Max, and 2020 iPad Pro.