In this project, it was studied to develop a deep-learning based water-tight surface fitting algorithm.The purpose of the algorithm is to fit a mesh system for the point cloud data in the local environment. GPU optimization is deemed necessary for online operations on the tablet.
The scientific and socioeconomic effect of the project is that since the process of fitting a water-tightsurface to the point cloud is a much-needed operation in AR/VR and CAD applications, this project provides an alternative.
You can see the simple flow diagram below:
Dataset needed for this project is prepared by me. To train the deep learning model, input and output sets were needed, hence I implemented a python script that handles ray-casting for point cloud data extraction from different 3D models and then applies Delaunay triangulation to generate surface fitted outputs.
The input is 400 adjacent-looking points on camera view that are chosen from the point cloud.
The output is normally an 400x400 adjacency matrix. Since this matrix is symmetric, to increase memory efficiency the upper half of the matrix is taken and converted into an one dimensional vector of 400 binary (0, 1) elements. In this vector “0” represents that there is no edge between two vertices and “1” represents that there is an edge between two vertices.
One of the targets of the project was to do the surface fitting through a deep learning model. Considering the testing environment that is aimed, on a tablet, I decided to use MobileNetV2 as deep learning model.
As an input this model expects a 32x32 matrix with 3 channels each. The inputs that I generated were 20x20. Due to this, I added an extra of 12 rows and columns for each of the inputs that I gave to this model. Additionally, the channels are given as (x, y, z) coordinates of the points.
| 1 | 2 | 3 |
|---|---|---|
 |  |  |
The pictures above show the meshed output of the model. The highlighted square respresents the 20x20 point cloud portion. After fitting faces to the returned edge vector, the result is the mesh. As you can see the depth of the model is preserved since the points have 3D coordinates even if the triangulation is done in 2D.
Testing could not be done due to hardware reachability. If I can acquire a latest Android or IOS tablet, I would like to develop an application for my model and test in on tablet.
For now the model does not work if there is less than 400 points in the given frame. To solve this problem I thought of a boolean channel addition to my input. Although I did not have enough time to implement this feature and embed it into my dataset.
In the future I would like to implement and test this solution.