/mapillary-ruler

Building height estimation using neural networks and Mapillary street-view imagery. MATLAB Feed-Forward Neural Network is used for geometric transformation

Primary LanguagePythonMIT LicenseMIT

Building Height Estimation using Mapillary API

This guide explains how to annotate new images for training a network to estimate building heights using Mapillary API data, and how to use trained networks for estimation.

Image Annotation (annotation_images)

  1. Directory setup and access token integration

    Create the following folders in annotation_images:

├── new_images/
└── processed_images/
    ├── .original/
    ├── .ai/
    └── .svg/

And integrate your Mapillary API access token into the code:

  • annotation_images/1_image_data_request.py Line 16
  • script_prog/python_script.py Line 155
  1. Image Search and Data Collection
  • Search for images fitting your criteria and save them and their data using 1_image_data_request.py.
  • The images are saved to new_images folder, data - to 1_new_images.xlsx.
  • Assign project IDs to the images in the first column that correspond to your needs.
  1. Data Preparation
  • Move data from 1_new_images.xlsx to 2_all_images.xlsx and save the data with camera position coordinates to GEOJSON using script 2_excel_to_geojson.py.
  • The GEOJSON reprojected into EPSG:3857 is saved into 2_all_images_3857.geojson.
  1. Image Processing
  • Cut each new image from new_images to proccessed_images/original and open them with Adobe Illustrator or any similar vector image processing tool that allows creating importing JPG, adding circles, and exporting to SVG.
  • In Adobe Illustrator:
    • Open the original JPG and adjust the artboard.
    • Add circles and name them correctly. In my work, the first three numbers stood for the project ID of the image, the next two symbols - for the position of points pair in the image from left to right, and the last one was either 1 for the top point or 2 for bottom point - e.g., 048062.
    • Then, geolocate the projections of points pair on the ground in QGIS. In the project 3_annotation_qgis.qgz, the 2_all_images_3857.geojson is open, as well as 3_ref_points_3857.geojson with points from my project geolocated. Add your points to the 3_ref_points_3857.geojson and name them correctly - e.g., the first five numbers of points names that correspond to the points pair.
    • Save points coordinates from SVG to an Excel file using 3_coordinates_svg_to_excel.py. The coordinates are saved to the worksheet new_coordinates in 3_ref_points.xlsx.
    • Assign to each point pair in the worksheet new_coordinates in 3_ref_points.xlsx the Building height for ground truth data.
  • Examples of images in original format and as SVG can be found in branch examples-building-height.
  1. Lines Creation
  • Create lines in your QGIS project using the tool "Join by Lines (Hub Lines)" from the processing toolbox. Refer to 4_how_to_create_lines.png.
  1. Lines Proccessing
  • Calculate the length and angles of the lines from the lines layer by copying fields from the layer 5_lines_attributes_example in the QGIS project 3_annotation_qgis.qgz or create new virtual fields as in 5_how_to_calculate_lines.png.
  • Export the layer as Excel to 5_lines_data.xlsx.
  1. Data Compilation
  • Prepare data for MATLAB Neural Network in matlab_input_all and filter images as needed.
  • Merge all data into matlab_table.xlsx in the script_prog folder:
    • 2_all_images.xlsx to image_data_all
    • 3_ref_points.xlsx to ref_coordinates
    • 5_lines_data.xlsx to lines
  • You can then filter the images by user as I did with the images of stefanhrt or filter them by the outlying values of building height or distance.

Neural Networks Use or Training (script_prog)

  • Train neural network from the required worksheet of matlab_table.xlsx using script_neural.m. You can adjust the worksheet used, data range, columns used as input, data split (training-validation-testing), and the way to handle the optimization algorthms and hidden layers number.
  • Use matlab_main_script.m to use neural network. In the current code, you can compare two neural networks.
  • The MATLAB program accesses compiled Python program in __pycache__/python_script.cpython-311.pyc. You can change the code in python_script.py and then recompile using compile.py.