Append Features to Layer

QGIS v3 plugin that adds a new Processing algorithm to append/update features from a source vector layer to an existing target vector layer.

License: This plugin is distributed under the GNU GPL v3 license.

➡️ Use cases
🛠️ How does it work
🔎 Where to find the algorithm
📝 Examples
🐍 Using Append Features to Layer in standalone PyQGIS scripts
⚙️ Using Append Features to Layer via QGIS Process
💻 Running Unit Tests Locally

➡️ Use cases

Copy & Paste features:

Think about Append Features to Layer as a Copy & Paste algorithm, which extracts features from a source vector layer and pastes them into an existing target vector layer.

In fact, the algorithm is based on the Paste tool that QGIS offers in its main user interface, enabling you to use it in your Processing workflows.
ETL (Extract, Transform and Load): (See example number 2)

The Append Features to Layer algorithm acts as the 'Load' in an ETL operation. If you need to 'Transform' your features before using the 'Load', QGIS offers the Refactor fields algorithm.

Using both algorithms in a model, you can create complex ETL processes to migrate entire data sets from a data structure into another. Fortunately, you can find such a model after installing this plugin!
Send the output of a Processing algorithm to an existing layer:

Unlike conventional QGIS Processing algorithms (except by in-place ones), Append Features to Layer allows you to store data into an existing target layer instead of into temporary layers.

For instance, if you need to send the buffers of a point layer into an existing polygon layer, you can chain the Buffer and Append Features to Layer algorithms in the Processing modeler, so that the output from the buffer gets copied into your polygon layer.
Update existing features in an existing (target) layer based on a source layer.

The Append Features to Layer algorithm can search for duplicates while copying features from source to target layers. If duplicates are found, the algorithm can update the existing feature's geometry/attributes based on the source feature, instead of appending it. In other words, the algorithm performs an Upsert (Update or Insert a feature). You can find more details in the next section of this document.
Update existing geometries in a (target) layer based on geometries from a source layer. 🆕

The Append Features to Layer algorithm can search for duplicates while copying features from source to target layers. If duplicates are found, the algorithm can update the existing feature's geometry (leaving all feature attributes intact) based on the source feature's geometry, instead of appending it. In other words, the algorithm performs an Upsert (Update a geometry or Insert a feature). You can find more details in the next section of this document.

🛠️ How does it work

Fields and geometries

Field mapping between source and target layers is handled automatically. Fields that are in both layers are copied. Fields that are only found in source are not copied to target layer.

Geometry conversion is done on the fly, if required by the target layer. For instance, single-part geometries are converted to multi-part if target layer handles multi-geometries; polygons are converted to lines if target layer stores lines; among others.

How the algorithm deals with duplicates

This algorithm allows you to choose a field in source and target layers to compare and detect duplicates. It has 4 modes of operation:

APPEND new feature, regardless of duplicates.
SKIP new feature if duplicate is found.
UPDATE EXISTING FEATURE in target layer with attributes (including geometry) from the feature in the source layer.
ONLY UPDATE EXISTING FEATURE's GEOMETRY in target layer (leaving its attributes intact) based on the feature's geometry in the source layer.

Note on Primary Keys

The algorithm deals with target layer's Primary Keys in this way:

PRIMARY KEY	APPEND mode	UPDATE mode
Automatic PK (e.g., serial)	It lets the provider (e.g., PostgreSQL, GeoPackage, etc.) fill the value automatically	It doesn't modify the value already stored
Non-automatic PK	You need to provide a value for the PK in the source layer, because such value wil be set in the target layer's PK	It doesn't modify the value already stored

Note on geometry updates

Mode UPDATE EXISTING FEATURE:

If target layer has geometries but input layer does not, then only attributes will be updated when a duplicate feature is found, i.e., the geometry in target layer will remain untouched.

🔎 Where to find the algorithm

Once installed and activated, this plugin adds a new provider (ETL_LOAD) to QGIS Processing. You can find the Append Features to Layer algorithm in the Processing Toolbox, under ETL_LOAD -> Vector table -> Append features to layer.

📝 Examples

Copy & Paste features
ETL (Extract, Transform and Load):

This plugin also installs 2 Processing models that chain Refactor Fields and Append features to layer algorithms. You can find the models under Models --> ETL_LOAD.

The models allow you to

i) Transform and Load (APPEND) all features from the source layer, or

ii) Transform and Load (UPDATE) all features from the source, updating those features that are duplicate.

🐍 Using Append Features to Layer in standalone PyQGIS scripts

Chances are you'd like to run this plugin from a PyQGIS script that works out of a QGIS GUI session. For that, take "Using QGIS Processing algorithms from PyQGIS standalone scripts" into account, as well as the following code snippet.

# Add the path to QGIS plugins so that you can import AppendFeaturesToLayer
# See paths in https://gis.stackexchange.com/questions/274311/274312#274312
sys.path.append(path_to_qgis_plugins)

from AppendFeaturesToLayer.processing.etl_load_provider import ETLLoadAlgorithmProvider

# Register the processing provider
provider = ETLLoadAlgorithmProvider()
qgis.core.QgsApplication.processingRegistry().addProvider(provider)

# Finally, enjoy!
result = processing.run("etl_load:appendfeaturestolayer",
                        {'SOURCE_LAYER': source_layer,
                         'SOURCE_FIELD': '',
                         'TARGET_LAYER': target_layer,
                         'TARGET_FIELD': '',
                         'ACTION_ON_DUPLICATE': 0})  # NO_ACTION: 0
                                                     # SKIP_FEATURE: 1
                                                     # UPDATE_EXISTING_FEATURE: 2
                                                     # ONLY_UPDATE_EXISTING_FEATURES_GEOMETRY: 3

The algorithm outputs are:

TARGET_LAYER
APPENDED_COUNT
SKIPPED_COUNT
UPDATED_FEATURE_COUNT
UPDATED_ONLY_GEOMETRY_COUNT 🆕

⚙️ Using Append Features to Layer via QGIS Process

If you'd like to run the plugin without GUI, but don't want to deal with PyQGIS scripts, you can use QGIS Process. You run QGIS Process from the operating system's terminal, in this way:

$ qgis_process run "etl_load:appendfeaturestolayer" -- SOURCE_LAYER=/tmp/source.shp TARGET_LAYER=/tmp/target.shp ACTION_ON_DUPLICATE=0

Where NO_ACTION: 0, SKIP_FEATURE: 1, UPDATE_EXISTING_FEATURE: 2, ONLY_UPDATE_EXISTING_FEATURES_GEOMETRY: 3

Make sure the plugin can be found in your QGIS plugins folder, that is, that you have installed the plugin in your QGIS.

💻 Running Unit Tests Locally

First, you need to set 2 environment variables:

export GITHUB_WORKSPACE=/path/to/AppendFeaturesToLayer/
export QGIS_TEST_VERSION="final-3_28_13"  # "final-3_34_1" for next LTR

After that, you could run unit tests locally with this command:

docker-compose -f .docker/docker-compose.yml run --rm qgis

You could rebuild the Docker image in this way:

docker-compose -f .docker/docker-compose.yml down --rmi local && docker-compose -f .docker/docker-compose.yml build

gacarrillor/AppendFeaturesToLayer