Data Import Handler

An Apache Solr™ package for importing records from database systems into Apache Solr collections.

About

This is a 3rd party package for Apache Solr, aiming at maintaining the Data Import Handler since it will no longer ship with the default Solr download in the future.

The success of this project relies on contributions from the broader community through Pull Requests. See "Issues" sections and help in any way you see fit.

Installing and running

Start Solr (version 8.X master) nodes with -Denable.packages=true

bin/solr -c -Denable.packages=true
Add repository:

bin/solr package add-repo data-import-handler "https://raw.githubusercontent.com/rohitbemax/dataimporthandler/master/repo/"
See available packages:

bin/solr package list-available
Install the package

bin/solr package install data-import-handler
Create a products collection

curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=products&numShards=1"
Deploy package on the collection

bin/solr package deploy data-import-handler -y -collections products

Create the DB configurations file

Create a new file data-config.xml:

<dataConfig>
    <dataSource type="JdbcDataSource" driver="org.mariadb.jdbc.Driver" 
            url="jdbc:mysql://localhost/myproducts" user="root" password="password"/>
    <document>
        <entity name="product" query="select ProductID AS id, Name as name_t from products"></entity>
    </document>
</dataConfig>

Add the configurations and reload the collection

./server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd putfile "/configs/products.AUTOCREATED/data-config.xml" data-config.xml

curl "localhost:8983/solr/admin/collections?action=RELOAD&name=products"
Run data import

curl http://localhost:8983/solr/products/dataimport?command=full-import

curl "http://localhost:8983/solr/products/select?q=*:*"

Contributing

The source code for DIH versions that are compatible with Solr 8.x are in branch_8x branch (branch_9x for Solr 9.x). Please feel free to open issues and/or open pull requests against that branch.

Known Issues

Only MariaDB connector supported out-of-the-box right now. The connectors to be used need to be shipped as part of this package, and currently only MariaDB connector is shipped.
Solr Admin UI's Dataimport tab could be a bit glitchy when used with this package

Attribution

The code in this repository was originally a fork of the original DIH code in Solr project and the original code is (C) Apache Software Foundation.