This is simple Java based solr sink kafka connector, that takes plain json data from kafka topic and push to solr, both solr cloud and standalone mode supported.
Note: Only JSON data is supported and keep schemas.enable=false for value converter.
| Config Name | Description | Config Value | Is Mandatory? |
|---|---|---|---|
| topics | topic to listen to | topic name | Yes |
| solr.collection | Solr Collection name where topic data needs to be pushed | Collection Name | Yes |
| solr.mode | Mode on which solr is running, pass solr node url | CLOUD or STANDALONE | Yes |
| solr.url | If solr.mode is CLOUD, pass comma seperated zookeeper url else pass standalone solr server url |
url | Yes |
| connector.class | Connector class name | com.bkatwal.kafkaproject.SolrSinkConnector | Yes |
| commit.within.ms | commit within ms value for solr update, if none passes defaults to 10 ms | int value | No |
Refer/Use configuration: config files
- Json Data
- One level of child document update supported. Just pass, additional field,
_childDocuments_with the parent doc. - Deleting a document is supported: To delete pass additional field
_delete_with valuetruein your record(no need to maintain this field in solr/solr schema file), this field will be removed before indexing data to solr. Based on boolean values(true/false) in_delete_, delete/insert operation is triggered - Dynamic fields update. Pass a Map field in json doc. The field name will be used as prefix, so set dynamic field in managed_schema accordingly. Example:
{
"field1": "doc1",
"id":"1",
"field2":"dome val,
"dynamicField":
{
"df1":"val1",
"df2":"val2",
"df3":"val3"
}
}
This will be transformed in solr as:
dynamicField_df1 : val1, dynamicField_df2 : val2, dynamicField_df3 : val3
You can have any field name for dynamic field, just have this defined in managed_schema. So, for above to work, dynamic field meeds to be defined in managed_schema file as below:
<dynamicField name="dynamicField_*" type="string" indexed="true" stored="true"/>
- build with: mvn clean package
- In target look for directory bkatwal-kafka-connect-solr-sink-< version >. Copy this directory to plugins path.