Let hot_swap work also when there are documents with errors
silviurosu opened this issue · 0 comments
silviurosu commented
There may be documents that do not pass ES validation. For example self intersecting polygons.
This leads hot_swap failying since every step needs to pass successfully:
with :ok <- create_from_file(config, name, settings_file),
:ok <- Bulk.upload(config, name, index_config),
:ok <- __MODULE__.alias(config, name, alias),
:ok <- clean_starting_with(config, alias, 2),
:ok <- refresh(config, name) do
:ok
I can not guarantee that all the documents will always be valid but I would want to reindex the others and inform at the end that some documents could not be inserted.
I had to rewrite bulk upload in my code to pass this limitation:
with :ok <- Index.create_from_file(config, name, settings_file),
bulk_upload(config, name, index_config),
:ok <- Index.alias(config, name, alias),
:ok <- Index.clean_starting_with(config, alias, 2),
:ok <- Index.refresh(config, name) do
:ok
.....
defp bulk_upload(config, name, index_config) do
case Bulk.upload(config, name, index_config) do
:ok ->
:ok
{:error, errors} = err ->
Bugsnag.report(
ElasticsearchError.exception("Errors encountered indexing restaurants"),
severity: "warn",
metadata: %{errors: errors}
)
err
end
end
My question is can we make this bulk upload step not failing in case there are some errors?