danielberkompas/elasticsearch-elixir

Let hot_swap work also when there are documents with errors

silviurosu opened this issue · 0 comments

There may be documents that do not pass ES validation. For example self intersecting polygons.
This leads hot_swap failying since every step needs to pass successfully:

with :ok <- create_from_file(config, name, settings_file),
         :ok <- Bulk.upload(config, name, index_config),
         :ok <- __MODULE__.alias(config, name, alias),
         :ok <- clean_starting_with(config, alias, 2),
         :ok <- refresh(config, name) do
      :ok

I can not guarantee that all the documents will always be valid but I would want to reindex the others and inform at the end that some documents could not be inserted.
I had to rewrite bulk upload in my code to pass this limitation:

with :ok <- Index.create_from_file(config, name, settings_file),
         bulk_upload(config, name, index_config),
         :ok <- Index.alias(config, name, alias),
         :ok <- Index.clean_starting_with(config, alias, 2),
         :ok <- Index.refresh(config, name) do
      :ok
.....
defp bulk_upload(config, name, index_config) do
    case Bulk.upload(config, name, index_config) do
      :ok ->
        :ok

      {:error, errors} = err ->
        Bugsnag.report(
          ElasticsearchError.exception("Errors encountered indexing restaurants"),
          severity: "warn",
          metadata: %{errors: errors}
        )

        err
    end
  end

My question is can we make this bulk upload step not failing in case there are some errors?