artefactual-labs/ops-helpers

Problem: es-reindex.sh fail - /usr/bin/curl: Argument list too long

mamedin opened this issue · 1 comments

On Ubuntu16.04:

Index list before reindexing:
health status index         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   transfers     ClkAfO1lSHKBWUP4hqd_Eg   5   1          0            0      1.2kb          1.2kb
yellow open   transferfiles 5kie5zleTCiV1F5HTOmuxw   5   1          0            0      1.2kb          1.2kb
yellow open   aips          LTWqFAfxSNaIlBIk3z9OZQ   5   1         16            0      1.3mb          1.3mb
yellow open   aipfiles      ngDzBSvNRDq8iWAMkV4_0w   5   1        178            0      3.5mb          3.5mb
Please back up your Elasticsearch indices before proceeding further.
Do you want to continue? [y/N] y
Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...
Deleting transferfiles_new...
Creating aips_new with old mappings...
es-reindex.sh: line 39: /usr/bin/curl: Argument list too long

The curl arguments seem very long:

root@am-XXXX:/tmp# curl -s localhost:9200/aips | jq '.'"${index}"' | del(.settings.index.provided_name, .settings.index.creation_date, .settings.index.uuid, .settings.index.version)' |wc -l
7614
root@am-XXXX:/tmp#  curl -s localhost:9200/aipfiles | jq '.'"${index}"' | del(.settings.index.provided_name, .settings.index.creation_date, .settings.index.uuid, .settings.index.version)' |wc -l
7591
root@am-XXXX:/tmp# curl -s localhost:9200/transfers | jq '.'"${index}"' | del(.settings.index.provided_name, .settings.index.creation_date, .settings.index.uuid, .settings.index.version)' |wc -l
78
root@am-XXXX:/tmp# curl -s localhost:9200/transferfiles | jq '.'"${index}"' | del(.settings.index.provided_name, .settings.index.creation_date, .settings.index.uuid, .settings.index.version)' |wc -l
119

Using a json file fixes the issue:

# Create tmp indices with old mappings
for index in $index_list; do
  echo -e "Creating ${index}_new with old mappings..."
  curl -s ${es_url}/${index} | jq '.'"${index}"' | del(.settings.index.provided_name, .settings.index.creation_date, .settings.index.uuid, .settings.index.version)' > /tmp/mapping_file.json
  curl -s -XPUT ${es_url}/${index}_new \
    -H 'Content-Type: application/json' \
    -d @/tmp/mapping_file.json > /dev/null
done
echo -e "\n"
```