Create missing locales on import
johanrex opened this issue · 12 comments
Ideally I'd like to export one space and import to another. Currently this results in a lot of these errors if the spaces don't have the same locales configured:
"can not find fallback locale within space locales"
If you have a fast moving space with lots of new locales in an expansive phase it gets tedious to manually create the locales and keep them in sync over several spaces. In the README.md it is mentioned as a known limitation of the tool that it does not create missing locales:
"This tool is expecting the target space to have the same default locale as your previously exported space."
Please add functionality so that the import can add missing locales to the import space.
Hello Johan,
the import tool assumes the same content model structure for the source and destination space. The import/export tooling is not made for structure migration. To apply migrations to your structure, we created the migration-cli.
This same structure is especially relevant for locales, since the default locale is permanent and it is not possible to change it via our management api (See https://www.contentful.com/developers/docs/references/content-management-api/#/reference/locales/locale)
So what can you do?
When creating a space, you can set the default locale for the space. That way you can make sure your destination space uses the same default locale as your source space (https://www.contentful.com/developers/docs/references/content-management-api/#/reference/spaces/spaces-collection/create-a-space/console)
As soon the source and destination space share the same default locale, the import tool will be able to create the same locale structure on the destination space as in the source space. The relevant code can be found here: https://github.com/contentful/contentful-batch-libs/blob/master/lib/push/push-to-space.js#L84-L91
If you followed this and still have issues creating/updating locales with the import tool, this is a bug and I'd like to ask you to give more details to reproduce this bug.
I hope that helps,
Benedikt
Hello Benedikt.
From your description above I gather that as long as the fallback locale is the same the missing locales in the destination should be created. Was that perhaps the wrong conclusion?
Both import and export spaces use "U.S. English (en-US)" as default locale. Yet on import I see the error "can not find fallback locale within space locales". Is this expected behavior?
From your description above I gather that as long as the fallback locale is the same the missing locales in the destination should be created. Was that perhaps the wrong conclusion?
Yes thats correct, they should be created.
Now I guess I got the issue. It might be the case, that the tool tries to create a locale with a fallback locale, which does not exist at that point of time.
Can you try reordering the locales in your json file in a way, that locales that are used as fallback, come first, the others later on?
The tool iterates the array in the json in the given order to create entities.
If this fixes your issue, we can automate this in the tool :)
Benedikt.
I only export the json with the export tool and import it with the import tool. Is there a way to tell the export tool to use a specific sort order?
Writing code to change the order myself would require development time that I unfortunately don't have at this point.
Thanks.
No there is not such an option.
I'd just like you to try it once manually so we can verify this is the actual issue. If this is the case, we could transform this into a bug ticket and we will act on it. No coding involved on your side :)
Best,
Benedikt
What I mean is that the export file is ~130000 lines of json. It would take a lot of work to change the order manually and I'm not familiar enough with the format to construct a smaller test case.
The file has a very simple structure, see here: https://github.com/contentful/contentful-export#exported-data
You would just need to reorder the locales, nothing else. So only a few of the ~130000 lines would be affected.
I can reproduce my assumption what the cause for your issue might be, but I can't give you a timeline when I will find time for this.
Best,
Benedikt
Here's the locales from the export. As far as I can tell everything looks fine.
],
"locales": [
],
{
"name": "U.S. English",
"code": "en-US",
"fallbackCode": null,
"default": true,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": false,
"sys": {
"type": "Locale",
"id": "6MhZFelR6ChcB3MBxofzf0",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-07-10T14:49:16Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-07-10T14:49:16Z"
}
},
{
"name": "Swedish",
"code": "sv",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "7v9cVa08oLztXovNWts6Ly",
"version": 1,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-07-25T09:17:10Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-11-09T13:17:52Z"
}
},
{
"name": "Spanish (Colombia)",
"code": "es-CO",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "7MWkEl8DUejnYeK90gSAJ6",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-08-10T12:47:32Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-08-10T12:47:32Z"
}
},
{
"name": "Finnish",
"code": "fi",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "0mBUzTbO5FMm2gdCCzEtb8",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-08-10T12:47:52Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-08-10T12:47:52Z"
}
},
{
"name": "Norwegian Bokmål (Norway)",
"code": "nb-NO",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "1kLWsbVNduoqnqglRUvOia",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-08-10T12:48:45Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-08-10T12:48:45Z"
}
},
{
"name": "Polish",
"code": "pl",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "1CJxidNb2iQ6ueZ0KmdM4S",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-08-10T12:49:01Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-08-10T12:49:01Z"
}
},
{
"name": "Swedish (Sweden)",
"code": "sv-SE",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "3ygWVoZcmlpZIWbTEjQacI",
"version": 1,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:14:02Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "5Fkr9nK4nyHQvg03nVA59k"
}
},
"updatedAt": "2017-11-29T02:15:56Z"
}
},
{
"name": "English (Sweden)",
"code": "en-SE",
"fallbackCode": "sv-SE",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "6E1yVH5JKvcSNhr7w4A6Ka",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:16:53Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"updatedAt": "2017-10-24T06:16:53Z"
}
},
{
"name": "English (Norway)",
"code": "en-NO",
"fallbackCode": "nb-NO",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "3hwhLFscQtyB0mTMnC1mhU",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:20:57Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"updatedAt": "2017-10-24T06:20:57Z"
}
},
{
"name": "Finnish (Finland)",
"code": "fi-FI",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "7qJfHoIF80Nh4sb5MuSEtg",
"version": 1,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:24:46Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "75aype09gR5ULlPxUJFjOS"
}
},
"updatedAt": "2018-01-10T12:12:31Z"
}
},
{
"name": "English (Finland)",
"code": "en-FI",
"fallbackCode": "fi-FI",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "1ay6oSBr7gryXUQ5UscOvy",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:26:10Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"updatedAt": "2017-10-24T06:26:10Z"
}
},
{
"name": "Polish (Poland)",
"code": "pl-PL",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "4EBkdhYopaPq0HGYyUWz6S",
"version": 1,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:29:22Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "75aype09gR5ULlPxUJFjOS"
}
},
"updatedAt": "2018-01-10T12:12:50Z"
}
},
{
"name": "English (Poland)",
"code": "en-PL",
"fallbackCode": "pl-PL",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "5lXIBmi2r0Vb249KaK0skA",
"version": 0,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"createdAt": "2017-10-24T06:30:01Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "0tC34GioF2pRAB0KOAHunT"
}
},
"updatedAt": "2017-10-24T06:30:01Z"
}
},
{
"name": "Thai (Thailand)",
"code": "th-TH",
"fallbackCode": "en-US",
"default": false,
"contentManagementApi": true,
"contentDeliveryApi": true,
"optional": true,
"sys": {
"type": "Locale",
"id": "6M5BpHEq78nYY9eLyKIxX0",
"version": 1,
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "hwcfqdf25vht"
}
},
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"createdAt": "2017-11-14T08:36:44Z",
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "3poNzpDX10lvHYqwwwUlD6"
}
},
"updatedAt": "2017-11-14T08:43:27Z"
}
}
],
Alright, thanks. I will have a look and come back to you :)
I found the issue, it looks like the concurrency of the creation process causes the issue here. We are planning to rework that process soon. I will make sure that this will be fixed when that refactoring happens.
Best,
Benedikt
I think I have opened an issue related to this in the contentful-cli repo: contentful/contentful-cli#335