Non published references are returned by DataSync and if included via IncludeReferences the uids are mixed up

[idmedia-kaiser]

Hi, we are facing another issue with DataSync which we already mentioned in #20 (comment)

We have a form content type with a reference field for fields
https://app.contentstack.com/#!/stack/blt81b4b787a1850697/content-type/form/de-de/entry/blt0e8af3662cf665b0/edit?branch=main

Some of those references are not published. If we check in the DEV Tool widget we correctly don't get this fields back that are not published.

https://cdn.contentstack.io/v3/content_types/form/entries/blt0e8af3662cf665b0?api_key=blt81b4b787a1850697&access_token=&environment=development&locale=de-de

{
...
  "fields": [
    {
      "uid": "blte52259e1308823f8",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "bltf45bb3e8d8c04dd1",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "bltb673a18f783fa5b6",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "blt9788f975762ad3e0",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "blt4984c9008d23711a",
      "_content_type_uid": "input_field"
    }
  ]
}

If we do the same request towards DataSync we also get the unpublished references back
DataSync Response

{
    "content_type_uid": "form",
    "locale": "de-at",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blta8fed94360b45377", // -> Not published
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blta22b07fa5fc5fce9",  // -> Not published
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "_content_type_uid": "input_field"
                }
            ]
        }
    ]
}

Even worse, if we use the "includeReferences" then the not published entries get mixed up with entries that are actually published and we get duplicated content

[{
    "content_type_uid": "form",
    "locale": "de-at",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1"
                },
                {
                    "uid": "bltb673a18f783fa5b6"  // -> Not published should be "blta8fed94360b45377" or actually not present at all
                },
                {
                    "uid": "blt9788f975762ad3e0"  // -> Not published should be "blta22b07fa5fc5fce9" or actually not present at all
                },
                {
                    "uid": "bltb673a18f783fa5b6"
                },
                {
                    "uid": "blt9788f975762ad3e0"
                },
                {
                    "uid": "blt4984c9008d23711a"
                }
            ]
        }
    ]
}]

Please let us know if you need stack access

[harshithad0703]

Hi @idmedia-kaiser can you please share stack access or may be contenttype json and entry json

[idmedia-kaiser]

@harshithad0703 Can you please provide your email address ? then we can give you stack access.

[harshithad0703]

@harshithad0703 Can you please provide your email address ? then we can give you stack access.

here is my mail id: harshitha.d@contentstack.com

[Chaitrasaurav]

@harshithad0703 Can you please provide your email address ? then we can give you stack access.

here is my mail id: harshitha.d@contentstack.com

I have sent you the invitation

[harshithad0703]

@Chaitrasaurav since it is sso enabled it is asking me to sign-in to sky

[harshithad0703]

Hi @idmedia-kaiser,
I created a similar content type with a reference field that can reference multiple entries, tried to reproduce the issue at our end which was not possible. Here are performed a series of checks:

1. Postman Check (Without includeReferences()):

I fetched an entry via Postman where the reference field contains other entries. The response correctly included only the uid and _content_type_uid of both published and unpublished referenced entries. This is the expected behavior, as unpublished entries should be returned with reference metadata but not full content.

2. Postman Check (With includeReferences()):

I then queried the same entry with includeReferences(). The response correctly included only published referenced entries with its data, and unpublished entries were excluded, as expected.

3. DataSync Check (Without includeReferences()):

I performed the same query using DataSync (without includeReferences()) and received the expected result: both published and unpublished entries were returned with just their uid and _content_type_uid (reference metadata), without any mismatching.

4. DataSync Check (With includeReferences()):

Using includeReferences(), DataSync correctly returned only published entries with their data, excluding unpublished entries from the response.

Please verify the webhook and its trigger condition to ensure it's set up correctly. Let me know if any further information or clarification is needed from my side!

cc: @aman19K @netrajpatel @cs-raj @shafeeqd959

[harshithad0703]

Hi @idmedia-kaiser

Could you please confirm if everything is working as expected on your end based on the checks performed? If the issue is still persisting, kindly provide more details so we can assist you further and close this issue.

[idmedia-kaiser]

@harshithad0703 We have a call today to look into this in more details with your support team

[idmedia-kaiser]

@harshithad0703 While reproducing this issue on localhost, I think the information shared here is not quite accurate. The issue actually does NOT happen if we use includeReferences() directly, but include(["fields"]), as we don't always want to include all references. I would not cancel the call yet, but just show this behavior then. The fields refs then are messed up as explained, which doesn't happen when using includeReferences().

[idmedia-kaiser]

Closed see comment above. includeReferences() is working as expected and we will create a new issue for the include(["fields"]) function not working correctly