Query on GSI returns zero results

Question

Query on GSI returns zero results

rocketana opened this issue 2 years ago · 8 comments

Describe the bug
When I query using a secondary index, I get zero results while doing it directly in the AWS UI returns them just fine. No problems with querying on primary index.

ElectroDB Version
2.4.1

Entity/Service Definitions

export const EntryEntity = new Entity(
    {
        model: {
            version: "1",
            entity: "Entry",
            service: "blog",
        },
        attributes: {
            id: {
                type: "string",
                required: true,
                readOnly: true,
            },
            userId: {
                type: "string",
                required: true,
                readOnly: true,
            },
            title: {
                type: "string",
            }
        },
        indexes: {
            primary: {
                pk: {
                    field: "PK",
                    composite: ["id"],
                    template: "E_${id}",
                    casing: "none"
                },
                sk: {
                    field: "SK",
                    composite: [],
                    template: "METADATA",
                    casing: "none"
                },
            },
            userEntries: {
                index: "userEntries",
                pk: {
                    field: "GSI1PK",
                    composite: ["userId"],
                    template: "U_${userId}",
                    casing: "none"
                },
                sk: {
                    field: "GSI1SK",
                    composite: ["id"],
                    template: "E_${id}#",
                    casing: "none"
                },
            }
        },
    },
    Dynamo.Configuration
);

export async function list(userId: string) {
    const result = await EntryEntity.query.userEntries({ userId }).go();
    return result.data;
}

builder.queryFields((t) => ({
    entries: t.field({
        type: [EntryType],
        args: {
            userId: t.arg.string({ required: true })
        },
        resolve: async (_, args) => {
            const result = await BlogResolvers.list(args.userId);
            console.log('resolver result', result);
            return result;
        },
    }),
}));

Answer 1 · 2023-03-03T19:05:41.000Z

I put your query into the playground here and the params it output are the following:

{
    "KeyConditionExpression": "#pk = :pk and begins_with(#sk1, :sk1)",
    "TableName": "your_table_name",
    "ExpressionAttributeNames": {
        "#pk": "GSI1PK",
        "#sk1": "GSI1SK"
    },
    "ExpressionAttributeValues": {
        ":pk": "U_2MVMkJOT9VHRuqF6LxynvC9Qicx",
        ":sk1": "E_"
    },
    "IndexName": "userEntries"
}

Are you able to confirm that you get results using those exact params with your document client? Additionally, do the items you find in the aws console contain the columns __edb_e__ and __edb_v__?

Answer 2 · 2023-03-03T22:05:17.000Z

Additionally, do the items you find in the aws console contain the columns edb_e and edb_v?

To expand on this question, if those properties do not exist on your items, can you try using execution option ignoreOwnership: true. Here is an example of how you would use it in your case:

export async function list(userId: string) {
    const result = await EntryEntity.query.userEntries({ userId }).go({ ignoreOwnership: true });
    return result.data;
}

I also merged a recent PR that impacts the implementation of this option, and possibly how your query will be evaluated. It was released as 2.4.2, which I recommend you upgrade to prior to trying the above example.

Answer 3 · 2023-03-04T01:46:43.000Z

do the items you find in the aws console contain the columns edb_e and edb_v?

Yes, the items have them, this is what I get when I preview an item from the database in the SST console:

{
  "GSI2PK": "E_2MVO30D9JkR3FvxGzmBiYGkoW16",
  "GSI1PK": "U_2MVMkJ0T9VHRuqF6LxynvC9QIcx",
  "__edb_e__": "Entry",
  "__edb_v__": "1",
  "userId": "2MVMkJ0T9VHRuqF6LxynvC9QIcx",
  "SK": "METADATA",
  "GSI1SK": "E_2MVO30D9JkR3FvxGzmBiYGkoW16#",
  "id": "2MVO30D9JkR3FvxGzmBiYGkoW16",
  "PK": "E_2MVO30D9JkR3FvxGzmBiYGkoW16",
  "GSI2SK": "#METADATA",
  "title": "Title"
}

And here's the AWS UI:

Adding ignoreOwnership: true solved the problem (without updating the version), now I'm getting the list of expected results (I see them printed in the console).

result { data: [ { title: 'Title' }, { title: 'Title' } ], cursor: null }

I got another problem Cannot return null for non-nullable field Entry.id. For some reason I don't get the id's though asking for them in the query, but I need to investigate it, it's probably something wrong with my design.

I also updated the version to 2.4.2and tried with and without ignoreOwnership: true - same, the empty results without ignoreOwnership and the expected results in the console with ignoreOwnership.

I tried to find more info about the ignoreOwnership option, but I couldn't. I found a mention about it in the release notes here, but the [read more] link opens a page without any mention of the option. Searching via Google also didn't work. I finally found some info about it in Core Concepts searching in this repository.

What is the problem with the ownership and how can I solve it to avoid using this option? Something wrong with my design? The Core Concepts page says that ElectroDB leaves some meta-data on items to help ensure data queried and returned from DynamoDB does not leak between entities. But I have edb_e and edb_v columns and I query the same entities of the same version that I created via ElectroDb. There should be something else?

P.S. Thank you for creating the playground, I'll check how to work with it, I'm new to ElectroDB and still learning it.

Answer 4 · 2023-03-05T19:46:57.000Z

So I just created this little debug file, but didn't experience the same behavior as you. I got a record back without using ignoreOwnership. The one thing I changed was the key fields and index names because I had a local table already created using different names: https://gist.github.com/tywalch/acd9ce3e39dc15175270cbf3b4a47848

The idea behind ignoreOwnership is to escape some of Electro's safety mechanisms that prevent items from other entities being returned in the results. Leaking entities can be a big hassle when implementing Single Table Design, so Electro does the work to filter return results: sometimes with actual dynamodb filters, sometimes in code. It does this using the attributes, __edb_e__ and __edb_v__. These are also used with Collections to ensure the correct entity instance is used for post processing.

I suggested using it because one of the big reasons why folks use the template functionality is use Electro with an existing table (already loaded with data). In the case items were created outside of Electro, they won't have these attributes, and filtering Electro does would end up filtering out valid records. Was any of your data in your table created outside of ElectroDB?

Lastly, if you are working on something green-field, and are not using template to match an existing schema, I suggest using the defaults without using template because they are robust and do a lot of heavy lifting for you.

Answer 5 · 2023-03-06T18:29:51.000Z

I created the items which I try to query back using create function on EntryEntity and DynamoDBClient from AWS SDK v3. Can some of those be the reason of the problem? In your example you used put on DocumentClient from AWS SDK v2.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { EntityConfiguration } from "electrodb";
import { Table } from "sst/node/table";

export const Client = new DynamoDBClient({});

export const Configuration: EntityConfiguration = {
  table: (Table as any).main.tableName,
  client: Client,
};

export async function create(userId: string, title?: string) {
    const result = await EntryEntity.create({
        id: KSUID.randomSync().string,
        userId,
        title: title,
    }).go();
    return result.data;
}

The following query on the primary index works without any problem.

export async function get(id: string) {
    const result = await EntryEntity.query.primary({ id }).go();
    // console.log('get() result', result);
    return result.data;
}

I also tested the same query on the primary index for multiple entries - same, no problem. This is how I assumed that only querying on GSI is problematic. But from your example I got new ideas for the reason. Could you please try with SDK v3 and see if it'll work for you?

Answer 6 · 2023-03-07T04:58:05.000Z

I discovered that my GSI didn't project the fields I required in the query, so it was the reason why I was getting Cannot return null for non-nullable field Entry.id error. It would be great to have a check for the projected fields and have an error message about it. I discovered it while making queries directly via the AWS SDK.

Answer 7 · 2023-03-07T16:51:27.000Z

I discovered that my GSI didn't project the fields I required in the query, so it was the reason why I was getting Cannot return null for non-nullable field Entry.id error. It would be great to have a check for the projected fields and have an error message about it. I discovered it while making queries directly via the AWS SDK.

Nice, glad you were able to find where the breakdown was 👍

Does this wrap up your need with this issue?

Answer 8 · 2023-03-23T10:31:02.000Z

I still have problems getting no data when not using ignoreOwnership: true, but I think this issue can be closed as basically the it is solved even though in not perfect way. Thank you for your help!