Housekeeping is Not Working Properly on Single Project

[krapie]

What happened:

Housekeeping is not working properly on single project(default project).

I think this is because listProjectInfos() queries projects on $gt(greater than) option. If there is only one project, the listProjectInfos() function queries for a larger _id, which excludes that single project. This will not fetch any projects, leading to empty candidates list for deactivation.

yorkie/server/backend/database/mongo/client.go

Lines 244 to 261 in 8987b45

	func (c *Client) listProjectInfos(
	ctx context.Context,
	pageSize int,
	housekeepingLastProjectID types.ID,
	) ([]*database.ProjectInfo, error) {
	encodedID, err := encodeID(housekeepingLastProjectID)
	if err != nil {
	return nil, err
	}
	opts := options.Find()
	opts.SetLimit(int64(pageSize))
	cursor, err := c.collection(colProjects).Find(ctx, bson.M{
	"_id": bson.M{
	"$gt": encodedID,
	},
	}, opts)

It seems like memDB is also causing the same problem.

yorkie/server/backend/database/memory/database.go

Lines 228 to 260 in 8987b45

	func (d *DB) listProjectInfos(
	_ context.Context,
	pageSize int,
	housekeepingLastProjectID types.ID,
	) ([]*database.ProjectInfo, error) {
	txn := d.db.Txn(false)
	defer txn.Abort()
	iter, err := txn.LowerBound(
	tblProjects,
	"id",
	housekeepingLastProjectID.String(),
	)
	if err != nil {
	return nil, fmt.Errorf("fetch projects: %w", err)
	}
	var infos []*database.ProjectInfo
	for i := 0; i < pageSize; i++ {
	raw := iter.Next()
	if raw == nil {
	break
	}
	info := raw.(*database.ProjectInfo).DeepCopy()
	if i == 0 && info.ID == housekeepingLastProjectID {
	pageSize++
	continue
	}
	infos = append(infos, info)
	}

MemDB is fetching projects successfully because it uses LowerBound(), but it skips appending project when i == 0 && info.ID == housekeepingLastProjectID.

What you expected to happen:

Housekeeping fetches single project and clients for deactivation.

How to reproduce it (as minimally and precisely as possible):

Simply run server with mongoDB attached and project deactivation threshold(--client-deactivate-threshold) with smaller interval (ex: 10s), and create clients for housekeeping.

Anything else we need to know?:

Environment:

• Operating system: Any
• Browser and version: Any
• Yorkie version (use yorkie version): v0.4.11
• Yorkie JS SDK version: v.0.4.11

[devleejb]

I am interested in this issue. Can I work on it?

[hackerwins]

Sure. If you have any questions, feel free to let me know.

[devleejb]

@hackerwins @krapie
I suspect that this issue occurs when the current HouseKeeping logic is trying to search for a page that aligns exactly with the end of the entire project when it runs. Below is a diagram illustrating the process when the HouseKeeping logic calls FindDeactivateCandidates.

In the second call of HouseKeeping, although there is a project that clearly needs to be deactivated, it fails to find any projects because there are no projects with an ID greater than 4.

To address this, what do you think about modifying the listProjectInfos function to make it circular? I would like to hear your opinions on this.

[devleejb]

func (d *DB) listProjectInfos(
	_ context.Context,
	pageSize int,
	housekeepingLastProjectID types.ID,
) ([]*database.ProjectInfo, error) {
    // Do something...
}

How about renaming this function to listProjectInfosGreaterThan?(If we intend to modify the function implementation to be circular, it seems like we might need a different name.) This function is used only in FindDeactivateCandidates and searches for projects after housekeepingLastProjectID by the specified pageSize. The current function name does not seem to accurately convey this role.

[krapie]

Thank you for clarifying the actual problem of this issue.

Modifying to circular form & clarifying function name seems like a good idea (listProjectInfosGreaterThan() name reminds me of Spring Data JPA's syntax though 😄).