yugabyte/yugabyte-db

[DocDB] Lock inversions in the Catalog manager, CM::mutex_ and object COW locks

Opened this issue · 0 comments

druzac commented

Jira Link: DB-11972

Description

We have code in the CM which acquires the CM lock first and then table locks:

LockGuard lock(mutex_);
table = VERIFY_RESULT(FindTableByIdUnlocked(req->table_id()));
write_lock = table->LockForWrite();

And we have code that acquires a table lock and then the CM lock:

auto l = table->LockForRead();
if (req->include_inactive()) {
// Do not return the schema of a deleted tablet even if include_inactive is set to true
SCHECK_EC_FORMAT(
l->is_running(), NotFound, MasterError(MasterErrorPB::OBJECT_NOT_FOUND),
"The object '$0.$1' is not running", l->namespace_id(), l->name());
} else {
RETURN_NOT_OK(CatalogManagerUtil::CheckIfTableDeletedOrNotVisibleToClient(l, resp));
}
if (get_fully_applied_indexes && l->pb.has_fully_applied_schema()) {
// An AlterTable is in progress; fully_applied_schema is the last
// schema that has reached every TS.
DCHECK(l->pb.state() == SysTablesEntryPB::ALTERING);
resp->mutable_schema()->CopyFrom(l->pb.fully_applied_schema());
} else {
// Case 1: There's no AlterTable, the regular schema is "fully applied".
// Case 2: get_fully_applied_indexes == false (for YCQL). Always return the latest schema.
resp->mutable_schema()->CopyFrom(l->pb.schema());
}
// Due to pgschema_name being added after 2.13, older YSQL tables may not have this field.
// So backfill pgschema_name for older YSQL tables. Skip for some special cases.
if (l->table_type() == TableType::PGSQL_TABLE_TYPE && resp->schema().pgschema_name().empty() &&
!table->is_system() && !table->IsSequencesSystemTable() &&
!table->IsColocationParentTable()) {
SharedLock lock(mutex_);

We don't seem to have an official locking order for the CM lock and the COW object locks. We should determine what the locking order should be and audit the CM code for lock inversions.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.