autoinit is broken for `libsql_open_v3`

[hgarrereyn]

Hi, its documented that calling sqlite3_initialize is required before using certain other functions.

However the library also supports "autoinit" whereby, when necessary, sqlite3_initialize is automatically called in the process of executing those functions:

libsql/libsql-sqlite3/src/main.c

Lines 200 to 204 in 268d169

	** This routine must be called to initialize the memory allocation,
	** VFS, and mutex subsystems prior to doing any serious work with
	** SQLite. But as long as you do not compile with SQLITE_OMIT_AUTOINIT
	** this routine will be called automatically by key routines such as
	** sqlite3_open().

One such function is openDatabase which is used by the sqlite3_open* and libsql_open* APIs.

However, there is a potential issue with libsql_open_v3 as it tries to perform memory allocation before invoking openDatabase via a call to make_ref_counted_wal_manager:

libsql/libsql-sqlite3/src/main.c

Lines 3860 to 3874 in 268d169

	int libsql_open_v3(
	const char *filename, /* Database filename (UTF-8) */
	sqlite3 **ppDb, /* OUT: SQLite db handle */
	int flags, /* Flags */
	const char *zVfs, /* Name of VFS module to use, NULL for default */
	libsql_wal_manager wal_manager /* wal_manager implemetation */
	) {
	RefCountedWalManager *wal_manager_rc;
	int rc = make_ref_counted_wal_manager(wal_manager, &wal_manager_rc);
	if (rc) {
	wal_manager.xDestroy(wal_manager.pData);
	return rc;
	}
	return openDatabase(filename, ppDb, (unsigned int)flags, zVfs, wal_manager_rc);
	}

Thus, while libsql_open_v3 intends to automatically initialize, it actually does it too late, leading to a crash.

(found via automated fuzzing)

The following testcase demonstrates the issue:

testcase.cpp

#include <cstdint>
extern "C" {
#include "sqlite3.h"
#include "sqlite3ext.h"
}
int main(){
  libsql_wal_manager wm{};
  wm.bUsesShm = 0;
  // Provide non-NULL stubs for required callbacks
  wm.xOpen = +[](wal_manager_impl*, sqlite3_vfs*, sqlite3_file*, int, long long, const char*, libsql_wal*)->int{return 1; };
  wm.xClose = +[](wal_manager_impl*, wal_impl*, sqlite3*, int, int, unsigned char*)->int{ return 0; };
  wm.xLogDestroy = +[](wal_manager_impl*, sqlite3_vfs*, const char*)->int{ return 0; };
  wm.xLogExists = +[](wal_manager_impl*, sqlite3_vfs*, const char*, int *pResOut)->int{ if(!pResOut) return 1; *pResOut=0; return 0; };
  wm.xDestroy = +[](wal_manager_impl*){};
  wm.pData = nullptr;
  const char *fname = ":memory:";
  sqlite3 *db=nullptr;
  int flags = SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE;
  // This call crashes inside libsql_open_v3 with pc=0x0
  int rc = libsql_open_v3(fname, &db, flags, nullptr, wm);
  (void)rc; (void)db;
  return 0;
}

crash report

==12==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0x7ffd3c874770 sp 0x7ffd3c8746a8 T0)
==12==Hint: pc points to the zero page.
==12==The signal is caused by a READ memory access.
==12==Hint: address points to the zero page.
    #0 0x0  (<unknown module>)
    #1 0x5650461d476e in main /fuzz/workspace/test.cpp:22:12
    #2 0x7faec1d67d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>)