assert(not any(type(p) is Page for p in db.pages.values()))

[demonnico]

I'm not sure if it's a logic error for this lib, just try it by sqbrite undelete X.sqlite X_undelete.sqlite. I'll provide my test db file if you'd like the figure out what's wrong with it. 😄
Thanks anyway for your lib.

@mattboyer

Traceback (most recent call last):
  File "/Users/nico/Library/Python/3.6/bin//sqbrite", line 11, in <module>
    sys.exit(main())
  File "/Users/nico/Library/Python/3.6/lib/python/site-packages/sqbrite/sqlite_recover.py", line 810, in main
    subcmd_dispatcher(cli_args)
  File "/Users/nico/Library/Python/3.6/lib/python/site-packages/sqbrite/sqlite_recover.py", line 727, in subcmd_dispatcher
    return subcmd_actions[arg_ns.subcmd](arg_ns)
  File "/Users/nico/Library/Python/3.6/lib/python/site-packages/sqbrite/sqlite_recover.py", line 636, in dump_to_csv
    db = _load_db(args.sqlite_path)
  File "/Users/nico/Library/Python/3.6/lib/python/site-packages/sqbrite/sqlite_recover.py", line 630, in _load_db
    assert(not any(type(p) is Page for p in db.pages.values()))
AssertionError

[mattboyer]

Hi @demonnico!

It's been a while since I've worked on sqbrite. As I recall, by the time this assertion statement is executed, all the values in the pages dict should be instances of types more specific than Page, hence why I look at type(p) instead of isinstance(p).

Clearly, that's not the case when your SQLite DB is being loaded and so, as you suggested, I'd need to have a look at the file to determine what page(s) cause that assertion to break. Can you put the file up on pastebin or something similar?

Cheers,

• Matt Boyer

[demonnico]

Thanks for your reply.
This is the db file which I mentioned above.
😸

[mattboyer]

Thanks for that. I'll have a look in ~12 hours' time.

[mattboyer]

Hello again @demonnico.
I had a look at the file you provided. As I suspected, this is a SQLite database that doesn't include a pointer map, meaning that sqbrite is missing a very valuable source of information to answer these two questions for any given page in the file:
a) What kind of page is it?
and
b) If it is an orphaned page, what table B-tree should it be reparented to for data recovery purposes?

As a human observer, it's pretty easy to look at the bytes in a given page and derive answers from that.

Right now, sqbrite is very conservative and doesn't even attempt to guess - this is a missing piece of functionality that will have to be implemented. I'll try a naive approach and see how that works.

[demonnico]

@mattboyer Sounds must be something interesting, look forward it's done ASAP.
Thanks.

[mattboyer]

Hi again,
Just to let you know: I haven't forgotten about you or this issue. While investigating using the file you made available to me, I realised that there's something else that will have to change to in sqbrite: namely, the mapping between table names and heuristics.

I'd assumed that table names were constant for a given DB used by a given application, but some of the tables in your DB have names that appear to include UUIDs - one approach to overcome that might be to use fuzzy regex matching to map a table name to a heuristic, another might be to simply provide a mechanism through which a user can explicitly tell sqbrite which heuristic they think is appropriate for a given table.

[demonnico]

Really happy to know you're digging the problem.
What you said that's really a tricky issue. Both of approaches you provided above are feasible. I'm not sure if I understand every words you said, because I'm not familiar with SQL binary format, and what I suggested is to tell sqbrite explicitly first if the user know how to find theheuristic you mentioned. But if not, a fuzzy regex matching may required as a support.