Database merging is broken after version 2.4.2 - error unsupported database URI provided
0xtavian opened this issue · 2 comments
Describe the bug
Merging databases is broken in version 2.5.0 and version 2.5.1. The last working version was 2.4.2.
To Reproduce
Steps to reproduce the behavior:
# make dirs for dbs
omnom01:~:% mkdir dbs; cd dbs
# create first input file
omnom01:dbs:% cat ../input
https://microsoft.com
https://tesla.com
https://google.com
# create second input file
omnom01:dbs:% cat ../input2
https://aol.com
https://paypal.com
https://verizon.com
# Download gowitness version 2.5.1
omnom01:dbs:% wget -q https://github.com/sensepost/gowitness/releases/download/2.5.1/gowitness-2.5.1-linux-amd64 -O ~/go/bin/gowitness ; chmod +x ~/go/bin/gowitness ; gowitness version
gowitness: 2.5.1
git hash: fa0246e
go version: go1.21.3_linux/amd64
# Create first db
omnom01:dbs:% gowitness file --file ../input --db-location sqlite://db1.sqlite3
01 Dec 2023 23:23:34 INF preflight result statuscode=200 title=Google url=https://google.com
01 Dec 2023 23:23:43 ERR preflight request failed error="Get \"https://tesla.com\": context deadline exceeded"
01 Dec 2023 23:23:43 ERR preflight request failed error="Get \"https://www.microsoft.com/\": context deadline exceeded"
01 Dec 2023 23:23:43 ERR failed to witness url error="Get \"https://tesla.com\": context deadline exceeded" url=https://tesla.com
01 Dec 2023 23:23:43 ERR failed to witness url error="Get \"https://www.microsoft.com/\": context deadline exceeded" url=https://microsoft.com
01 Dec 2023 23:23:43 INF processing complete
# Create second db
omnom01:dbs:% gowitness file --file ../input2 --db-location sqlite://db2.sqlite3
01 Dec 2023 23:23:56 INF preflight result statuscode=200 title="Verizon: Wireless, Internet, TV and Phone Services | Official Site" url=https://verizon.com
01 Dec 2023 23:23:57 INF preflight result statuscode=200 title="Digital Wallets, Money Management, and More | PayPal US" url=https://paypal.com
01 Dec 2023 23:24:00 INF preflight result statuscode=200 title="News, Politics, Sports, Mail & Latest Headlines - AOL.com" url=https://aol.com
01 Dec 2023 23:24:09 INF processing complete
# list files
omnom01:dbs:% ls
db1.sqlite3 db2.sqlite3 screenshots
# attempt to merge dbs and fail
omnom01:dbs:% gowitness merge --input-path .
01 Dec 2023 23:24:53 INF number of dbs to process database-count=2
01 Dec 2023 23:24:53 FTL could not open destination db error="unsupported database URI provided" destination=gowitness-merged.sqlite3
# revert to gowitness 2.4.2
omnom01:dbs:% wget -q https://github.com/sensepost/gowitness/releases/download/2.4.2/gowitness-2.4.2-linux-amd64 -O ~/go/bin/gowitness; chmod +x ~/go/bin/gowitness ; gowitness version
gowitness: 2.4.2
git hash: c9c6b17
go version: go1.19_linux/amd64
# merge is successful with version 2.4.2
omnom01:dbs:% gowitness merge --input-path .
01 Dec 2023 23:25:26 INF number of dbs to process database-count=2
01 Dec 2023 23:25:26 INF writing results to a new database db-path=gowitness-merged.sqlite3
01 Dec 2023 23:25:26 INF processing source database file=db1.sqlite3
01 Dec 2023 23:25:26 INF done processing db processed-rows=1 source-db=db1.sqlite3
01 Dec 2023 23:25:26 INF processing source database file=db2.sqlite3
01 Dec 2023 23:25:26 INF done processing db processed-rows=3 source-db=db2.sqlite3
I also tried with version 2.5.0 with the same result
Expected behavior
Merging should work with versions higher than 2.4.2
Screenshots
Version Information:
- OS: Ubuntu 20.04.6 LTS \n \l
- gowitness: 2.5.1 - merging fails
- gowitness: 2.5.0 - merging fails
- gowitness: 2.4.2 - merging works!
Additional context
Add any other context about the problem here.
@0xtavian Hey, I think I found a workaround:
Taking the error I found it was at: (https://github.com/sensepost/gowitness/blob/6b10eaeba7a6dcaad4b487fc52f114a952e01295/storage/db.go#L87C1-L87C62), Working backwards while learning Go programming I recompiled with some more debugging commands and found that from line 68 calling the parseDBLocation function was the issue, (
Line 30 in 6b10eae
checking what the dbLocation and the location.Scheme parameters show the issue quite well, the dblocation parameter will parse in the filename of the db, the if statement of "sqlite" if obtained from the go URL module and its Parse function,
func parseDBLocation(dbLocation string) (*url.URL, string, error) {
// Parse the DB URI.
location, err := url.Parse(dbLocation)
if err != nil {
return nil, "", err
}
// Ensure the sqlite DB file path is correctly parsed via url.Parse
if location.Scheme == "sqlite" {
switch {
case location.Host == "" && location.Path != "":
return location, location.Path, nil
case location.Host != "" && location.Path == "":
return location, location.Host, nil
case location.Host == "" && location.Path == "":
return location, "gowitness.sqlite3", nil
default:
return location, strings.TrimPrefix(dbLocation, "sqlite://"), nil
}
}
return location, dbLocation, nil
adding in a filename such as blah.sqlite3 the URL module does not give me a scheme back
./gowitness-2.5.1-linux-amd64 merge -i 01.sqlite3 -i 02.sqlite3 -o temp.sqlite3
26 May 2024 11:10:25 INF number of dbs to process database-count=2
DBLocation: temp.sqlite3
Scheme:
Host:
26 May 2024 11:10:25 FTL could not open destination db error="unsupported database URI provided" destination=temp.sqlite3
however if you change the filenames to be sqlite///blah.sqlite3 , the scheme is extracted from url.parse to "sqlite" which satisfies the if statement then then the trim (strings.TrimPrefix(dbLocation, "sqlite://")) command kicks in and removes the prefix.
So really your commands should be
./gowitness-2.5.1-linux-amd64 merge -i sqlite:///path/01.sqlite3 -i sqlite:///path/02.sqlite3 -o sqlite:///path/temp.sqlite3
26 May 2024 11:13:42 INF number of dbs to process database-count=2
DBLocation: sqlite://temp.sqlite3
Scheme: sqlite
Host: temp.sqlite3
26 May 2024 11:13:42 INF writing results to a new database db-path=sqlite://temp.sqlite3
26 May 2024 11:13:42 INF processing source database file=sqlite:///path/01.sqlite3
DBLocation: sqlite:///path/01.sqlite3
Scheme: sqlite
Host:
26 May 2024 11:13:43 INF done processing db processed-rows=46 source-db=sqlite:///path/01.sqlite3
26 May 2024 11:13:43 INF processing source database file=sqlite:///path/02.sqlite3
DBLocation: sqlite:///path/02.sqlite3
Scheme: sqlite
Host:
26 May 2024 11:13:43 INF done processing db processed-rows=37 source-db=sqlite:///path/02.sqlite3
My guess is that a better logic of file identification is required and url.parse isn't fit for dealing with sqlite file checking
Merging has been reworked in v3.