ETL processing for log/response files.
All instructions assume you're working in one of the following environments:
- Linux; or
- Windows with Cygwin installed.
You'll also need SQLite.
If you're in Windows without Cygwin, you'll have to adjust the instructions accordingly.
Open a terminal in the project root directory.
Go to the database directory:
cd LogFileETL/database
Generate a new database:
make create
Change to the LogFileETL root folder, then run the project:
cd ..
dotnet run
Open another terminal in the project root directory.
Go to the LogFileWatcher directory, purge existing target log files, and run the project:
cd LogFileWatcher
make purge
dotnet run
Open another terminal in the project root directory.
Go to the LogFileWatcher directory, and drop a test file:
cd LogFileWatcher
make drop
A new Splunk-friendly log file will be generated in the logfiles/drop/member/formatted folder; and
The project_root/LogFileETL/database/logetl.db database will have new entries added to its archive table:
sqlite3 logetl.db
Then:
.header on
.mode column
select * from archive;