A logger that is based on Python's sqlite3 library. Log entries are stored in a SQLite table and can be accessed with SQL queries. It is inspired by the logging library, but does not stay strictly faithful to the API.
The strength of sqlogging isn't speed. It typically takes a few milliseconds to write a log entry (about 7 ms on my machine). But if that's not a blocker for you, the accessibility and flexibility of analysis it gives is a sheer delight. Skip json parsing and unwieldy pandas syntax. If you speak SQL you can make your log data dance for you.
pip install sqlite-logging
from sqlogging import logging
logger = logging.create_logger(name="test_logger", columns=["iter", "score"])
logger.info({"iter": 0, "score": .4})
logger.info({"iter": 1, "score": .1})
logger.info({"iter": 2, "score": .8})
result = logger.query(f"SELECT SUM(score) FROM {logger.name}")
print("sum of scores:", result[0][0])
logger.delete()
logging.create_logger(name="log", dir_name=".", level="info", columns=["ts", "data"])
For creating a new Logger from scratch. If you try to create a Logger by the same name
as a pre-existing logger you'll get a sqlite3.OperationalError
.
- Parameters
name
(str
) - The name of the Logger. This will be both the name of the name of the table and the name of the sqlite3 database file (<name>.db
).dir_name
(str
) - The directory in which the database file will be stored. If it doesn't already exist, it will be created.level
(str
) - The logging severity level. Must be one of{'debug', 'info', 'warning', 'error', 'critical'}
(case insensitive). Only log messages of equal or higher severity will be processed.columns
(List
ofstr
) - The names of the columns to be created in the sqlite database.
- Return type:
Logger
- Raises:
ValueError
: Iflevel
is not one of the 5 allowed levels.
logging.open_logger(name="log", dir_name=".", level="info")
For re-opening an existing Logger. If you try to open a Logger
that doesn't already exist you'll get a RuntimeError
.
- Parameters: as in
create_logger()
- Return type:
Logger
class logging.Logger(name, dir_name, level="info", columns=None, create=True)
- Parameters: as in
create_logger()
create
(bool
) - Whether a new Logger should be created or an existing one re-opened.
Close the connection to the logger database. Can be reopened later with
logging.open_logger()
.
Close the connection to the database and delete the database file. Remove it from existence.
- Parameters
data
(dict
) - Write (at the specified severity level) a row into the sqlite db. The dictionary contains keys with the name of the column to be written, and values with the data element corresponding to that column. Any columns not included in the dict keys will be populated withNULL
. (These will beNone
when queried and converted to Python.)
Returns a list of all column names.
- Return type:
list
ofstr
Run a SQL query against the logger database. Here's a reference for the particular dialect of SQL. It's mostly standard stuff, but as with all SQL dialects can have some surprises, especially if you use some of the fancier features.
- Parameters:
query_str
(str
)
- Return type:
list
oftuple