A database where the entries can be stored on the next generation persistent memory storage media. This allows for transactions to occur with the same speeds as one would have when using volatile memory (RAM), however, the entries and changes are saved immediately. There is thus no need for saving or loading of the database, and it will persist across multiple, independent sessions.
The code is an implementation of functional linked lists based upon an example by Andy Rudoff, Intel (https://www.usenix.org/system/files/login/articles/login_summer17_07_rudoff.pdf) which can be found as a part of the Persistent Memory Development Kit.
It is here extended to act as a database for job_manager. This includes some ugly, hard coded fields for each list entry:
jobid-- (intended to be unique, rising from 0) integer,job-- Encoded JSON command, up toMAX_JOB_SIZEcharacters (specified in header file),jobstage-- integer determining at what stage a job is,savepath-- character arrray specifying location of saved file,datecommitted-- character array specifying the timestamp of the entry.
The pmem_queue by Rudoff is extended and has the following functions:
push-- adds an individual entry to the list and stores it in the persistent memory databaseset-- sets parameters of an entrycount-- returns the number of entries in the persistent databaseshow-- displays the contents of the databasesearch_all-- searches, sets elements of an array to 0 or 1 (-1 if it fails) at the indicies where the contents of an entry match the conditions. The conditions and operators are specified:***_oper: operation to check (==,>, etc)***_q: value that has to be satisfied- If the operation is a null pointer the field is not checked.
- An extra flag,
only_firstreturns the first index where the conditions match and breaks the loop.
The Python extension can be built either via python3.6 setup.py build_ext --inplace for a version that will reside only source directory, or built and installed to the system with pip install ..
The following functions are available through import pmdb:
init_pmdb(path_to_pmem_file, n_elements)- initializes a persistent database at the given path,
insert(path_to_pmem_file, status, n_elements, job_id, job, jobstage, jobpath, jobdatecommitted)- receives the properties as list entries that are batch added to the persistent database,
get(path_to_pmem_file, status, n_elements, index)- retrieves a list holding the contents of the entry at
index,
- retrieves a list holding the contents of the entry at
set(path_to_pmem_file, status, n_elements, jobid, job=None, jobstage=None, jobpath=None, jobdatecomitted=None)- sets the non-None fields to
jobid(the index),
- sets the non-None fields to
search(path, n_elements, jobid=(,), job=(,), jobstage=(,), jobpath=(,), jobdatecommitted=(,), only_first=False)- returns list of indices where the non-zero criteria, specified as
(operator, value)are satisfied.- The
operatorcan be==,>=,>,<=,<and!=for numeric fields (jobid, jobstage), and==or!=for character fields (job,jobpathorjobdatecommitted). only_first=Falsemeans that a full search will be done on all elements. With this flag set toTrue, the first index will be returned in the output list only.
- The
- returns list of indices where the non-zero criteria, specified as
pmdk-- must be installed, can be run even on both next generation (non-volatile, persistent memory-only) or traditional machines (volatile memory and storage). A Linux kernel can be built for traditional machines where a region of the RAM is reserved for persistent storage (however, only for as long as the computer runs), yielding the speed gains one could expect from this form of storage media.Python 3.6for use of the Python extensions.
The size of the persistent memory file must be specified before generating the database. This value is set in calls to pool<pmem_queue>::create(path_to_pmem_file, "queue", FILE SIZE HERE IN BYTES, create mode).
The maximum length of the character string entries is given in the compiler flag MAX_JOB_SIZE.