Task graph scheduler with a minimalistic API.
dwork builds using cmake. An example build.sh script is included. It requires the following dependencies:
- TKRZW
- Protobuf
- cppzmq
- more information on ZeroMQ at https://zeromq.org/
Most people will be interested in the following workflow:
- run the dwork hub
dhub &
- load up a list of tasks
dquery addfile <file listing 1 task per line>
- start up a pool of workers
see example in tests/kworker
(can also usedquery steal/complete
from shell) - Note: Processor elements that steal a task are responsible for callingcomplete
when it's finished, or else the task's successors will not run. - monitor task progress
bin/dquery status
Steps 1, 2 and 3 can be run in any order as long as 2 comes after 1.
What's happening in the background is that dhub
is listening for client connections on tcp port 6125.
The dquery
tool sends and receives messages from the server to accomplish adding/dequeuing tasks.
All valid messages are enumerated documentation.
The underlying format is in the protobuf file.
Since dwork is a server that speaks protobuf over zmq, you can easily interact with the task server from python.
Python Package Setup:
- pip3 install protobuf pyzmq
Example Script:
import taskMsg
import zmq
# Create a request message
msg = taskMsg.TaskMsg()
msg.type = taskMsg.REQUEST
print("Sending: ")
print(msg)
# Open a connection to the dhub
context = zmq.Context()
dhub = context.socket(zmq.REQ)
dhub.connect("tcp://localhost:5555")
# Send the request. Receive and process the reply.
dhub.send(msg.SerializeToString())
msg.ParseFromString( dhub.recv() )
print("Received: ")
print(msg)
The behavior of the above programs can be altered by command-line arguments. Notably, dhub
accepts a "-f"
argument to store/recover its task database to/from file. dquery
also accepts several options that
are documented by running dquery -h
.
It is our intent to build variants of dquery
that work as (library-level) function calls from various
languages (python being a major one). Basically, this just requires
creating machinery to send/receive protobuf over zmq in those languages.
If you are interested in this project, contact the developers directly!
A C++ variant is essentially done, since it just requires adapting test/kworker
and linking to libdwork
.
The bash variant is dquery
itself.
The following packages are required to build the docs:
- cmake
- doxygen
- python3-sphinx
In addition, you'll need some python packages:
pip3 install sphinx_rtd_theme
pip3 install breathe