lthread

Introduction

lthread is a multicore/multithread coroutine library written in C. It uses Sam Rushing's _swap function to swap lthreads. What's special about lthread is that it allows you to make blocking calls and expensive computations inside a coroutine, providing you with the advantages of coroutines and pthreads. See the http server example below.

lthreads are created in userspace and don't require kernel intervention, they are light weight and ideal for socket programming. lthread scheduler uses a shared stack for all running coroutines to save space, allowing you to create thousands(tested with a million lthreads) of coroutines and maintain a low memory footprint. The scheduler is hidden from the user and is created automagically in each pthread, allowing the user to take advantage of cpu cores and distribute the load by creating several pthreads, each running it's own lthread scheduler and handling its own share of coroutines. Locks are necessary when accessing global variables from lthreads running in different pthreads, and lthreads must not block on pthread condition variables as this will block the whole lthread scheduler in the pthread.

To run an lthread scheduler in each pthread, launch a pthread and create lthreads using lthread_create() followed by lthread_run() in each pthread.

lthread scheduler's inner works

The lthread scheduler has a main stack that it uses to execute/resume lthreads on, and before it yields, the scheduler saves the current stack state + registers into the lthread and copies it back on the scheduler stack when it resumes.

The scheduler is build around epoll/kqueue and uses an rbtree to track which lthreads needs to run next.

If you need to execute an expensive computation or make a blocking call inside an lthread, you can surround the block of code with lthread_compute_begin() and lthread_compute_end(), which moves the lthread into an lthread_compute_scheduler that runs in its own pthread to avoid blocking other lthreads. lthread_compute_schedulers are created when needed and they die after 60 seconds of inactivity. lthread_compute_begin() tries to pick an already created and free lthread_compute_scheduler before it creates a new one.

Installation

Currently, lthread is supported on FreeBSD, OS X, and Linux.

To build and install, simply:

./autogen.sh
./configure
sudo make install

Usage

#include <lthread.h>

Pass -llthread to gcc to use lthread in your program.

Library calls

/*
 * Creates a new lthread and returns it via `*new_lt`.
 * Returns 0 if success, -1 on failure.
 */
int lthread_create(lthread_t **new_lt, void *fun, void *arg);

/*
 * Destroys an lthread and cancels any events it was expecting.
 */
void lthread_destroy(lthread_t *lt);

/*
 * Blocks until all lthreads created have exited.
 */
void lthread_run(void);

/*
 * Marks the current lthread as detached, causing it to
 * get freed once it exits. Otherwise `lthread_join()` must be
 * called on the lthread to free it up.
 * If an lthread wasn't marked as detached and wasn't joined on then
 * a memory leak occurs.
 */
void lthread_detach(void);

/*
 * Blocks the calling lthread until lt has exited or timeout occurred.
 * In case of timeout, lthread_join returns -2 and lt doesn't get freed.
 * If you don't want to join again on the lt, make sure to call lthread_destroy()
 * to free up the the lthread else a leak occurs.
 * **ptr will get populated by lthread_exit(). ptr cannot be from lthread's
 * stack space.
 * Joining on a joined lthread has undefined behavior.
 * Returns 0 on success or -2 on timeout.
 */
int lthread_join(lthread_t *lt, void **ptr, uint_64 timeout);

/*
 * Sets ptr value for the joining lthread and exits lthread.
 */
void lthread_exit(void **ptr);

/*
 * Moves lthread into a pthread to run its expensive computation or make a blocking 
 * call like `gethostbyname()`.
 * This call *must* be followed by lthread_compute_end() after the computation and/or
 * blocking calls have been made to resume the lthread in its original lthread scheduler.
 * No lthread_* calls can be made during lthread_compute_begin()/lthread_compute_end(). 
 * WARNING: local variable addresses change inside lthread_compute_begin()/lthread_compute_end(),
 * you MUST NOT access pointers to local variables inside this section. Local variables
 * can still be accessed directly.
 */
 void lthread_compute_begin(void);

/*
 * Moves lthread from pthread back to the lthread scheduler it was running on.
 */
 void lthread_compute_end(void);

/*
 * Puts an lthread to sleep until msecs have passed.
 */
void lthread_sleep(uint64_t msecs);

/* 
 * Wake up a sleeping lthread. 
 */
void lthread_wakeup(lthread_t *lt);

/* 
 * Creates a condition variable that can be used between lthreads to block/signal each other.
 */
int lthread_cond_create(lthread_cond_t **c);

/*
 * Puts the lthread calling `lthread_cond_wait` to sleep until `timeout` expires or another lthread signals it.
 * Returns 0 if it was signaled or -2 if it expired.
 */
int lthread_cond_wait(lthread_cond_t *c, uint64_t timeout);

/*
 * Signals an lthread blocked on `lthread_cond_wait` to wake up and resume.
 */
void lthread_cond_signal(lthread_cond_t *c);

/*
 * Returns the value set for the current lthread.
 */
void *lthread_get_data(void);

/* 
 * Sets data bound to the lthread. This value can be retrieved anywhere in the lthread using `lthread_get_data()`.
 */
void lthread_set_data(void *data);

/*
 * Returns the lthread Id.
 */
uint64_t lthread_id();

/*
 * Returns a pointer to the current lthread.
 */
lthread_t *lthread_current();

/*
 * Sets the lthread method name to the current function.
 * It makes debugging easier by knowing which function a specific lthread was executing.
 */
void lthread_set_funcname(const char *f);

You can use a shortcut macro `DEFINE_LTHREAD;` to set the function name.

###Socket related functions

Refer to the appropriate man pages of each function to learn about their arguments.

/*
 * An lthread version of socket(2).
 * Returns a socket fd.
 */
int lthread_socket(int domain, int type, int protocol);

/*
 * An lthread version of accept(2).
 */
int lthread_accept(int fd, struct sockaddr *, socklen_t *);

/*
 * Close an lthread_socket.
 */
int lthread_close(int fd);

/*
 * An lthread version of connect(2) with an additional argument `timeout` to specify how 
 * long the function waits before it gives up on connecting.
 * `timeout` of 0 means no timeout.
 * Returns 0 on a successful connection or -2 if it expired waiting.
 */
int lthread_connect(int fd, struct sockaddr *, socklen_t, uint64_t timeout);

/*
 * An lthread version of recv(2) with an additional argument `timeout` to 
 * specify how long to wait before it gives up on receiving.
 * `timeout` of 0 means no timeout.
 * Returns the number of bytes received or -2 if it expired waiting.
 */
ssize_t lthread_recv(int fd, void * buf, size_t buf_len, int flags, uint64_t timeout);

/*
 * An lthread version of recvfrom(2) with an additional argument `timeout` to 
 * specify how long to wait before it gives up on receiving.
 * `timeout` of 0 means no timeout.
 * Returns the number of bytes received or -2 if it expired waiting.
 */
ssize_t lthread_recvfrom(int fd, void *restrict buffer, size_t length, int flags,
    struct sockaddr *restrict address, socklen_t *restrict address_len, uint64_t timeout);

/*
 * An lthread version of recvmsg(2) with an additional argument `timeout` to 
 * specify how long to wait before it gives up on receiving.
 * `timeout` of 0 means no timeout.
 * Returns the number of bytes received or -2 if it expired waiting.
 */
ssize_t lthread_recvmsg(int socket, struct msghdr *message, int flags, uint64_t timeout);

/*
 * An lthread version of send(2).
 */
ssize_t lthread_send(int fd, const void *buf, size_t buf_len, int flags);

/*
 * An lthread version of sendmsg(2).
 */
ssize_t lthread_sendmsg(int fd, const void *buf, size_t buf_len, int flags);

/*
 * An lthread version of sendto(2).
 */
ssize_t lthread_sendto(int socket, const void *buffer, size_t length, int flags,
    const struct sockaddr *dest_addr, socklen_t dest_len);

/*
 * An lthread version of writev(2).
 */
ssize_t lthread_writev(int fd, struct iovec *iov, int iovcnt);

freebsd only

/*
 * An lthread version of FreeBSD sendfile(2).
 */
int lthread_sendfile(int fd, int s, off_t offset, size_t nbytes, struct sf_hdtr *hdtr);

Examples

gcc -I/usr/local/include -llthread test.c -o test

A simple timer in lthread

#include <lthread.h>
#include <stdio.h>
#include <sys/time.h>

void
a(lthread_t *lt, void *x)
{
    int i = 3;
    DEFINE_LTHREAD;
    struct timeval t1 = {0, 0};
    struct timeval t2 = {0, 0};
    printf("I am in a\n");
    lthread_detach();

    while (i--) {
        gettimeofday(&t1, NULL);
        lthread_sleep(2000);
        gettimeofday(&t2, NULL);
        printf("a (%d): elapsed is: %ld\n",i ,t2.tv_sec - t1.tv_sec);
    }
    printf("a is exiting\n");
}

void
b(lthread_t *lt, void *x)
{
    int i = 8;
    DEFINE_LTHREAD;
    struct timeval t1 = {0, 0};
    struct timeval t2 = {0, 0};
    printf("I am in b\n");
    lthread_detach();

    while (i--) {
        gettimeofday(&t1, NULL);
        lthread_sleep(1000);
        gettimeofday(&t2, NULL);
        printf("b (%d): elapsed is: %ld\n",i ,t2.tv_sec - t1.tv_sec);
    }
    printf("b is exiting\n");
}

void
c(lthread_t *lt, void *x)
{
    DEFINE_LTHREAD;
	printf("c is running\n");
    lthread_compute_begin();
        sleep(1);
    lthread_compute_end();
	printf("c is exiting\n");
}

void
d(lthread_t *lt ,void *x)
{
	lthread_t *lt_new = NULL;

    DEFINE_LTHREAD;
    lthread_detach();

    printf("d is running\n");
    lthread_create(&lt_new, c, NULL);
    lthread_join(lt_new, NULL, 0);
    printf("d is done joining on c.\n");
}

int
main(int argc, char **argv)
{
    lthread_t *lt = NULL;

    lthread_create(&lt, a, NULL);
    lthread_create(&lt, b, NULL);
    lthread_create(&lt, d, NULL);
    lthread_run();

    return 0;
}

An incomplete web server to illustrate some of lthread features

In this example, the dummy http server will accept a connection, runs a relatively expensive fibonacci(35) in lthread_compute_begin() / lthread_compute_end() and replies back.

While fibonacci is running, we'll continue to accept new connections and handle them without blocking because fibonacci is running between lthread_compute_begin() and lthread_compute_end(), which moves the lthread into a pthread and resumes it there until lthread_compute_end() is called and that's when it moves back to its previous scheduler.