kherud/java-llama.cpp

what are slots?

Opened this issue · 0 comments

hello, could you tell me what are slots?

https://github.com/kherud/java-llama.cpp/blob/master/src/main/cpp/server.hpp#L106

struct server_slot
{
    int id;
    int id_task = -1;
    int id_multi = -1;

    struct slot_params params;

    slot_state state = SLOT_STATE_IDLE;
    slot_command command = SLOT_COMMAND_NONE;

    // used to determine the slot that has been used the longest
    int64_t t_last_used = -1;

    // generation props
    int32_t n_ctx = 0; // context size per slot
    int32_t n_past = 0;
    int32_t n_decoded = 0;
    int32_t n_remaining = -1;
    int32_t i_batch = -1;
    int32_t n_predict = -1; // T