Issues
- 2
Multi-dataset abstraction layer
#142 opened by JackKelly - 0
Utility to extract, reshape, and store data for a subset of the data. e.g. for extracting timeseries for single PV sites from gridded NWPs
#141 opened by JackKelly - 0
- 0
Why does `lsio_bench` give insanely high bandwidth numbers when reading a single chunk per file?
#137 opened by JackKelly - 3
Build a command-line benchmarking app
#130 opened by JackKelly - 0
Generic `user_data` type for `Chunk`
#140 opened by JackKelly - 0
Remove `crates /lsio_uring/benches`
#139 opened by JackKelly - 0
Write unit-tests for `AlignedBuffer` to check behaviour when using `start_offset`.
#98 opened by JackKelly - 0
- 0
Benchmark `lsio_uring` vs `object_store_bridge`
#123 opened by JackKelly - 0
Maybe use `Bend` to parallelise compute
#132 opened by JackKelly - 1
Benchmark `lsio_uring`
#135 opened by JackKelly - 0
- 0
- 0
- 0
Rigorous automated testing
#118 opened by JackKelly - 0
Rigorous benchmarking
#119 opened by JackKelly - 0
- 9
What would user's code would look like the for simple use-cases? (After dropping `async`, and instead only using `channels` and `Rayon`)?
#104 opened by JackKelly - 1
- 0
Keep track of SQ entries, in a 1:1 relationship
#131 opened by JackKelly - 0
- 15
Maybe change name (away from LSIO)
#110 opened by JackKelly - 0
Benchmark for new io_uring code
#127 opened by JackKelly - 0
Implement optimisation of byte ranges
#126 opened by JackKelly - 0
Test new io_uring code
#128 opened by JackKelly - 1
Plan to re-write LSIO
#121 opened by JackKelly - 10
- 1
- 0
Use a flat crate structure so this git repo can store multiple (interconnected) crates in a single workspace
#94 opened by JackKelly - 6
Allow `AlignedBuffer` to return "owned slices" (like `Bytes`), which refer to the same buffer in RAM
#114 opened by JackKelly - 0
`AlignedBuffer` should use `valid_slice` to define the `start` and `len` that the user is interested in
#115 opened by JackKelly - 0
Stop/Start: Enable LSIO to be gracefully stopped & started (e.g. in response to carbon intensity)
#124 opened by JackKelly - 1
Drop `ObjectStore` & `async`/`await`. Use Channels instead. Focus entirely (for now) on io_uring for local file storage.
#93 opened by JackKelly - 2
Maybe use `Bytes` within `AlignedBuffer`
#112 opened by JackKelly - 1
How to identify each returned `Item`?
#105 opened by JackKelly - 4
`Typestate`: Use a custom type for each state (for each `IoOperation`) & use Rust's type system to make sure we only progress forwards
#117 opened by JackKelly - 2
Implement `AlignedBuffer::into_box`
#108 opened by JackKelly - 1
Check if `ndarray`, `numpy`, & compression crates can consume `&[u8]` (and hence consume from `AlignedBuffer`)
#113 opened by JackKelly - 0
Document decisions taken
#120 opened by JackKelly - 1
New crate: `lsio_optimise_io_plan` (e.g. merge nearby operations; split large ops)
#102 opened by JackKelly - 2
- 3
look into using Rayon for the IO thread pool. Can we associate an `IoUring` instance with each thread? Can we publish tasks to other threads?
#103 opened by JackKelly - 0
- 5
- 1
- 0
Change `MaybeDone` to `Pending` and `Done`
#91 opened by JackKelly - 0
Change `Operation` to a `Trait`, & use a struct for each operation (`get`, `get_range`, etc.). Store Trait objects in `user_tasks_in_flight`.
#86 opened by JackKelly - 0
- 0