mochi-hpc/mochi-bake

file backend module - [merged]

carns opened this issue · 26 comments

carns commented

In GitLab by @carns on Mar 26, 2020, 15:50

Merges carns/dev-modular-file-backend -> master

This MR is porting forward the file backend to the new modular infrastructure in Bake.

Now passes all simple test cases. There are a few general bug fixes in here, both to core bake and to the file backend logic. File backend supports eager mode now. The core will implement create_write_persist by composing out of separate backend function pointers if the backend doesn't natively provide special functionality for this path.

Future work will be to generalize the mkpool function (and other admin routines). For now this is manually provided to the bake-mkpool command line utility via extern.

@mdorier can you please sanity check this, in particular any changes to core code? I will probably merge it anyway soon to simplify testing, but we can fix problems in master if you don't look at it until later.

carns commented

In GitLab by @carns on Mar 26, 2020, 18:02

added 1 commit

  • 35ceabfe - ability to create file backend targets

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 10:26

added 4 commits

  • 7d90ec5d - add autoconf test for abt-io
  • ad4704e3 - fill in bake_file_backend_initialize
  • 4a571962 - extra usage info for formatting file: backend
  • 19447b59 - update server to allow file: backend format

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 10:33

added 1 commit

  • 02057df5 - fill in bake_file_backend_finalize

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 11:41

added 1 commit

  • 5b92b65f - implement bake_file_create()

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 12:25

added 1 commit

  • 0c9f3c90 - implement bake_file_write_raw() (for eager writes)

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 13:18

added 1 commit

  • efb32cd4 - implement bake_file_persist()

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 13:23

added 12 commits

  • 16d8ba2 - 1 commit from branch master
  • 9782793 - remove pmemobj include from daemon
  • e8cc916 - stub file backend
  • df59cae - ability to create file backend targets
  • efc78b0 - add autoconf test for abt-io
  • d2aa9ee - fill in bake_file_backend_initialize
  • 3a123b5 - extra usage info for formatting file: backend
  • b18ea5f - update server to allow file: backend format
  • a26b0b0 - fill in bake_file_backend_finalize
  • e73ea46 - implement bake_file_create()
  • fdaeff2 - implement bake_file_write_raw() (for eager writes)
  • 7ac0ece - implement bake_file_persist()

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 13:37

added 1 commit

  • e1c33bf - print functionality warnings on startup

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 14:29

added 2 commits

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 16:11

added 1 commit

  • a3a2b53 - add default impl for create_write_persist

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 17:14

added 1 commit

  • 6e895c7 - check for pipelining at file backend init

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 18:59

added 1 commit

  • 20fb836 - read_bulk and write_bulk to common transfer stub

Compare with previous version

carns commented

In GitLab by @carns on Apr 1, 2020, 22:31

added 3 commits

  • 04aefa2 - correct some comments
  • 864ff5b - define struct to track xfer state
  • 743a3bc - set up state for ULTs to run transfers

Compare with previous version

carns commented

In GitLab by @carns on Apr 2, 2020, 17:19

added 1 commit

  • 3ea9c82 - implement primary read/write path

Compare with previous version

carns commented

In GitLab by @carns on Apr 2, 2020, 20:38

added 1 commit

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 08:48

added 1 commit

  • dc73e46 - add file backend variants of make check tests

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 08:58

added 1 commit

  • d48cff1 - fix bugs in previous commit

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 14:18

added 2 commits

  • 50c2258 - fill missing field in request
  • 214475f - implement create_write_persist_bulk as generic fn

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 15:11

added 1 commit

  • 9a588b3 - cut backend object checks in file tests

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 15:18

added 1 commit

  • d047281 - pipelining for multi target and provider tests

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 15:34

added 1 commit

  • 1f53128 - remove "first call" guard in pipeline conf cb

Compare with previous version

carns commented

In GitLab by @carns on Apr 3, 2020, 15:41

unmarked as a Work In Progress

carns commented

In GitLab by @carns on Apr 3, 2020, 15:41

changed title from {-WIP: modular file backend-} to {+file backend module+}

carns commented

In GitLab by @carns on Apr 3, 2020, 15:41

changed the description

carns commented

In GitLab by @carns on Apr 7, 2020, 08:17

merged

carns commented

In GitLab by @carns on Apr 7, 2020, 08:17

mentioned in commit 0b38d8c