Does zenfs support YCSB？
Closed this issue · 15 comments
I'm using an emulator to emulate zns ssd.
I can run zenfs and db_bench but I can't run rocksdb in YCSB, is there any way run YCSB?
Hi @YummuyWang,
It looks like YCSB does not support ZenFS out of the box.
If you would like to explore enabling ZenFS within YCSB it seems like you would need to add a maven dependency for RocksDB with ZenFS enabled.
Then you should be able to configure your ZenFS fs-uri through the rocksdb.optionsfile https://github.com/brianfrankcooper/YCSB/tree/master/rocksdb#rocksdb-configuration-parameters.
Please note that I am not a maven/YCSB expert, so you might want to contact the YCSB maintainers to see if this is the way to go.
Feel free to open a PR with some instructions on how to run YCSB with ZenFS if you get it to work :)
Thanks for reply.
I can run YCSB in /tmp/ycsb-rocksdb-data, but I can run YCSB in /dev/nvmeXnY after created ZenFS.
I ran ./bin/ycsb load rocksdb -s -P workloads/workloada -p rocksdb.dir=/dev/nvme4n1 and got
site.ycsb.DBException: org.rocksdb.RocksDBException: /dev/nvme4n1' exists but is not a directory
It seems I didn't mount a directory.
Does ZenFS support mount to a directory?
But I tried to use zonefs and mount to /mnt
./bin/ycsb load rocksdb -s -P workloads/workloada -p rocksdb.dir=/mnt
It seems there is something wrong.
2024-01-23 23:51:49:666 0 sec: 0 operations; est completion in 0 second
site.ycsb.DBException: org.rocksdb.RocksDBException: while open a file for lock: /mnt/LOCK: Permission denied
at site.ycsb.db.rocksdb.RocksDBClient.init(RocksDBClient.java:80)
at site.ycsb.DBWrapper.init(DBWrapper.java:90)
at site.ycsb.ClientThread.run(ClientThread.java:91)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.rocksdb.RocksDBException: while open a file for lock: /mnt/LOCK: Permission denied
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:235)
at site.ycsb.db.rocksdb.RocksDBClient.initRocksDB(RocksDBClient.java:159)
at site.ycsb.db.rocksdb.RocksDBClient.init(RocksDBClient.java:77)
... 3 more
site.ycsb.DBException: org.rocksdb.RocksDBException: while open a file for lock: /mnt/LOCK: Permission denied
at site.ycsb.db.rocksdb.RocksDBClient.init(RocksDBClient.java:80)
at site.ycsb.DBWrapper.init(DBWrapper.java:90)
at site.ycsb.ClientThread.run(ClientThread.java:91)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.rocksdb.RocksDBException: while open a file for lock: /mnt/LOCK: Permission denied
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:235)
at site.ycsb.db.rocksdb.RocksDBClient.initRocksDB(RocksDBClient.java:159)
at site.ycsb.db.rocksdb.RocksDBClient.init(RocksDBClient.java:77)
... 3 more
Maybe YCSB doesn't support ZNS device or my emulator doesn't work well.
ZenFS is not a POSIX-compatible filesystem. It is a file system plugin that utilizes RocksDB's FileSystem interface to place files into zones on a raw zoned block device.
In other words, you can't access a ZenFS disk the way you are trying to by mounting the ZenFS formatted disk.
When you create a ZenFS instance you then have to specify the fs_uri RocksDB parameter to the application that utilizes the RocksDB storage engine (that is compiled with the ZenFS plugin enabled).
You should be able to configure your ZenFS fs-uri through the rocksdb.optionsfile as soon as you are using RocksDB with ZenFS installed within YCSB https://github.com/brianfrankcooper/YCSB/tree/master/rocksdb#rocksdb-configuration-parameters.
Let me know if that helps or if you have follow-up questions.
Hi, I configured my rocksdb.optionsfile and added line like this
[DBOptions]
...
use_direct_io_for_flush_and_compaction=true
fs_uri=zenfs://dev:nvme4n1
...
And I got something wrong likes this
site.ycsb.DBException: org.rocksdb.RocksDBException: Unrecognized option DBOptions:: fs_uri
at site.ycsb.db.rocksdb.RocksDBClient.init(RocksDBClient.java:80)
at site.ycsb.DBWrapper.init(DBWrapper.java:90)
at site.ycsb.ClientThread.run(ClientThread.java:91)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.rocksdb.RocksDBException: Unrecognized option DBOptions:: fs_uri
at org.rocksdb.OptionsUtil.loadOptionsFromFile(Native Method)
at org.rocksdb.OptionsUtil.loadOptionsFromFile(OptionsUtil.java:110)
at org.rocksdb.OptionsUtil.loadOptionsFromFile(OptionsUtil.java:91)
at site.ycsb.db.rocksdb.RocksDBClient.initRocksDBWithOptionsFile(RocksDBClient.java:105)
It that similar to Changes to NewObject() function signature. ?
Sorry for the confusion about the fs_uri parameter. This parameter is not a RocksDB option. It is a parameter specific to db_bench.
Assuming you compiled RocksDB with the ZenFS plugin for YCSB to link with, there are a few code changes in YCSB needed to access a ZenFS target.
Within YCSB you would need to modify the options parameter when doing a RocksDB.open (e.g. https://github.com/brianfrankcooper/YCSB/blob/master/rocksdb/src/main/java/site/ycsb/db/rocksdb/RocksDBClient.java#L108).
The options.env must be a rocksdb::Env that was created with the fs_uri string (see CreateFromUri definition).
I am not familiar with the RocksDB Java bindings, but there is an open PR (#285) in C++ that shows you how to achieve what I just described (link to code section):
rocksdb::Env::CreateFromUri(config_options, "", "zenfs://dev:" + device_name,
&fs_env, &fs_env_guard);
options.env = fs_env;
// Open database
rocksdb::Status status = rocksdb::DB::Open(options, "rocksdbtest", &db);
It is also worth looking at the code of db_bench and how the fs_uri parameter is handled there. :)
Hi, I am also trying to run YCSB with ZenFS over FEMU emulator. I am able to run db_bench but when it comes to YCSB, I am facing some issues.
I am able to make the fs_uri option be read by the YCSB (which is mentioned above as invalid argument) but during the options validation process, it is giving an error saying "Missing configurable parameter: file_system" which is giving me a hint that although fs_uri is set but it is still not able to provide any path to create the database as open function of RocksDB require a path to open a database.
fs uri zenfs://dev:nvme0n1 // this is fs_uri path
env.cc - CreateFromUri else // this is varification print statement that CreateFromUri being executed normally
options_help.cc - ValidateOptions //another print system showing that options are being validated
Caught exception: RocksDB Open: NotFound: Validate - Missing configurable object: file_system //this is the error message
any help in this regard will be appreciated.
Hi @safdarjamil95 , sorry for the problem you met. I was on my holiday recently, so I didn't try to revise the code.
I think the problem you met maybe is that the ZenFS is not a POSIX file system, it can't be recognized by kernel and it doesn't support command such as mkfs.
It's awkward that I'm not familiar with Rocksdb, so I'm not sure if my assumption is correct.
Can you put your code here or your github page? I will try to revise the code recently, it would be helpful.
Hi @YummuyWang, I understand the confusion and have some of understanding of RocksDB's code and how it works. The problem so far I have identified is that within YCSB, RocksDB is not able to identify ZenFS however, the ZenFS is build and initialized using the guidelines provided at the ZenFS's github page. I am trying to understand how db_bench's implementation is able to identify ZenFS as underneath file system.
Hi @safdarjamil95,
The db_bench implementation links to the ZenFS enabled RocksDB library.
You also need to link YCSB to the ZenFS enabled RocksDB library that you compiled yourself. Unfortunately, I am not a big help on how to do that with Java/Maven.
Within
static Status CreateFromUri(const ConfigOptions& options,
const std::string& env_uri,
const std::string& fs_uri, Env** result,
std::shared_ptr<Env>* guard);
you supply the fs_uri which then recognises the ZenFS target (if linked correctly).
As mentioned in my last comment you can see how to link ZenFS and use CreateFromUri within a C++ application in the open PR #285 (or in the db_bench implementation which is a bit more complicated to follow).
Hi @MaisenbacherD,
Thank you for your response. I am able to make it run with YCSB and also, I was working C++ version of YCSB so there was linking problem during the compilation of YCSB. I provided that linkage path and it worked very well.
YCSB C++ version: https://github.com/ls4154/YCSB-cpp
Modification to the Makefile: EXTRA_LDFLAGS ?=-L/home/femu/BlobZenFS/rocksdb -ldl -lz -lsnappy -lzstd -lbz2 -llz4 -lssl -lcrypto -lgflags -u zenfs_filesystem_reg -lzbd
Modification to rocksdb/rocksdb_db.cc at the definition of RocksdbDB::GetOptions
rocksdb::Env* env = rocksdb::Env::Default();;
if (!env_uri.empty() || !fs_uri.empty()) {
rocksdb::Status s = rocksdb::Env::CreateFromUri(rocksdb::ConfigOptions(), env_uri, fs_uri, &env, &env_guard);
if (!s.ok()) {
throw utils::Exception(std::string("RocksDB CreateFromUri: ") + s.ToString());
}
printf("env is nullptr ? %s %p filesystem %p\n",env ? "no" : "yes",env,env->GetFileSystem().get());
opt->env = env;
}
I hope this will be of help.
Could you please point me to a YCSB-cpp fork of yours that includes all the changes you made and provide some detailed instructions on how you are compiling and running all of that? :)
I am still suspecting some linking issues.
Hi everyone
i wrote down the code that @safdarjamil95 write down that can run YCSB on zenfs.
As my codebase is too much modified for my research(not published), I cannot open my all my source code.
But you need to focus on rocksdb FileSystem pointer.
printf and check it the FileSystem pointer is pointing at ZenFS , not Posix Filesystem.
Also, you need to compile YCSB-cpp(github repository) as below options. i use these flags.
sudo make BIND_ROCKSDB=1 EXTRA_CXXFLAGS=-I/home/sungjin/rocksdb/include \
EXTRA_LDFLAGS="-L/home/sungjin/YCSB-cpp/build -ldl -lsnappy -lgflags -ldl -lz -u zenfs_filesystem_reg -lzbd -lzstd -llz4 -lbz2"
fs uri should be in rocksdb.properties, not options.ini file.
example :
rocksdb.properties
rocksdb.fs_uri=zenfs://dev:sdb
# Load options from file
rocksdb.optionsfile=zenfsoptions.ini
sudo ~/YCSB-cpp/ycsb -load -db rocksdb -P /home/sungjin/YCSB-cpp/workloads/your_workload -P /home/sungjin/YCSB-cpp/rocksdb/rocksdb.properties -s
reference my repository https://github.com/sg20180546/YCSB-cpp , branch DEBUG4 (i am still using),
but don't use for your rocskdb because there's many customized options added for my research.
@sg20180546 @safdarjamil95 Thanks a lot both of you.
I can run YCSB on zenfs now.
Thanks to the reference of sg20180546's code.
after modify rocksdb.properties
rocksdb.dbname=/
It works.
Thanks @sg20180546 for providing your references!
I am glad it works now. :)
Closing this issue. Feel free to open a new issue in case something new comes up.