This is an IDS storage plugin adapted to the needs of HZB. For the moment, it is in an experimental stage. It is mostly used to explore the IDS and to try out how IDS storage plugins can be customized.
Customized, meaningful file names in the storage. The file name layout is:
<storage-root>/<name-space>/<cycle>/<inv-name>/data/<dataset-name>/<datafile-name>
where
<name-space>is a name space prefix and<cycle>corresponds to the facility cycle, both are derived from the investigation name<inv-name>.Configurable permissions and posix group for files created in the storage.
External storage areas: a prefix on the location of Datasets and Datafiles determines whether it is to be searched in an external storage area or not. These external storage area are accessed read-only and are excluded from the standard two-level storage mechanism.
Use file system locking when accessing the storage to allow concurrent access to the storage by external processes.
Check the location argument in the callbacks for consistency to work around a vulnerability in IDS.
ids.storage_hzb is based on ids.storage_file. There is no binary release, you need to build the distribution from the sources. Once this is done, the original install instructions for ids.storage_file mostly applies to ids.storage_hzb as well, see the general installation instructions and the specific installation instructions for ids.storage_file.
There are a few additional configuration options in ids.storage_hzb.main.properties though, see the comments in the file.
This plugin requires ids.plugin 1.4.0 and ids.server 1.9.0 or greater.
| plugin version | ids.server version |
|---|---|
| 0.6.0 - 0.6.5 | 1.10.0 - 1.12.0 |
| 0.5.0 - 0.5.2 | 1.9.0 - 1.9.1 |
| 0.4.0 | 1.8.0 |
| 0.3.3 - 0.3.5 | 1.6.0 - 1.7.0 |
- It is impossible to safely remove a lock file from the storage without a race condition. For this reason, this plugin does not remove the lock files at all, but rather leaves this to an external tool that tries to minimize the risk. Note that this is not a bug in this particular implementation, but rather a design flaw in POSIX fcntl() style locks. The problem is that fcntl() locks are obtained on open file descriptors, but there is no way to open and lock a file in one single atomic operation.
- Fix setup script for Python 3
- Allow colon character in file names
- Update regular expression for GATE proposal numbers.
- Bump logback-classic from 1.1.3 to 1.2.0: CVE-2017-5929
- Bugfix: fixed regular expression for misc proposal numbers.
- Bugfix: Should ignore IOException while trying to set permissions or file modification time of the lock file or the directory in DirLock.
- Base file locking in main storage on the new lock() method introduced with ids.plugin 1.5.0. As a consequence, file locking will only work with ids.server 1.10 or newer.
- Merge ids.storage_file 1.4.2: delete() methods do not throw an exception if the files to be deleted do not exist.
- Require ids.plugin 1.5.0.
- Add proposal number name spaces 'pub' and 'misc'.
- Throw InvalidPathException rather then IOException if a file path fails sanitation checks.
- Reenable logging.
- Allow underscore as first character in file and dataset names.
- Adapt storage path to new style proposal numbers.
- Drop facility name and visit id from the zipper name.
- Add a configuration option to define the group for files and directories created by the plugin in the storage.
- Implement file locking in ArchiveFileStorage also for put().
- Merge ids.storage_file 1.4.1. Derive the storage classes from the new abstract classes defined in the plugin. Require ids.plugin 1.4.0.
- Merge ids.storage_file 1.4.0. Adapt to new configuration interface introduced with ids.server 1.8.0.
- Rename package to org.icatproject.site.hzb.ids.storage.
- Add a configuration option to define the permissions for files and directories created by the plugin in the storage.
- Reenable deleting of parent directories on delete() in MainFileStorage. Note that this will not work if file locking is enabled.
- Add configuration options to switch file locking on or off (default off).
- Do not remove old lock files.
- ArchiveFileStorage.delete() does not throw an exception if the archive does not exist. (See also icatproject/ids.server#61.)
- Upgrade to icat.utils 4.15.1.
- Add another directory level "data" above the dataset.
- Merge ids.storage_file 1.3.3. This requires ids.server 1.6.0.
Bugfixes:
- Must not lock directories in external storage areas.
- Create the directory lock file in the parent directory rather then in the directory itself.
Do not check the existence of dataset directories in external storage areas, rather assume them always to exist, e.g. always return true. This removes any semantic from the location attribute in Dataset other then the presence of a storage area prefix, e.g. for Dataset, the location does not need to be an existing file or directory.
Implement locking of dataset directories in the main storage.
Implement external storage areas.
Basically a customized version of ids.storage_file. Originally derived from ids.storage_file 1.2.0, later ported to 1.3.0.
Copyright 2012-2018 The ICAT Collaboration Copyright 2015-2018 Helmholtz-Zentrum Berlin für Materialien und Energie GmbH
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.