Vamp Plugin Tester ================== This program tests Vamp audio feature extraction plugins (http://vamp-plugins.org/) for certain common failure cases. To test a single plugin, run vamp-plugin-tester with the name of your plugin library and plugin identifier, separated by a colon. For example, $ vamp-plugin-tester vamp-example-plugins:amplitudefollower The plugin library must be installed in the Vamp plugin path (you cannot give the path to the library file). Options ======= Supply the -a or --all option to tell vamp-plugin-tester to test all plugins found in your Vamp path. Supply the -v or --verbose option to tell vamp-plugin-tester to print out the whole content of its returned feature log for diagnostic purposes each time it prints an error or warning that arises from the contents of a returned feature. Supply the -n or --nondeterministic option to tell vamp-plugin-tester that your plugins are expected to return different results each time they are run. The default behaviour is to treat different results on separate runs with the same input data as an error. Supply the -t or --test option with a test ID argument to tell vamp-plugin-tester to run only a single test, rather than the complete test suite. To find out what test ID to use for a given test, run vamp-plugin-tester with the --list-tests or -l option. Errors and Warnings =================== Each test may cause one or several notes, warnings, or errors to be printed. A note is printed when behaviour is observed that may be correct behaviour but that is not always anticipated by the plugin developer. A warning is printed when behaviour is observed that is technically legal but that in practice most often happens by mistake. An error is printed when behaviour is observed that cannot be correct. vamp-plugin-tester prints all of its commentary to the standard output. Standard error is usually used for diagnostic output printed by the plugins themselves. In addition to reports, vamp-plugin-tester runs some tests that are intended to provoke the plugin into unexpected behaviour such as memory errors. If vamp-plugin-tester crashes during a test, this may be why. If you have access to a memory checker utility such as valgrind, you are advised to run vamp-plugin-tester under it so as to be informed of any memory errors that do not happen to cause crashes (as well as memory leaks). The vamp-plugin-tester binaries distributed by QMUL have been compiled with debug information included, in order to facilitate this type of use. Error and Warning Reference =========================== ** ERROR: Failed to load plugin The plugin could not be loaded. Remember that the plugin must be installed in the Vamp plugin path. Normally this message will be preceded by one of the following errors: Invalid plugin key <key> in loadPlugin - The argument given to vamp-plugin-tester could not be split into library name and plugin identifier. Check the usage description above. No library found in Vamp path for plugin <key> - No Vamp plugin library of that name was found in the Vamp path. This message will often be accompanied by one of the following errors; if it isn't, then that probably means the file did not exist at all. Plugin <id> not found in library <name> - The library was found and loaded and was apparently a valid Vamp plugin library, but it didn't contain a plugin of that id. Check you typed the id correctly, and if this is your library, check that the vampGetPluginDescriptor function returns the plugin descriptor properly. Unable to load library <name> - A dynamic library of that name was found, but the system library loader could not load it. Perhaps it depends on another library that is not available, or it was built for the wrong architecture. There may be more information in the error message. No vampGetPluginDescriptor function found in library <name> - A dynamic library of that name was found and loaded, but it lacked the necessary public vampGetPluginDescriptor function. * Are you sure this is a Vamp plugin library? * If you made it, did you remember to include the global vampGetPluginDescriptor function in your library along with your plugin classes? * If you are using Visual C++, did you remember to mark the vampGetPluginDescriptor symbol exported, as described in the README.msvc file in the SDK? ** ERROR: (plugin|parameter|output) identifier <x> contains invalid characters An identifier contains characters other than the permitted set (ASCII lower and upper case letters, digits, "-" and "_" only). ** ERROR: <field> is empty A mandatory field, such as the name of a parameter or output, contains no text. ** WARNING: <field> is empty An optional field, such as the description of a parameter or output, contains no text. ** ERROR: Plugin parameter <x> maxValue <= minValue The minimum and maximum values given for a parameter are equal or in the wrong order. ** ERROR: Plugin parameter <x> defaultValue out of range The default value for a parameter is not within the range defined by the minimum and maximum values for the parameter. ** ERROR: Plugin parameter <x> is quantized, but quantize step is zero The quantizeStep value in a parameter with isQuantized true is set to zero. ** WARNING: Plugin parameter <x> value range is not a multiple of quantize step A parameter's stated maximum value is not one of the possible values obtained by adding multiples of the quantize step on to the minimum value. ** WARNING: Plugin parameter <x> has (more|fewer) value names than quantize steps A quantized parameter lists some value names for its quantize steps, but not the right number. ** WARNING: Plugin parameter <x> default value is not a multiple of quantize step beyond minimum The default value for a parameter is not a value that the user could actually obtain, if only offered the quantized values to choose from. ** ERROR: Data returned on nonexistent output The output number key for a returned feature is outside the range of outputs listed in the plugin's output descriptor list. ** NOTE: No results returned for output <x> The plugin returned no features on one of its outputs, when given a simple test file. This may be perfectly reasonable behaviour, but you might like to know about it. ** ERROR: Plugin output <x> has FixedSampleRate but gives sample rate as 0 A plugin output that has a sample type of FixedSampleRate must have a non-zero sample rate. See https://code.soundsoftware.ac.uk/projects/vamp-plugin-sdk/wiki/SampleType ** NOTE: Plugin returns features with timestamps on OneSamplePerStep output ** NOTE: Plugin returns features with durations on OneSamplePerStep output Hosts will usually ignore timestamps and durations attached to any feature returned on a OneSamplePerStep output. See https://code.soundsoftware.ac.uk/projects/vamp-plugin-sdk/wiki/SampleType ** ERROR: Plugin returns features with no timestamps on VariableSampleRate output Timestamps are mandatory on all features associated with a VariableSampleRate output. See https://code.soundsoftware.ac.uk/projects/vamp-plugin-sdk/wiki/SampleType ** WARNING: Plugin returned one or more NaN/inf values The plugin returned features containing floating-point not-a-number or infinity values. This warning may be associated with a test involving feeding some unexpected type of data to the plugin. ** ERROR: Consecutive runs with separate instances produce different results The plugin was constructed and run twice against the same input data, and returned different features each time. If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this error to a note. ** ERROR: Consecutive runs with the same instance (using reset) produce different results The plugin was constructed, initialised, run against some input data, reset with a call to its reset() function, and run again against the same data; and it returned different features on each run. This is often a sign of some simple error such as forgetting to implement reset(). If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this error to a note. ** ERROR: Simultaneous runs with separate instances produce different results Two instances of the plugin were constructed and run against the same input data, giving each block of data to one plugin's process call and then to the other's, "interleaving" the processing between the two instances (but within a single application thread); and the two instances returned different features. This may indicate ill-advised use of static data shared between plugin instances. If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this error to a note. ** WARNING: Consecutive runs with different starting timestamps produce the same result The plugin was run twice on the same audio data, but with different input timestamps, and it returned the same results each time. While this is often unproblematic, it can indicate that a plugin failed to take the input timestamp into account when calculating its output timestamps (if any). If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this warning to a note. ** ERROR: Explicitly setting current program to its supposed current value changes the results The plugin was constructed and run twice on the same data, once without changing its "program" setting, and again having set the program to the vaule returned by getCurrentProgram() (i.e. the same program that was supposed to be in effect already). It returned different results for the two runs, suggesting that some internal data was changed in selectProgram in a way that differed from its default. If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this error to a note. ** ERROR: Explicitly setting parameters to their supposed default values changes the results The plugin was constructed and run twice on the same data, once without changing any of its parameters, and again having set the parameters to their specified default values. It returned different results for the two runs, suggesting that some internal data was changed when a parameter was set to its default, in a way that differed from the plugin's initially constructed state. If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this error to a note. ** ERROR: Call to reset after setting parameters, but before processing, changes the results (parameter values not retained through reset?) The plugin was constructed and run twice on the same data. The first time, its parameters were set to some arbitrary values and it was initialised and run. The second time, its parameters were set to the same values and it was initialised, then reset(), then run. The two runs returned different results, suggesting that perhaps some parameter value was being modified within the reset() function. (This function should reset internal state within the plugin, but not parameter configuration.) If you give the -n or --nondeterministic option, vamp-plugin-tester will downgrade this error to a note. ** WARNING: Constructor takes some time to run: work should be deferred to initialise? The plugin took a long time to construct. You should ensure that the constructor for the plugin runs as quickly as possible, because it may be called by a host that is only scanning the properties of all available plugins on startup. Any serious initialisation work should be done in the initialise() function rather than the constructor. Authors ======= This program was written at the Centre for Digital Music at Queen Mary, University of London, by Chris Cannam. Copyright (c) 2009-2015 QMUL.
c4dm/vamp-plugin-tester
Automated tester for Vamp audio analysis plugins, capable of picking up a number of common failure cases.
Standard MLNOASSERTION