/NET

Reservoir computing library for .NET. Enables ESN , LSM and hybrid RNNs using analog and spiking neurons working together.

Primary LanguageC#MIT LicenseMIT

Reservoir Computing for .NET (RCNet)

This project is frozen and no further updates will be available. I leave it here only for inspiration and as a possible source of some implemented functionality. The successor project is EasyML, where you can find the good stuff from here + much more, in a simpler form and higher maturity.

Reservoir Computing conceptual view
This repository offers a .NET machine learning library primarily focused on prediction and classification of time series. RCNet library implementation is based on the "Reservoir Computing" concept where the two most well-known methods are Echo State Network (ESN) and Liquid State Machine (LSM). The main component of RCNet is called "State Machine". State Machine has quite wide configuration options of the reservoirs and except of both mentioned methods (ESN, LSM) it also supports "hybrid" recurrent reservoirs where hidden neurons having an analog and spiking activation can be synaptically connected. The correct synaptic linkage of hidden neurons having heterogeneous activations is enabled by a specific implementation of a hidden neuron, where each hidden neuron provides analog and spiking output regardless of the type of activation function is used. Dual output of hidden neuron also enables to always offer the full set of predictors on hidden neuron level (very important is the FiringTrace predictor). Firing a spike of the hidden neuron having analog activation is based on defined firing event depending on current and previous value of the activation. Use of spiking based predictor (FiringTrace) dramatically improves classification accuracy of the classical ESN. "Analog spiking" is an unique feature discovered and introduced by this project and thanks that, ESN is now capable to classify time series with high accuracy with relatively low resources and to be fully competetive to other classification algorithms.

The demo application (.NET Core 3.1)

A simple demo application demonstrates the StateMachine performance and it also includes several code examples showing how to use key RCNet components.
Application input data is standardly located in the "Data" sub-folder relative to the location of the executable DemoConsoleApp.exe. Data is expected in csv format and data delimiter can be a tab, semicolon or comma character.

  • Continuous feeding regime requires a standard csv format, where the first line contains the names of the data fields and each next line contains the data. Here is an example.
  • Patterned feeding regime requires specific logical csv format without colum names (header). Each data line contains values of steady (optional) and repetitive pattern features followed by expected output values at the end. Values of repetitive pattern features can be organized in two ways: groupped [v1(t1),v2(t1),v1(t2),v2(t2),v1(t3),v2(t3)] or sequential [v1(t1),v1(t2),v1(t3),v2(t1),v2(t2),v2(t3)]. Here is an example.


Application has no startup parameters and when started, it shows the root menu.
_Note that in some cases application may write data into "Examples" sub-folder relative to the location of the executable DemoConsoleApp.exe._

1. menu choice: Demonstration of the State Machine performance and configuration options

Application performs sequence of tasks defined in the SMDemoSettings.xml xml file, where each task is defined in the xml element "case" so you can easily insert new task or tune existing one by simple modification of xml content. SMDemoSettings.xml has to be located in the same folder as the executable DemoConsoleApp.exe.

Notes for SMDemoSettings.xml modification

If you modify SMDemoSettings.xml within the open solution in MS Studio, then for the correct connection of xml and xsd it is enough to have SMDemoSettings.xsd and RCNetTypes.xsd open at the same time. If you modify SMDemoSettings.xsd outside the open solution, then you must first add the schemaLocation = "RCNetTypes.xsd" attribute to the "import" element in the SMDemoSettings.xsd schema in the destination directory where the DemoConsoleApp.exe executable is located. Demo application has internally implemented the xml validation so connection of xml and xsd directly on file level is not necessary. But functional external connection of xml and xsd is useful for manual modifications, because consistency checks, annotations, selections from allowed values and checks of the range of entered values are then available instantly.

Time-series classification results comparison

SMDemoSettings.xml currently includes several classification problems from the: Anthony Bagnall, Jason Lines, William Vickers and Eamonn Keogh, The UEA & UCR Time Series Classification Repository, www.timeseriesclassification.com site and State Machine is able to achieve competetive results to the best classification algorithms referenced on that website.

Dataset State Machine Accuracy Best Ref. Accuracy Best Ref. Algorithm
CricketX 82.05% 81.4% COTE
Worms 83.12% 73.49% BOSS
BeetleFly 100% 94.85% BOSS
BirdChicken 100% 98.4% BOSS
ProximalPhalanx 87.8% 88.09% ST
Yoga 91.37% 90.99% BOSS
Libras 92.78% 89.4% DTWi

2. menu choice: Code examples related to State Machine

Switches to sub-menu with several code examples showing how to configure State Machine manually from scratch or using the StateMachineDesigner component to perform classification or forecasting, then how to train State Machine and verify its performance.

3. menu choice: Code examples related to State Machine

Switches to sub-menu with several code examples showing how to independently use non-recurrent network components. State Machine's readout layer consists of hirarchical structure of non-recurrent network components, but here is shown how to use them as the standalone components with no relation to State Machine.

4. menu choice: Playground

This choice runs the Playground module where you can quickly discover and try to use RCNet components.

RCNet library (.NET Standard 2.0)

Maintenance index Cyclomatic complexity Depth of inheritance Code lines Executable code lines
82 (green) 7482 3 53958 11949

Components by category:

Math

Component Description
BasicStat Implements the basic statistics of sample data.
WeightedAvg Implements the weighted average.
MovingDataWindow Implements the moving data window providing additional functions such as statistics, weighted average, etc.
ODENumSolver Implements a simple numerical solver of the Ordinary Differential Equation(s).
Vector Implements the vector.
Matrix Implements the real matrix. It does not support the sparse matrix format.
EVD Implements the Eigenvalue decomposition of a square matrix.
SVD Implements the Singular Value decomposition of a matrix.
QRD Implements the QR decomposition of a matrix.
LUD Implements the LU (Lowed-Upper) decomposition of a square matrix.
ParamValFinder Implements a simple iterative error-driven search for the parameter's optimal value.
HurstExpEstim Implements the Hurst Exponent estimator using the rescaled range analysis.
"RandomValue" Implements the random value. Supports Uniform, Gaussian, Exponential and Gamma distributions. Here is an extension code
Others Set of small additional helper components like PhysUnit, Interval, Bitwise, Combinatorics, Discrete,...

XML handling

Component Description
DocValidator Implements the xml document loader and validator.

Data generators

Component Description
PulseGenerator Implements the constant pulse generator.
MackeyGlassGenerator Implements the Mackey-Glass generator.
RandomGenerator Implements the random signal generator.
SinusoidalGenerator Implements the sinusoidal signal generator.

Data Filtering

Component Description
BinFeatureFilter Implements the binary feature filter.
RealFeatureFilter Implements the real number feature filter.

Data holding

Component Description
SimpleQueue Implements a simple FIFO queue template. Supports access to enqueued elements so it can be also used as the moving data window.
DelimitedStringValues Implements the single row of the delimited string values (csv format).
CsvDataHolder Implements the holder of csv data and supports its loading and saving to a file.
VectorBundle Implements a bundle of input and output data vector pairs.
InputPattern Implements an input pattern. Pattern can be both univariate or multivariate. Supports data resampling (including simple detection of signal begin/end) and amplitude unification.
ResultBundle Implements a bundle of input, computed and ideal (desired) data vector triplets.

Chainable Input Data Transformations

Component Description
CDivTransformer Implements the transformer of values from one input field. Divides the constant by the value from the input field.
DiffTransformer Implements the transformer of values from one input field. Subtracts the previous value of the input field from the current value.
DivTransformer Implements the transformer of values from two input fields. Divides the value of the first input field by the value from the second input field.
ExpTransformer Implements the transformer of values from one input field. Raises the fixed base (by default the Euler's number) to the power of the value from the input field.
LinearTransformer Implements the transformer of values from two input fields. Computes the linear equation (aX + bY) where "X" is the value from the first input field and "Y" is the value from the second input field. Coefficients "a" and "b" are specified constants.
LogTransformer Implements the transformer of values from one input field. Computes the input field value's logarithm to the specified base.
MulTransformer Implements the transformer of values from two input fields. Multiplies the value of the first input field by the value from the second input field.
MWStatTransformer Implements the transformer of values from one input field. It keeps statistics of the input field recent values and provides specified statistical figure as the transformed value.
PowerTransformer Implements the transformer of values from one input field. Raises the value from the input field to the power of fixed exponent.
YeoJohnsonTransformer Implements the transformer of values from one input field. Calculates the Yeo-Johnson transformation. See the wiki pages.

Analog signal to spikes coders

Component Description
A2SCoderSignalStrength Implements the signal strength coder. Uses a novel coding algorithm to meet together two important spike-train conditions. Stronger signal leads to earlier first spike and higher spiking frequency.
A2SCoderGaussianReceptors Implements the Gaussian Receptive Fields coder.
A2SCoderUpDirArrows Implements the upward signal direction receptors coder. Each receptor is sensitive to upward signal direction of the current signal against the past signal at the time T-x (where x is 1...number of receptors). The positive difference of current and past signal is then expressed as spikes through a novel coding algorithm meeting together two important spike-train conditions where stronger stimulation leads to earlier first spike and higher spiking frequency.
A2SCoderDownDirArrows Implements the downward signal direction receptors coder. Each receptor is sensitive to downward signal direction of the current signal against the past signal at the time T-x (where x is 1...number of receptors). The negative difference of current and past signal is then expressed as spikes through a novel coding algorithm meeting together two important spike-train conditions where stronger stimulation leads to earlier first spike and higher spiking frequency.

Analog activation functions (stateless)

See the wiki pages.

Component Description
AFAnalogBentIdentity Implements the Bent Identity activation function.
AFAnalogElliot Implements the Elliot (aka Softsign) activation function.
AFAnalogGaussian Implements the Gaussian activation function.
AFAnalogIdentity Implements the Identity activation function (aka Linear).
AFAnalogISRU Implements the ISRU (Inverse Square Root Unit) activation function.
AFAnalogLeakyReLU Implements the LeakyReLU (Leaky Rectified Linear Unit) activation function.
AFAnalogSigmoid Implements the Sigmoid activation function.
AFAnalogSinc Implements the Sinc activation function.
AFAnalogSinusoid Implements the Sinusoid activation function.
AFAnalogSoftExponential Implements the Soft Exponential activation function.
AFAnalogSoftMax Implements the Soft Max activation function.
AFAnalogSoftPlus Implements the Soft Plus activation function.
AFAnalogSQNL Implements the Square Nonlinearity activation function.
AFAnalogTanH Implements the Hyperbolic Tangent activation function.

Spiking activation functions (stateful)

See the wiki pages.

Component Description
AFSpikingAdExpIF Implements the Adaptive Exponential Integrate and Fire neuron model.
AFSpikingExpIF Implements the Exponential Integrate and Fire neuron model.
AFSpikingIzhikevichIF Implements the Izhikevich Integrate and Fire neuron model.
AFSpikingLeakyIF Implements the Leaky Integrate and Fire neuron model.
AFSpikingSimpleIF Implements a very simple form of the Integrate and Fire neuron model.

Non-recurrent networks and trainers

Component Description
FeedForwardNetwork Implements the Feed Forward network supporting multiple hidden layers.
RPropTrainer Implements the Resilient Backpropagation iRPROP+ trainer of the feed forward network.
QRDRegrTrainer Implements the QRD regression trainer of the feed forward network. The feed forward network to be trained must have no hidden layers and the Identity output activation.
RidgeRegrTrainer Implements the Ridge regression trainer of the feed forward network. The feed forward network to be trained must have no hidden layers and the Identity output activation.
ElasticRegrTrainer Implements the Elastic linear regression trainer of the feed forward network.. The feed forward network to be trained must have no hidden layers and the Identity output activation.
ParallelPerceptron Implements the Parallel Perceptron network.
PDeltaRuleTrainer Implements the P-delta rule trainer of the parallel perceptron network.
TNRNet Implements the holder of trained non-recurrent network and its error statistics.
TNRNetBuilder Implements the builder of the trained non-recurrent network.
TNRNetCluster Implements the cluster of trained non-recurrent networks.
TNRNetClusterBuilder Implements the builder of trained non-recurrent networks cluster based on the cross-validation approach.
TNRNetClusterChain Implements the chain of cooperating non-recurrent network clusters.
TNRNetClusterChainBuilder Implements the builder of the chain of cooperating non-recurrent network clusters.

State Machine components

StateMachine Components Relationship

Component Description
AnalogInputNeuron Implements the input analog neuron. The input analog neuron is a special case of the neuron without an activation function. Its purpose is to provide an analog input for the reservoir's synapses.
SpikingInputNeuron Implements the input spiking neuron. The input spiking neuron is a special case of the neuron without an activation function. Its purpose is to provide a spiking input for the reservoir's synapses.
InputEncoder Implements a mediation layer between the external input data and the internal reservoirs of the neural preprocessor. Processes the external input data in the natural form and provides it's representation on analog and spiking input neurons for the next processing in the reservoirs. Allows to create new computed input fields using chainable transformations of existing external input fields, as well as adding independently generated input fields using various generators. Supports two input feeding modes: Continuous and Patterned. The Continuous feeding mode processes an input vector as the variable(s) values at the single time-point T. The Patterned feeding mode processes an input vector as an alone input pattern consisting of a time series of the variable(s) values. Supports three ways how to represent an analog value as the spikes: Horizontal, Vertical or Forbidden. The Horizontal way of coding means a simultaneous activity of the neuronal population where every input field is coded by several spiking input neurons (a horizontal spike-train). It is fast, it leads to a single computation cycle of the reservoirs per the input field value. The Vertical way of coding means that the input field value is coded as a spike-train on a single spiking input neuron. It is slower, it leads to multiple computation cycles of the reservoirs according to the spike-train length. The Forbidden way of coding means there is no coding of an analog value as the spikes. It is fast, it leads to a single computation cycle of the reservoirs per the input field value and it does not utilize any spiking input neuron(s).
Synapse Implements the synapse. Supports the signal delaying and the short-term plasticity.
PredictorsProvider Implements the provider of the unified set of computed predictors.
HiddenNeuron Implements the hidden neuron. Supports engagement of both analog and spiking activation functions and provides unified set of available predictors.
ReservoirInstance Implements the reservoir.
NeuralPreprocessor Implements the neural preprocessor supporting multiple reservoirs.
ReadoutUnit Implements the readout unit of the readout layer. It can do the Forecast or Classification.
OneTakesAllGroup Implements the "One Takes All" group of readout units. Supports basic decision-making based directly on the results of readout units and also more advanced decision-making based on the result of a dedicated chain of network clusters.
PredictorsMapper Implements the mapper of specific predictors to readout units.
ReadoutLayer Implements the readout layer consisting of trained readout units (and "One Takes All" groups).

State Machine

The main serializable StateMachine component encapsulates the NeuralPreprocessor and ReadoutLayer components into the single component and adds support for routing of specific predictors and input fields to the specific readout units. State Machine also allows to bypass NeuralPreprocessor and to use input data directly as a predictors for the readout layer.

Setup

Each executive component that makes up StateMachine (including StateMachine itself) has its own related settings class providing configuration, which is required by the executive component's constructor.
Each settings class can be instantiated manually from scratch or from a xml element encapsulating all parameters. RCNetTypes.xsd defines all xml elements used in settings classes constructors.
Each settings class also implements the GetXml method so it can be instantiated from scratch and the initialization xml element can be exported by calling the GetXml method (and stored for later use). Using xml constructors is generally preferable because the initialization xml can be edited without the need to modify source code of the manual setup.
To make things easier, RCNet also implements helper component StateMachineDesigner for easier setup of simple ESN and LSM StateMachine configurations from the code (see the examples in demo application).

Documentation and contact

More detailed documentation will be gradually published here. Questions, ideas, suggestions for improvement and constructive comments are welcome at my email address oldrich.kozelsky@email.cz or newly you can use github discussions.