This project is frozen and no further updates will be available. I leave it here only for inspiration and as a possible source of some implemented functionality. The successor project is EasyML, where you can find the good stuff from here + much more, in a simpler form and higher maturity.
This repository offers a .NET machine learning library primarily focused on prediction and classification of time series.
RCNet library implementation is based on the "Reservoir Computing" concept where the two most well-known methods are Echo State Network (ESN) and Liquid State Machine (LSM).
The main component of RCNet is called "State Machine". State Machine has quite wide configuration options of the reservoirs and except of both mentioned methods (ESN, LSM) it also supports "hybrid" recurrent reservoirs where hidden neurons having an analog and spiking activation can be synaptically connected.
The correct synaptic linkage of hidden neurons having heterogeneous activations is enabled by a specific implementation of a hidden neuron, where each hidden neuron provides analog and spiking output regardless of the type of activation function is used.
Dual output of hidden neuron also enables to always offer the full set of predictors on hidden neuron level (very important is the FiringTrace predictor).
Firing a spike of the hidden neuron having analog activation is based on defined firing event depending on current and previous value of the activation. Use of spiking based predictor (FiringTrace) dramatically improves classification accuracy of the classical ESN.
"Analog spiking" is an unique feature discovered and introduced by this project and thanks that, ESN is now capable to classify time series with high accuracy with relatively low resources and to be fully competetive to other classification algorithms.
A simple demo application demonstrates the StateMachine performance and it also includes several code examples showing how to use key RCNet components.
Application input data is standardly located in the "Data" sub-folder relative to the location of the executable DemoConsoleApp.exe. Data is expected in csv format and data delimiter can be a tab, semicolon or comma character.
- Continuous feeding regime requires a standard csv format, where the first line contains the names of the data fields and each next line contains the data. Here is an example.
- Patterned feeding regime requires specific logical csv format without colum names (header). Each data line contains values of steady (optional) and repetitive pattern features followed by expected output values at the end. Values of repetitive pattern features can be organized in two ways: groupped [v1(t1),v2(t1),v1(t2),v2(t2),v1(t3),v2(t3)] or sequential [v1(t1),v1(t2),v1(t3),v2(t1),v2(t2),v2(t3)]. Here is an example.
Application has no startup parameters and when started, it shows the root menu.
_Note that in some cases application may write data into "Examples" sub-folder relative to the location of the executable DemoConsoleApp.exe._
Application performs sequence of tasks defined in the SMDemoSettings.xml xml file, where each task is defined in the xml element "case" so you can easily insert new task or tune existing one by simple modification of xml content. SMDemoSettings.xml has to be located in the same folder as the executable DemoConsoleApp.exe.
If you modify SMDemoSettings.xml within the open solution in MS Studio, then for the correct connection of xml and xsd it is enough to have SMDemoSettings.xsd and RCNetTypes.xsd open at the same time. If you modify SMDemoSettings.xsd outside the open solution, then you must first add the schemaLocation = "RCNetTypes.xsd" attribute to the "import" element in the SMDemoSettings.xsd schema in the destination directory where the DemoConsoleApp.exe executable is located. Demo application has internally implemented the xml validation so connection of xml and xsd directly on file level is not necessary. But functional external connection of xml and xsd is useful for manual modifications, because consistency checks, annotations, selections from allowed values and checks of the range of entered values are then available instantly.
SMDemoSettings.xml currently includes several classification problems from the:
Anthony Bagnall, Jason Lines, William Vickers and Eamonn Keogh, The UEA & UCR Time Series Classification Repository, www.timeseriesclassification.com
site and State Machine is able to achieve competetive results to the best classification algorithms referenced on that website.
| Dataset | State Machine Accuracy | Best Ref. Accuracy | Best Ref. Algorithm |
|---|---|---|---|
| CricketX | 82.05% | 81.4% | COTE |
| Worms | 83.12% | 73.49% | BOSS |
| BeetleFly | 100% | 94.85% | BOSS |
| BirdChicken | 100% | 98.4% | BOSS |
| ProximalPhalanx | 87.8% | 88.09% | ST |
| Yoga | 91.37% | 90.99% | BOSS |
| Libras | 92.78% | 89.4% | DTWi |
Switches to sub-menu with several code examples showing how to configure State Machine manually from scratch or using the StateMachineDesigner component to perform classification or forecasting, then how to train State Machine and verify its performance.
Switches to sub-menu with several code examples showing how to independently use non-recurrent network components. State Machine's readout layer consists of hirarchical structure of non-recurrent network components, but here is shown how to use them as the standalone components with no relation to State Machine.
This choice runs the Playground module where you can quickly discover and try to use RCNet components.
| Maintenance index | Cyclomatic complexity | Depth of inheritance | Code lines | Executable code lines |
|---|---|---|---|---|
| 82 (green) | 7482 | 3 | 53958 | 11949 |
Components by category:
| Component | Description |
|---|---|
| BasicStat | Implements the basic statistics of sample data. |
| WeightedAvg | Implements the weighted average. |
| MovingDataWindow | Implements the moving data window providing additional functions such as statistics, weighted average, etc. |
| ODENumSolver | Implements a simple numerical solver of the Ordinary Differential Equation(s). |
| Vector | Implements the vector. |
| Matrix | Implements the real matrix. It does not support the sparse matrix format. |
| EVD | Implements the Eigenvalue decomposition of a square matrix. |
| SVD | Implements the Singular Value decomposition of a matrix. |
| QRD | Implements the QR decomposition of a matrix. |
| LUD | Implements the LU (Lowed-Upper) decomposition of a square matrix. |
| ParamValFinder | Implements a simple iterative error-driven search for the parameter's optimal value. |
| HurstExpEstim | Implements the Hurst Exponent estimator using the rescaled range analysis. |
| "RandomValue" | Implements the random value. Supports Uniform, Gaussian, Exponential and Gamma distributions. Here is an extension code |
| Others | Set of small additional helper components like PhysUnit, Interval, Bitwise, Combinatorics, Discrete,... |
| Component | Description |
|---|---|
| DocValidator | Implements the xml document loader and validator. |
| Component | Description |
|---|---|
| PulseGenerator | Implements the constant pulse generator. |
| MackeyGlassGenerator | Implements the Mackey-Glass generator. |
| RandomGenerator | Implements the random signal generator. |
| SinusoidalGenerator | Implements the sinusoidal signal generator. |
| Component | Description |
|---|---|
| BinFeatureFilter | Implements the binary feature filter. |
| RealFeatureFilter | Implements the real number feature filter. |
| Component | Description |
|---|---|
| SimpleQueue | Implements a simple FIFO queue template. Supports access to enqueued elements so it can be also used as the moving data window. |
| DelimitedStringValues | Implements the single row of the delimited string values (csv format). |
| CsvDataHolder | Implements the holder of csv data and supports its loading and saving to a file. |
| VectorBundle | Implements a bundle of input and output data vector pairs. |
| InputPattern | Implements an input pattern. Pattern can be both univariate or multivariate. Supports data resampling (including simple detection of signal begin/end) and amplitude unification. |
| ResultBundle | Implements a bundle of input, computed and ideal (desired) data vector triplets. |
| Component | Description |
|---|---|
| CDivTransformer | Implements the transformer of values from one input field. Divides the constant by the value from the input field. |
| DiffTransformer | Implements the transformer of values from one input field. Subtracts the previous value of the input field from the current value. |
| DivTransformer | Implements the transformer of values from two input fields. Divides the value of the first input field by the value from the second input field. |
| ExpTransformer | Implements the transformer of values from one input field. Raises the fixed base (by default the Euler's number) to the power of the value from the input field. |
| LinearTransformer | Implements the transformer of values from two input fields. Computes the linear equation (aX + bY) where "X" is the value from the first input field and "Y" is the value from the second input field. Coefficients "a" and "b" are specified constants. |
| LogTransformer | Implements the transformer of values from one input field. Computes the input field value's logarithm to the specified base. |
| MulTransformer | Implements the transformer of values from two input fields. Multiplies the value of the first input field by the value from the second input field. |
| MWStatTransformer | Implements the transformer of values from one input field. It keeps statistics of the input field recent values and provides specified statistical figure as the transformed value. |
| PowerTransformer | Implements the transformer of values from one input field. Raises the value from the input field to the power of fixed exponent. |
| YeoJohnsonTransformer | Implements the transformer of values from one input field. Calculates the Yeo-Johnson transformation. See the wiki pages. |
| Component | Description |
|---|---|
| A2SCoderSignalStrength | Implements the signal strength coder. Uses a novel coding algorithm to meet together two important spike-train conditions. Stronger signal leads to earlier first spike and higher spiking frequency. |
| A2SCoderGaussianReceptors | Implements the Gaussian Receptive Fields coder. |
| A2SCoderUpDirArrows | Implements the upward signal direction receptors coder. Each receptor is sensitive to upward signal direction of the current signal against the past signal at the time T-x (where x is 1...number of receptors). The positive difference of current and past signal is then expressed as spikes through a novel coding algorithm meeting together two important spike-train conditions where stronger stimulation leads to earlier first spike and higher spiking frequency. |
| A2SCoderDownDirArrows | Implements the downward signal direction receptors coder. Each receptor is sensitive to downward signal direction of the current signal against the past signal at the time T-x (where x is 1...number of receptors). The negative difference of current and past signal is then expressed as spikes through a novel coding algorithm meeting together two important spike-train conditions where stronger stimulation leads to earlier first spike and higher spiking frequency. |
See the wiki pages.
| Component | Description |
|---|---|
| AFAnalogBentIdentity | Implements the Bent Identity activation function. |
| AFAnalogElliot | Implements the Elliot (aka Softsign) activation function. |
| AFAnalogGaussian | Implements the Gaussian activation function. |
| AFAnalogIdentity | Implements the Identity activation function (aka Linear). |
| AFAnalogISRU | Implements the ISRU (Inverse Square Root Unit) activation function. |
| AFAnalogLeakyReLU | Implements the LeakyReLU (Leaky Rectified Linear Unit) activation function. |
| AFAnalogSigmoid | Implements the Sigmoid activation function. |
| AFAnalogSinc | Implements the Sinc activation function. |
| AFAnalogSinusoid | Implements the Sinusoid activation function. |
| AFAnalogSoftExponential | Implements the Soft Exponential activation function. |
| AFAnalogSoftMax | Implements the Soft Max activation function. |
| AFAnalogSoftPlus | Implements the Soft Plus activation function. |
| AFAnalogSQNL | Implements the Square Nonlinearity activation function. |
| AFAnalogTanH | Implements the Hyperbolic Tangent activation function. |
See the wiki pages.
| Component | Description |
|---|---|
| AFSpikingAdExpIF | Implements the Adaptive Exponential Integrate and Fire neuron model. |
| AFSpikingExpIF | Implements the Exponential Integrate and Fire neuron model. |
| AFSpikingIzhikevichIF | Implements the Izhikevich Integrate and Fire neuron model. |
| AFSpikingLeakyIF | Implements the Leaky Integrate and Fire neuron model. |
| AFSpikingSimpleIF | Implements a very simple form of the Integrate and Fire neuron model. |
| Component | Description |
|---|---|
| FeedForwardNetwork | Implements the Feed Forward network supporting multiple hidden layers. |
| RPropTrainer | Implements the Resilient Backpropagation iRPROP+ trainer of the feed forward network. |
| QRDRegrTrainer | Implements the QRD regression trainer of the feed forward network. The feed forward network to be trained must have no hidden layers and the Identity output activation. |
| RidgeRegrTrainer | Implements the Ridge regression trainer of the feed forward network. The feed forward network to be trained must have no hidden layers and the Identity output activation. |
| ElasticRegrTrainer | Implements the Elastic linear regression trainer of the feed forward network.. The feed forward network to be trained must have no hidden layers and the Identity output activation. |
| ParallelPerceptron | Implements the Parallel Perceptron network. |
| PDeltaRuleTrainer | Implements the P-delta rule trainer of the parallel perceptron network. |
| TNRNet | Implements the holder of trained non-recurrent network and its error statistics. |
| TNRNetBuilder | Implements the builder of the trained non-recurrent network. |
| TNRNetCluster | Implements the cluster of trained non-recurrent networks. |
| TNRNetClusterBuilder | Implements the builder of trained non-recurrent networks cluster based on the cross-validation approach. |
| TNRNetClusterChain | Implements the chain of cooperating non-recurrent network clusters. |
| TNRNetClusterChainBuilder | Implements the builder of the chain of cooperating non-recurrent network clusters. |
| Component | Description |
|---|---|
| AnalogInputNeuron | Implements the input analog neuron. The input analog neuron is a special case of the neuron without an activation function. Its purpose is to provide an analog input for the reservoir's synapses. |
| SpikingInputNeuron | Implements the input spiking neuron. The input spiking neuron is a special case of the neuron without an activation function. Its purpose is to provide a spiking input for the reservoir's synapses. |
| InputEncoder | Implements a mediation layer between the external input data and the internal reservoirs of the neural preprocessor. Processes the external input data in the natural form and provides it's representation on analog and spiking input neurons for the next processing in the reservoirs. Allows to create new computed input fields using chainable transformations of existing external input fields, as well as adding independently generated input fields using various generators. Supports two input feeding modes: Continuous and Patterned. The Continuous feeding mode processes an input vector as the variable(s) values at the single time-point T. The Patterned feeding mode processes an input vector as an alone input pattern consisting of a time series of the variable(s) values. Supports three ways how to represent an analog value as the spikes: Horizontal, Vertical or Forbidden. The Horizontal way of coding means a simultaneous activity of the neuronal population where every input field is coded by several spiking input neurons (a horizontal spike-train). It is fast, it leads to a single computation cycle of the reservoirs per the input field value. The Vertical way of coding means that the input field value is coded as a spike-train on a single spiking input neuron. It is slower, it leads to multiple computation cycles of the reservoirs according to the spike-train length. The Forbidden way of coding means there is no coding of an analog value as the spikes. It is fast, it leads to a single computation cycle of the reservoirs per the input field value and it does not utilize any spiking input neuron(s). |
| Synapse | Implements the synapse. Supports the signal delaying and the short-term plasticity. |
| PredictorsProvider | Implements the provider of the unified set of computed predictors. |
| HiddenNeuron | Implements the hidden neuron. Supports engagement of both analog and spiking activation functions and provides unified set of available predictors. |
| ReservoirInstance | Implements the reservoir. |
| NeuralPreprocessor | Implements the neural preprocessor supporting multiple reservoirs. |
| ReadoutUnit | Implements the readout unit of the readout layer. It can do the Forecast or Classification. |
| OneTakesAllGroup | Implements the "One Takes All" group of readout units. Supports basic decision-making based directly on the results of readout units and also more advanced decision-making based on the result of a dedicated chain of network clusters. |
| PredictorsMapper | Implements the mapper of specific predictors to readout units. |
| ReadoutLayer | Implements the readout layer consisting of trained readout units (and "One Takes All" groups). |
The main serializable StateMachine component encapsulates the NeuralPreprocessor and ReadoutLayer components into the single component and adds support for routing of specific predictors and input fields to the specific readout units. State Machine also allows to bypass NeuralPreprocessor and to use input data directly as a predictors for the readout layer.
Each executive component that makes up StateMachine (including StateMachine itself) has its own related settings class providing configuration, which is required by the executive component's constructor.
Each settings class can be instantiated manually from scratch or from a xml element encapsulating all parameters. RCNetTypes.xsd defines all xml elements used in settings classes constructors.
Each settings class also implements the GetXml method so it can be instantiated from scratch and the initialization xml element can be exported by calling the GetXml method (and stored for later use). Using xml constructors is generally preferable because the initialization xml can be edited without the need to modify source code of the manual setup.
To make things easier, RCNet also implements helper component StateMachineDesigner for easier setup of simple ESN and LSM StateMachine configurations from the code (see the examples in demo application).
More detailed documentation will be gradually published here. Questions, ideas, suggestions for improvement and constructive comments are welcome at my email address oldrich.kozelsky@email.cz or newly you can use github discussions.