"A statistical language model assigns a probability to a sequence of m words P(w_1,\ldots,w_m) by means of a probability distribution." (Wikipedia)
This module provides a unigram-based cross-lingual language model, inspired by: Leuski Anton, Traum David. A Statistical Approach for Text Processing in Virtual Humans tech. rep.University of Southern California, Institute for Creative Technologies 2008.
A CrossLanguageModel can be used as a multi-class classifier/ranker, but, it also takes into account the structure of the output classes.
npm install languagemodel
var CrossLanguageModel = require('languagemodel').CrossLanguageModel;
var model = new CrossLanguageModel({
smoothingFactor : 0.9
});
model.trainBatch([
{input: {i:1, want:1, aa:1}, output: {a:1}},
{input: {i:1, want:1, bb:1}, output: {b:1}},
{input: {i:1, want:1, cc:1}, output: {c:1}},
]);
console.dir(model.similarities({i:1, want:1, aa:1, and:1, bb:1}));
==>
[ { output: { a: 1 }, similarity: -0.29506393100486217 },
{ output: { b: 1 }, similarity: -0.29506393100486217 },
{ output: { c: 1 }, similarity: -0.6413224201123425 } ]