delphidabbler/code-snippets

Add LogSumExp functions

Closed this issue · 3 comments

LogSumExp

LogSumExp or LSE for a sequence x1,...,xn is is defined as the logarithm of the sum of the exponentials of the arguments:

LSE(x1,...,xn) = log(exp(x1) + ... + exp(xn))

Source: Wikipedia

function LSE(const A: array of Extended): Extended;
begin
  var Sum: Extended := 0.0;
  for var X in A do
    Sum := Sum + Exp(X):
  Result := Ln(Sum);
end;

Strictly Convex LogSumExp

Screenshot_20230720-040354_Chrome

Source: Wikipedia

Since exp(0) = e^0 = 1, we have

LSE(0, x1,...,xn) = log(exp(0) + exp(x1) + ... + exp(xn))
                  = log(1 + exp(x1) + ... + exp(xn)),

So, in Pascal we can have:

function ConvexLSE(const A: array of Extended): Extended;
begin
  var Sum: Extended := 1.0; // for additional Exp(0) term
  for var X in A do
    Sum := Sum + Exp(X):
  Result := Ln(Sum);
end;

This issue was extracted from issue #16

The Softmax function is closely related to LSE. This should also be implemented.

Strictly Convex LogSumExp

Decided not to implement this.

Implemented by merge commit 4186f6e