Introduction - formalization of optimization: What is meant by "mathematical optimization"?

[SimonKruse0]

Regarding the "Introduction - formalization of optimization" section.
I really like the idea of generalizing the field mathematical optimization as much as possible.
However, I am a bit confused that "mathematical optimization" is described in the following sentence as a "sequence of experiments"

Mathematical optimization instead takes an indirect approach: we design a sequence of experiments to probe the objective function for information that, we hope, will reveal the solution to (1.1).
The iterative procedure in algorithm 1.1 formalizes this process.[...]

What about a direct solution of a optimization problem - e.g. normal equations solution in linear least squares? And how does algorithms like gradient descent fit into the algorithm 1.1 framework?

Maybe what is meant is "sequential optimization" instead of "Mathematical optimization"?

[bayesoptbook]

I suppose this is a question of (admittedly blurry) semantics. Wikipedia describes "mathematical optimization" (https://en.wikipedia.org/wiki/Mathematical_optimization) as "consist[ing] of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function." It is in this sense that I mean to use the term, and I believe this likely agrees with your (and the book's) interpretation of "sequential optimization." But yes, perhaps the phrase is best avoided.

Direct solves are mentioned at the beginning of that paragraph as being infeasible in many scenarios.

Gradient (ascent) fits into algorithm 1.1 fairly neatly: the "observe" step returns the function value and the gradient (in which case perhaps y would be a better notation for the result), and the "policy" step returns x + γ∇ as the location for the next evaluation.

[SimonKruse0]

Okay, I see. But there is a "In the simplest case" just before that Wikipedia formulation you quote. And after the formulation:
"More generally, optimization includes finding "best available" values of some objective function given a defined domain (or input) [...]"

In the simplest case, an optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function. The generalization of optimization theory and techniques to other formulations constitutes a large area of applied mathematics. More generally, optimization includes finding "best available" values of some objective function given a defined domain (or input), including a variety of different types of objective functions and different types of domains.

In my eyes mathematical optimization would be the class of all methodologies which solves a mathematical optimization problem - I just found that defined the Convex Optimization book:

So directly solving least squares min_x ||Ax-b||^2_2 (which is a mathematical optimization problem) with the normal equations, should belong in the mathematical optimization class.

Thank you for your clarification of how to frame gradient ascent/descent.

[bayesoptbook]

I will make a note to take a second look at this passage. Problems that can be solved directly are not typically studied under the banner of "mathematical optimization," but rather the development of procedures that can aid in solving those problems that can't be solved directly. Indeed, that is the bulk of the focus in the convex optimization book, as it is here.

Perhaps more precise would be "for those problems that cannot be solved directly, mathematical optimization instead takes an indirect approach..." I suppose that should be unobjectionable.