How does ReLU affect metric entropy?
ranyishere opened this issue · 1 comments
ranyishere commented
Related to: #3
How do you relate the decrease in point sizes to holes? We are inclined to look into doubling dimension and covering number as a way to explain distance and holes.
mathemonads commented
The n-dimensional ball has doubling dimension n. Situated in the appropriate orthant, ReLU transforms it to a metric space with doubling dimension 0.
Similarly, ReLU can increase the doubling dimension by 1, e.g. {(x,x+1) : for all real numbers x}.
.
Both of these results are yet to be proven explicitly.