Shrink Pt. 1: Sparsify

[stephenjfox]

• Prove that sparsify(tensor, threshold) performs as intended
• Prove shrink on a given layer produces the correct count of neurons for a given layer
  • ... for all layers of a given nn.Module

I recommend an internal encapsulation to handle this relaying of information.

• Simple: a named tuple for carrying the data
• Complex: a small control class that makes focused decisions

[stephenjfox]

Definition of "correct count of neurons":

• The smallest, rounded-down value that can be computed
• Example: given a Linear(5, 4)
  1. We have a 4x5 matrix (PyTorch represents the transpose internally for ease of computation)
  2. We compute the sparsity with morph.sparse.sparsify. This give us a float-as-percentage
  • In this example, 40%
  3. We can safely multiply the last dimension of our matrix by .6 -> 4x3 matrix.
  • No rounding necessary
  • BUT THIS CHANGES OUR INPUT SIZE. What do we do?
  4. Consider making 4x5 -> 2x5, as the alternative (3x5) has more parameters and we want to trim down as quickly as possible

---

Is this comment still relevant?