{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Turn TensorFlow functions into mathematical notations and diagrams"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is based on some helper classes I started writing,\n",
"to help my self make less mistakes and understand the code better.\n",
"It is still work in progress.\n",
"I want to share the idea to see if it interests others."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**TLDR**: it is a bunch of helper functions that can act\n",
"as a direct replacement of tensorflow. It renders mathematical formulas\n",
"and output tensor dimensions. I'm looking for feedback."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example\n",
"\n",
"\n",
"There's a bunch of things I've been working on.\n",
"In order to show them I've used the initial model creation code of\n",
"[this tensorflow implementation of capsule nets](https://github.com/ageron/handson-ml/blob/master/extra_capsnets.ipynb).\n",
"*Starting from **Input Images** upto **Digit Capsules**.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Import the library. I'm trying to make it work as a direct replacement."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from for_beginners import tensorflow_wrapper as tf\n",
"from for_beginners.model import Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All the operations and variables are assigned to a model.\n",
"This takes care of variable names, so that you don't have \n",
"to explicitly specify `name=`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Model()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a place holder just like you do with [TensorFlow](https://www.tensorflow.org),\n",
"and assign it to `model`. The library will set the tensorflow name, and output a nice\n",
"3D diagram showing the dimensions of the placeholder."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.X = tf.placeholder(shape=[None, 28, 28, 1], dtype=tf.float32)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The dimensions of the tensor is visualized in 3D.\n",
"This is more convenient than having to write an extra line\n",
"to output the shape of $X$.\n",
"I'm not sure if this is the best way to visualize;\n",
"I have discussed some other ideas later.\n",
"\n",
"Also notice that we didn't pass the variable name as an argument;\n",
"it was extracted from the assigned property name.\n",
"The cost of this is that we have to use a `model` to store all\n",
"variables and operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"caps1_n_maps = 32\n",
"caps1_n_caps = caps1_n_maps * 6 * 6 # 1152 primary capsules\n",
"caps1_n_dims = 8"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"conv1_params = {\n",
" \"filters\": 256,\n",
" \"kernel_size\": 9,\n",
" \"strides\": 1,\n",
" \"padding\": \"valid\",\n",
" \"activation\": tf.nn.relu,\n",
"}\n",
"\n",
"conv2_params = {\n",
" \"filters\": caps1_n_maps * caps1_n_dims, # 256 convolutional filters\n",
" \"kernel_size\": 9,\n",
" \"strides\": 2,\n",
" \"padding\": \"valid\",\n",
" \"activation\": tf.nn.relu\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can add `conv2d` layers similarly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.conv1 = tf.layers.conv2d(model.X, **conv1_params)\n",
"model.conv2 = tf.layers.conv2d(model.conv1, **conv2_params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Above diagram shows that there are `6x6` feature maps of `256` channels after the second convolution.\n",
"I'm planning to indicate the *strides*, *kernel size* and *padding* on the 2-D diagram."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.caps1_raw = tf.reshape(model.conv2, [-1, caps1_n_caps, caps1_n_dims])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's a function definition.\n",
"You can add a naming scope,\n",
"so that it will not render internal operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def squash(s, axis=-1, epsilon=tf.constant(1e-7), name=None):\n",
" model.epsilon = epsilon\n",
" if name is None:\n",
" name = 'squash'\n",
" model.add_scope(name)\n",
" model.squared_norm = tf.reduce_sum(tf.square(s), axis=axis,\n",
" keep_dims=True)\n",
" model.safe_norm = tf.sqrt(tf.add(model.squared_norm, epsilon))\n",
" model.squash_factor = tf.divide(model.squared_norm, tf.add(1., model.squared_norm))\n",
" model.unit_vector = tf.divide(s, model.safe_norm)\n",
" model.squashed = tf.multiply(model.squash_factor, model.unit_vector)\n",
"\n",
" model.remove_scope() # Need to implement with mode.scope ....\n",
" \n",
" return model.squashed "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Call to the function will render the full formula and output tensor dimensions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.caps1_output = squash(model.caps1_raw, name=\"caps1_squash\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"This helps you verify the formulas when coding.\n",
"It help readers quickly understand whats going on,\n",
"without having to look back at the function definition.\n",
"\n",
"Notice that, tries to convert the variable names into mathematical symbols;\n",
"e.g. `epsilon` -> $\\epsilon$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why?\n",
"\n",
"This was developed to make machine learning code easier to understand,\n",
"at the time of writing as well as for readers.\n",
"I have only tried this on small networks (mostly based on tutorials).\n",
"So far I've found it helpful.\n",
"\n",
"I feel it's helpful for beginners than for experts,\n",
"who get confused on the detailed workings of the functions.\n",
"This is espcially true for me because I only work on tensorflow\n",
"on and off, and I forget all the details.\n",
"\n",
"It is also useful when writing notebooks intended for others to read (like tutorials)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What's next?\n",
"\n",
"The above example shows most of the things I've implemented so far.\n",
"It's just a couple of hunderd lines of python and javascript I've written on free time.\n",
"I am planning to cleanup the code and host it on github.\n",
"Right now the code is messy, and needs to be fixed almost everytime I use it.\n",
"\n",
"### Cover more tensorflow operations\n",
"\n",
"Current implementaion is only for a few common tensorflow functions.\n",
"\n",
"Also, the library doesn't do operator overloading which is really important.\n",
"That is, we should be able to write `X / Y`\n",
"instead of `tf.divide(X, Y)`\n",
"\n",
"### Diagrams to how operations work\n",
"\n",
"We need to include more details in diagrams to explain complex operation.\n",
"For instance, we can *indicate padding, strides and kernal size\n",
"in the diagram for convolution operation*.\n",
"We can do similar diagrams for tensor transformations.\n",
"\n",
"Also, the current 2-D and 3-D tensor diagrams might \n",
"not be the best in all situations.\n",
"It sometimes makes more sense not to show individual cells,\n",
"butto show the sizes of the dimensions.\n",
"\n",
"Why I decided to show individual cells is with the intention of\n",
"adding details like highlighted cells, kernels, etc on the same diagram.\n",
"\n",
"\n",
"\n",
"It's sometimes important to show higher dimensianality,\n",
"especially for operations like reshaping and tiling.\n",
"\n",
"\n",
"\n",
"And, on some occasions,\n",
"it may be just enough to show the shape of the tensor in text;\n",
"e.g. `([?, 10, 50, 20, 10])`.\n",
"\n",
"### Function Wrappers\n",
"\n",
"For example, we can wrap the function `squash`.\n",
"\n",
" wrap(tensor(['N', 'H', 'W'])\n",
" def squash(s, axis=-1, epsilon=Noob(tf.constant, 1e-7), name=None):\n",
" ...\n",
" \n",
"And it can display the full formula for the function and the output tensor dimensions.\n",
"This will help us test functions before calling them, and help readers get an overview\n",
"of the function without reading the internals. \n",
"\n",
"### Please give your feedback. Find me on twitter [@vpj](https://twitter.com/vpj).\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}