Discovering ketosis: how to effectively lose weight

Here is a chart of my weight vs. time in the past 16 months or so:

The chart was generated from the data file weight.2015.csv by the script date-weight.r in this git repository. It requires R and ggplot2.

In the following I'll describe the thought process, some other people ideas, and the software I used to lead me in the right direction.

Disclaimer:

The below is what worked for me. Your situation may be different. Listen to your own body.

OK, with that behind us, let's now channel Galileo in the face of the inquisition: evolution has been hard at work for over 2 billion years shaping the chemistry of all eukaryotes, multi-cellular life and eventually mammals. The Krebs cycle, glucose metabolism, insulin spikes, glycogen in the liver, carnitine, lipase, are as real for you as they are for me. We may be very different in our genes and traits, some are more insulin resistant, for example, but we cannot be too different in our most fundamental metabolic chemistry. The chemistry which drives fat synthesis and its reverse.

Salient facts & initial observations

I used to be a pretty thin person. My 1st DMV card below, says 143 lb.
Unfortunately, since moving to the US, I've been gaining more and more weight. I peaked in 2015, over 50 lbs higher.
The US is a country where obesity is an epidemic.
Poorer demographics in the US have higher levels of obesity.

Does a US typical lifestyle has anything to do with this epidemic? After reading on the subject, I could point at a few of the main suspects:

Fast food is highly available, and is very cheap compared to most alternatives
Most food we buy and eat is heavily processed -- watch Food, Inc. (documentary)
"No Fat" and "Low Fat" labels are everywhere on supermarket shelves
Many foods are enriched and sweetened with high-fructose corn-syrup -- watch Sugar Coated (documentary)

As in many other instances, I realized I need to think for myself. Ignore all "expert" advice. Question widely accepted ideas like the FDA "food pyramid". Start listening to my own body, my own logic & data I can collect myself and trust.

Once I did, the results followed.

What didn't work

In the past, I tried several times to change my diet. After reading one of Atkins' books, I realized, checked, and accepted the fact that excess carbs are a major factor in gaining weight. But that realization alone has not led to success.

My will power, apparently, was insufficient. I had too much love of pizza and bread. I would reduce my carb consumption, lose a few pounds (typically ~5 pounds), and then break-down, go back to consuming excess carbs, and gain all these pounds back, and then some. My longest diet stretch lasted just a few months.

It was obvious that something was missing in my method. I just had to find it. I could increase my physical activity, say start training for a mini-marathon, but that's not something I felt comfortable with.

I realized early on that I need to adopt a lifestyle that not just reduces carbs, or add exercise, but is also sustainable and even enjoyable so it can turn into a painless routine. Something that:

I could do for years

Never feel the urge to break habits

Is not hard, or unpleasant for me to do

Early insights & eureka moments

Early in the process I figured I could use machine learning to identify the factors that made me gain or lose weight. I used a simple method: every morning I would weigh myself, and record both the new weights and whatever I did in the past ~24 hours, not just the food I ate, but also whether I exercised, slept too little or too much, etc.

The file I kept was fairly simple. A CSV with 3 columns:

Date, MorningWeight, Yesterday's lifestyle/food/actions

The last column is a arbitrary-length list of word[:weight] items.

The (optional) numerical-weight following :, expresses higher/lower quantities. The default weight, when missing is 1:

#
# -- Comment lines (ignored)
#
Date,MorningWeight,YesterdayFactors
2012-06-10,185.0,
2012-06-11,182.6,salad sleep bacon cheese tea halfnhalf icecream
2012-06-12,181.0,sleep egg
2012-06-13,183.6,mottsfruitsnack:2 pizza:0.5 bread:0.5 date:3 dietsnapple splenda milk nosleep
2012-06-14,183.6,coffeecandy:2 egg mayo cheese:2 rice meat bread:0.5 peanut:0.4
2012-06-15,183.4,meat sugarlesscandy salad cherry:4 bread:0 dietsnapple:0.5 egg mayo oliveoil
2012-06-16,183.6,caprise bread grape:0.2 pasadena sugaryogurt dietsnapple:0.5 peanut:0.4 hotdog
2012-06-17,182.6,grape meat pistachio:5 peanut:5 cheese sorbet:5 orangejuice:2
# and so on ...

Then I wrote a script to convert this file to vowpal-wabbit training-set regression format. In the converted train-set the label (target feature) is the change in weight (delta) in the past 24 hours, and the input features are what I've done or ate in the ~24 hours leading to this delta -- a straight copy of the 3rd column.

I was not dieting at that time. Just collecting data.

The machine learning process error-convergence after partly sorting the lines descending, by abs(delta) to smooth it out and try to amplify very weak signals from the data, and 4-passes over the data, looks like this:

You can reproduce my work by compiling your own data-file, installing all prerequisites, and running make in this directory. I wrote a HOWTO file with more detailed instructions. Please open an issue, if anything doesn't work for you.

When you type make in this directory -- some magic happens.

Here's how a typical result looks like.

$ make

... (output trimmed for brevity) ...

FeatureName       HashVal   ...   Weight RelScore
nosleep            143407   ...  +0.6654 90.29%
melon              234655   ...  +0.4636 62.91%
sugarlemonade      203375   ...  +0.3975 53.94%
trailmix           174671   ...  +0.3362 45.63%
bread              135055   ...  +0.3345 45.40%
caramelizedwalnut  148079   ...  +0.3316 44.99%
bun                  1791   ...  +0.3094 41.98%

... (trimmed for brevity. Caveat: data is too noisy anyway) ...

stayhome           148879   ...  -0.2690 -36.50%
bacon               64431   ...  -0.2998 -40.69%
egg                197743   ...  -0.3221 -43.70%
parmesan             3119   ...  -0.3385 -45.94%
oliveoil           156831   ...  -0.3754 -50.95%
halfnhalf          171855   ...  -0.4673 -63.41%
sleep              127071   ...  -0.7369 -100.00%

The positive (top) relative-score values are life-style choices that make you gain weight, while the negative ones (bottom) make you lose weight.

And here's a variable-importance chart made from a similar data-set:

Big disclaimer: please don't read too much into the particulars of this data. Working with this particular data set, was pretty challenging, since:

The number of original data-points (days) is too small to establish enough significance.
Typical daily changes in body weight are very small, often ~0.1 lb.
My scales are not accurate: you may note that my data has 0.2 pound resolution. This is not ideal. Getting scales with 0.1 pound resolution is highly recommended.
You may also note that the loss-convergence chart hits a hard floor at ~0.2 even when you do multiple-passes over the data (overfit the training-set) for a similar reason.
Items that make you lose and gain weight, often appear together on the same line so they cancel each other. This throws the automatic learning process off-course.

So I focused mostly on the extremes (start and end) of the list as presented above, and just used the hints as general guidance for further study, experimentation, and action.

Despite the noisy & insufficient data, and the inaccuracies in weighting, the machine-learning experiments made 4 facts obvious, pretty early:

Sleeping longer consistently appeared as the #1 factor in losing weight.
Lack of sleep did the opposite: too little sleep lead to weight gains.
Carbs made me gain weight. The worst were high-starch and sugary foods.
Fatty and oily foods did the opposite: they were positively correlated with weight-loss.

The 'stayhome' lifestlye, which fell mostly on weekends, may have been a red-herring: I slept longer when I didn't have to commute to work, OTOH: my diet on stay-home days may have been different.

It took me a while to figure out the sleep part. When we sleep we don't eat. It is that simple.

Moreover: we tend to binge and snack while not particularly hungry, but we never do it during sleep.

Our sleeping time is our longest daily fasting time.

Please note that my explanations of the effects may not in fact be accurate or deeply scientific. The goal of all this was incremental discovery: experiment, check effect, rinse, repeat.

Further progress

You may note that in the top (date vs. weight) chart there's a notable acceleration in the rate of weight-loss. The cause was deeper insights and better ability to sustain the diet the more I understood the problem.

Extending the fasting time was one major accelerator of weight-loss rate. I did that by:

Skipping breakfast and

Stop eating earlier in the evening before going to bed.

This gave me 14-16 hours of fasting each day. Rather than the more typical 10-12 hours/day of fasting.

The 2nd accelerator was loading up on fat in order to feel full.

The 3rd accelerator was understanding the concepts of Glycemic index and Glycemic Load, and shifting whatever I chose to eat towards lower Glycemic loads.

I now believe and hope that I can go all the way back to my original weight when I first landed on US soil.

If I can keep the present rate, it should take 1-2 years to completely reverse the damage of the past ~20 years.

It is important to stress that I also feel much better the more weight I lose. As a welcome side-effect, the few borderline/high levels in my blood tests, have moved significantly towards normal averages, during the period I lost weight.

What was my data and clear improvement in health saying?

Looking at my data, and reading more, convinced me that I should beware of doctors who push statins instead of suggesting a better diet. I started doubting anyone who told me I need to reduce fat. I run away if anyone now tells me "high cholesterol" in the diet is dangerous.

Cholesterol, by the way, is an essential building block for many essential body by-products. The liver produces as much cholesterol as we need.

Our body is an amazing machine. Billions of years of evolution have made it extremely adaptive.

It is not our high fat consumption, it is the storage of fat process that makes us acummulate fat in the tissues and become unhealthy.

An enzyme called Lipase breaks-up fat. Raise the levels of Lipase and our body fat gets consumed faster. To get there, we need to give the body fat as an alternative to carbohydrates. When the body has depleted both the blood sugar, and the glycogen (hydrated sugar) buffer in the liver, it has no other choice but to adapt and compensate. Our source of energy -- ATP synthesis -- switches from carbs to fats by producing more fat-breaking agents. The body is a "Flex Fuel" kind of machine, that has simply replaced one fuel (carbs) with another (fat).

When Lipase, and all other agents in the fat-to-ATP chemical path, aka Beta oxidation mobilize, and their levels are elevated, we burn more fat and lose weight over time.

In a low-carb/high-fat (LCHF) regime, our night sleep (fasting time) becomes our friend. The fat-breaking agents keep working while we sleep, breaking-up the stored fat. This leads to weight-loss, and a healthier state.

And when we push even further, and cut carbs to really low levels, we may reach a new steady state, called ketosis, in which practically all our energy comes from fat, and that's when we really win big in the weight-loss battle.

The above is a very simplified, and hopefuly easy to digest, version of what some diet books try to explain in hundreds of pages.

My bottom-line recipe:

The hardest part (especially at the beginning) is reducing carbs. The worst are starch rich foods (pizza, pasta, bread etc.), then processed foods with high sugar content (sweet sodas, no-pulp juices, etc). This doesn't mean no carbs. You may afford yourself carbs from time to time (say a pizza once a week). As it turns out, an occasional lapse isn't enough to completely reverse any steady-state. However, you need to make sure you consume much less carbs and less frequently than before. In particular, you must avoid binging on snacks like chips, pizza, doughnuts, pasta, and bread, or drinking sugar-rich drinks.
Look-up Glycemic index and Glycemic Load on wikipedia. Avoid foods with high glycemic load. This prevents the blood sugar spikes which lead to insulin spikes and tell the body chemical cycles to revert back from ketosis, or near ketosis, to fat-accumulation. Have a sweet tooth? Eat an orange instead of drinking orange juice. The two have vastly different glycemic loads and this makes a huge difference. If you must add sweetness to your cup of tea or coffee, use a Splenda (sucralose+dextrose) tablet, or a Stevia drop/tablet which typically weight just ~0.1 gram, rather than a tea-spoon of sugar (~4.2g, about 40x more). Result: similar sweetness effect, but much lower Glycemic load and resulting levels of blood-glucose.
High fat: I switched from milk to half-and-half and am considering heavy (and unsweetened) whipped cream. It has less carbs (lactose) and more fat; plus, it tastes better. Eat avocados, olive oil, mayo, coconut oil, nuts. I never worry about natural fat, I eat as much fat as I want. This is what makes it much easier to avoid carbs. When I stuff myself with fat I feel much less hungry and miss the carbs less. The body is very good at figuring this out: "I have too much fat in the blood, so let's increase the amount of enzymes which break-up fat" and this makes me lose weight in the long run. Most importantly, I always avoid any products labeled "low-fat" or "fat-free". The food industry usually replaces fat with sugar, so it tastes better - otherwise it tastes awful. You'll often hear about "bad" vs "good" fat. My take: as long as it is natural, it is ok. The worst trans-fat is fat that's artificially hydrogenated, to increase shelf-life, by the food industry. The less saturated fat is, the better. Mono-saturated (plant) liquid oil is the best, then come the poly-unsaturated fats, and finally near saturated (but not fully saturated) fats that come from animals. My buttery-spread spectrum is: Margarine: no; Butter: ok; Earth Balance: no problem. At any rate, even the most saturated fat, gets broken and depleted by the natural processes in the body.
A bit of exercise. Of course, more is better, but for many this may prove difficult. I don't excercise too much. I just bike to work and back about 20 min each way, meaning 40 min/day, 5 out of 7 days/week. You can try walking the dog (but walk faster), or Zumba dance to music. The trick is to find something that you don't find hard to do. Or find company to do it together. Then, do a little bit of it every day. For the record, I'm not 100% sure excercise is absolutely essential to losing weight, but I believe it helps accelerate the process.
Longer fasting periods: This is the #1 contributor to weight-loss. sleep longer, stop eating as early as possible before going to sleep and start eating as late as possible after sleeping. Skip breakfast, after some time you won't feel hungry in the morning anymore. After long periods of fasting, the body chemistry adjusts: I need more ATP, but I don't have glucose in the blood, and the glycogen in the liver is all fully consumed (takes about 1-2 days of low or no carbs) so there's no other option, but to start breaking-up stored fat. This elevates the enzymes that help with breaking up fat and the Krebs cycle reverses direction in the critical paths. Instead of transforming excess-carbs into stored fat, it breaks-up the fat for energy.
Eat eggs. They are a wonderful combo of fat and protein with no carbs at all. I read an interview with a Japanese woman who reached 114 years and one of her secrets was to eat eggs daily. My favorite food is a scrambled egg with grilled onions (onions are a bit high on carbs, but too tasty to give up) and olives.
Eat slower, and chew longer... don't swallow just yet! Humans, just like dogs, tend to swallow too soon. Stop eating when you feel full. There's about 20 min delay before your brain registers that you are full so don't over-eat.

Acknowledgements

Big thanks to the following people for contributing to this project in myriad ways, comments, references, corrections, etc.

Anat Faigon, Ingrid Kane, Hans Lee, Steve Malmskog, Eyal Friedman, Shiri Shoham, Gabi Harel, Shingi, Noa

Update: 2016-08-12: this project made Hacker News and reached the top place for a while. Thanks for some great comments by benkuhn, aab0, zzleeper, and others which helped me make it better.

davidrichards/weight-loss

Discovering ketosis: how to effectively lose weight

Here is a chart of my weight vs. time in the past 16 months or so:

Disclaimer:

Salient facts & initial observations

What didn't work

Early insights & eureka moments

And here's a variable-importance chart made from a similar data-set:

Further progress

What was my data and clear improvement in health saying?

My bottom-line recipe:

Further reading:

Documentaries:

More videos

Questions, Answers, Comments

More friendly interface

Acknowledgements