CRITICAL failure with latest version (08047c)

[hugorodgerbrown]

Apologies in advance of the lack of detail in this issue, but as you'll see I don't have much!

On Friday I deployed a version of our app, only to find that certain sections of our site caused our app to hang; and once hung only a restart of the app could recover it. A carpet-bombing of debug statement later it appears that the problem was caused by any template that included the {% assets %} instruction. This was a new problem (and we deploy multiple times a day) that only began yesterday. Further investigation showed that we were including the latest build in our deployment using the pip '-e' flag, which suggests that the problem lies the 08407c4 commit.

The symptoms were simple - a complete lack of logging / debug output, but a consistent (and repeatable) timeout from our reverse proxy (which cut the connection at 30s). It just died. And once dead, would not recover. Looking at the commit, it seems like something happens on a separate thread that gets stuck.

Some details:

• We are running our app on Heroku (hence the 30s router timeout).
• Python 2.7.4
• Django 1.4.3
• django-assets included in requirements.txt using '-e' from the github repo

The output from our Heroku instance:

2013-10-27 09:08:49.891 269 <158>1 2013-10-27T09:08:49.358189+00:00 heroku router - - at=error code=H12 desc="Request timeout" method=GET path=/ host=[redacted].herokuapp.com fwd="2.102.222.71" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0

The issue was solved by fixing the deployment to v0.8 of django-assets in our requirements.txt:

django-assets==0.8

instead of

-e git://github.com/miracle2k/django-assets.git#egg=django-assets

Sorry for not having any more detail on exactly what the problem was, but I can confirm that this behaviour can be reproduced locally (running on Ubuntu 12.04, 64-bit,running on VirtualBox as a Vagrant VM).

In terms of timing, this is the output from running heroku releases on our affected app:

v214  Deploy b82f8be    hugo@rodger-brown.com  2013/10/27 17:56:22 <-- FIXED
v213  Rollback to v211  hugo@rodger-brown.com  2013/10/26 12:09:01
v212  Deploy 4dc11fe    hugo@rodger-brown.com  2013/10/26 12:00:29 <-- DOES NOT WORK
v211  Deploy 6a60784    hugo@rodger-brown.com  2013/10/25 09:56:19

The problem occurred in deployment 4dc11fe, which we immediately rolled back to 6a6o784.

[miracle2k]

Turns out this was a pretty stupid mistake on my part to make.

[hugorodgerbrown]

Thanks for the prompt update.