Internal Error in stats view

Question

Internal Error in stats view

tonioo opened this issue 9 years ago · 20 comments

From @fionnb on February 18, 2016 20:25

Hi, on a fresh 1.4.3 install I get errors in the stats module.

File "modoboa_stats/graphics.py", line 94, in export
    timestamp = int(row.find('t').text)

AttributeError: 'NoneType' object has no attribute 'text'

i went digging a bit and had it dump the command, which is:

rrdtool xport --start -1week --end 1455825274 DEF:size_recv=/mail/modoboa/stats/global.rrd:size_recv:AVERAGE CDEF:size_recvpm=size_recv,60,* XPORT:size_recvpm:"Empfangsgr\xc3\xb6sse" DEF:size_sent=/mail/modoboa/stats/global.rrd:size_sent:AVERAGE CDEF:size_sentpm=size_sent,60,* XPORT:size_sentpm:"Versendegr\xc3\xb6sse"

Apparently, when using this command, the result will not contain any 't' fields:

<data>
    <row><v>3.9836091071e+05</v><v>6.2045392857e+04</v></row>
  ...

... and the subsequent code fails. I wonder what goes wrong while constructing the command?

Can I do anything to make it work properly?

Using rrdtool 1.5.5 btw.

Copied from original issue: modoboa/modoboa#836

Answer 1 · 2016-02-22T10:34:04.000Z

Indeed, it looks pretty strange... Could you try with an older version of rrdtool (like 1.4.7) ?

Answer 2 · 2016-02-22T10:34:04.000Z

From @fionnb on February 21, 2016 16:30

confirmed: appears in 1.4.7 and is missing in 1.5.5.

Answer 3 · 2016-02-22T10:34:04.000Z

Looks like the output is a bit different with 1.5.5. Anyway, 1.5.5 should have a working JSON output (unlike 1.4.7), I'll make a change to use it if present.

Answer 4 · 2016-02-22T10:34:05.000Z

From @fionnb on February 22, 2016 9:31

I get it that 1.4.4 does not have the fix yet?

Answer 5 · 2016-02-22T10:34:05.000Z

You're right. The stats plugin has its own repository (https://github.com/modoboa/modoboa-stats) so this issue is misplaced.

Answer 6 · 2016-02-22T10:34:06.000Z

From @fionnb on February 22, 2016 10:27

I am sorry then. Should I open an issue there or just wait for an update to appear?
Feel free to close this issue if inappropriate.

Answer 7 · 2016-02-22T10:34:06.000Z

Don't worry, I'm going to move this issue.

Answer 8 · 2016-05-06T11:36:48.000Z

The above output shows that the timestamps are missing. This is true for the JSON output as well.

Answer 9 · 2016-05-06T11:58:15.000Z

I had a look at the rrd codebase and recognized that it's required to use the --showtime flag to get timestamps in the output.

The fix is easy:

Change line 103 from

        cmd = "{} xport --start {} --end {} ".format(

to

        cmd = "{} xport --start {} --end {} --showtime ".format(

Answer 10 · 2016-05-08T08:50:28.000Z

@awein Great, thank you ! I guess this option has been added to rrdtool 1.5.x, right ?

Answer 11 · 2016-05-08T09:52:50.000Z

@tonioo on the changelog, http://oss.oetiker.ch/rrdtool/pub/CHANGES

RRDtool 1.6.0 - 2016-04-19
==========================
* re-introducted --showtime option on rrdxport

I don't see in the changelog, when this option have been removed

Answer 12 · 2016-05-19T15:13:29.000Z

Too bad ubuntu 16.04 still has 1.5.5 and that version does not have the showtime option :-(

Answer 13 · 2016-06-15T17:17:49.000Z

Helo Guys,

first...sorry my bad english : |

Im getting a message "internal error" when i try to see the stats into modoboa 1.5.3

I tried edit the file graphics.py removing the option "extra_args" and "{}" to try to solve the problem but I stiil getting the message and the graphics dont show up.

Im using Ubuntu 16.04 with rrdtool 1.5.5

As fionnb told, the rrdtool 1.5.5 doesnt have the showtime option

this is part of my current graphics.py:
.
.
.
"""Export data to XML using rrdtool and convert it to JSON."""
result = []
cmdargs = []
for curve in self._curves:
result += [{
"name": curve.legend, "color": curve.color, "data": []
}]
cmdargs += curve.to_rrd_command_args(rrdfile)
code = 0
for extra_args in [" --showtime", ""]:
cmd = "{} xport --start {} --end {}{} ".format(
self.rrdtool_binary, str(start), str(end), extra_args)
cmd += " ".join(cmdargs)
if isinstance(cmd, unicode):
cmd = cmd.encode("utf-8")
code, output = exec_cmd(cmd)
if code:
continue
if code:
return []
tree = etree.fromstring(output)
for row in tree.xpath('/xport/data/row'):
.
.
.

and I tried to solve using this way:

.
.
.
"""Export data to XML using rrdtool and convert it to JSON."""
result = []
cmdargs = []
for curve in self._curves:
result += [{
"name": curve.legend, "color": curve.color, "data": []
}]
cmdargs += curve.to_rrd_command_args(rrdfile)
code = 0
for extra_args in [" --showtime", ""]:
cmd = "{} xport --start {} --end {} ".format(
self.rrdtool_binary, str(start), str(end))
cmd += " ".join(cmdargs)
if isinstance(cmd, unicode):
cmd = cmd.encode("utf-8")
code, output = exec_cmd(cmd)
if code:
continue
if code:
return []
tree = etree.fromstring(output)
for row in tree.xpath('/xport/data/row'):
.
.
.

any idea to solve the "internal error" ?

Best Regards,
Ricardo

Answer 14 · 2016-06-17T14:17:07.000Z

Could you please enable DEBUG and send a proper stack trace ? (use a debugger in your browser)

Answer 15 · 2016-06-17T18:54:13.000Z

Hello Tonioo,

Sorry... but How I get a "Stack Trace" ?

Im using chrome.. and it is showing me these errors/warnings:

warning:
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
GET http://mail.skymsen.com.br/stats/graphs/?gset=mailtraffic&_=1466187900551 500 (Internal Server Error)
XHR finished loading: GET "http://mail.skymsen.com.br/core/top_notifications/check/".

error:
jquery.min.js:5 GET http://mail.skymsen.com.br/stats/graphs/?gset=mailtraffic&_=1466188139587 500 (Internal Server Error)send @ jquery.min.js:5b.extend.ajax @ jquery.min.js:5History.check @ history.js:272b.isFunction.i @ jquery.min.js:3
jquery.min.js:5 XHR finished loading: GET "http://mail.skymsen.com.br/stats/graphs/?gset=mailtraffic&_=1466188139587".send @ jquery.min.js:5b.extend.ajax @ jquery.min.js:5History.check @ history.js:272b.isFunction.i @ jquery.min.js:3

Thank you
Ricardo

Answer 16 · 2016-06-22T08:15:50.000Z

Thanks for your feedback but it is not usefull. Could send the response body instead ?

Answer 17 · 2016-07-20T01:39:21.000Z

@tonioo i am not sure whether this is what you want. I am having the same problem with the stats as well

<!DOCTYPE html>

<html lang="">
  <head>
    <meta charset="utf-8" />
    <title></title>
    <link rel="icon" type="image/png" href="/sitestatic/css/logo-icon.png" />
    <link rel="stylesheet" type="text/css" href="/sitestatic/bootstrap/dist/css/bootstrap.min.css" />
    <script src="/sitestatic/bootstrap/dist/js/bootstrap.min.js" type="text/javascript"></script>
    <script src="/sitestatic/eonasdan-bootstrap-datetimepicker/build/js/bootstrap-datetimepicker.min.js" type="text/javascript"></script>

    <script type="text/javascript" src="/jsi18n/"></script>
    <style>
      body {
          padding-top: 60px;
     }
    </style>
  </head>

  <body>
    <div class="container-fluid">
      <div class="row-fluid">

<div class="alert alert-danger">
  <h4>Sorry</h4>
  <p>An internal error occured.</p>
</div>

      </div>
    </div>
  </body>
</html>

Answer 18 · 2016-08-26T10:35:03.000Z

You need to enable debug information. Just set DEBUG to True inside the settings.py file, you should then see the real issue.

Answer 19 · 2016-09-23T20:24:58.000Z

Im getting this too on the "Statistics" pane in admin GUI. Ubuntu 16.04 with rrdtool Version: 1.5.5-4. Requested route gives a 500 error - /stats/graphs/?gset=mailtraffic&_=1474661219137 causing "internal error" message in "Statistics" pane.

FYI those trying to use debug, do an "updatedb && locate settings.py". Find the one in your modoboa instance (prob in /srv/modoboa/instance/). Edit it to use True for DEBUG. Restart uwsgi and nginx services. Open chrome console, hit refresh on "Statistics" and look for a server error. Click the link in the error and you should see a debug trace. Click "Switch to copy-and-paste view" on the trace part. Here is an output from mine:

AttributeError at /stats/graphs/

'NoneType' object has no attribute 'text'
Request Method: GET
Request URL: https://srv.example.com/stats/graphs/?gset=mailtraffic&_=1474661219137
Django Version: 1.9.9
Exception Type: AttributeError
Exception Value:
'NoneType' object has no attribute 'text'
Exception Location: /srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/graphics.py in export, line 117
Python Executable: /usr/bin/uwsgi-core
Python Version: 2.7.12
Python Path:
['.',
'',
'/srv/modoboa/env/lib/python2.7',
'/srv/modoboa/env/lib/python2.7/plat-x86_64-linux-gnu',
'/srv/modoboa/env/lib/python2.7/lib-tk',
'/srv/modoboa/env/lib/python2.7/lib-old',
'/srv/modoboa/env/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/srv/modoboa/env/local/lib/python2.7/site-packages']
Server time: Fri, 23 Sep 2016 16:07:30 -0400

Environment

Request Method: GET
Request URL: https://srv.example.com/stats/graphs/?gset=mailtraffic&_=1474661219137

Django Version: 1.9.9
Python Version: 2.7.12
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.sites',
'django.contrib.staticfiles',
'reversion',
'rest_framework.authtoken',
'rest_framework_swagger',
'modoboa',
'modoboa.core',
'modoboa.lib',
'modoboa.admin',
'modoboa.relaydomains',
'modoboa.limits',
'modoboa_amavis',
'modoboa_pdfcredentials',
'modoboa_postfix_autoreply',
'modoboa_sievefilters',
'modoboa_stats',
'modoboa_webmail')
Installed Middleware:
('x_forwarded_for.middleware.XForwardedForMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'modoboa.lib.middleware.AjaxLoginRedirect',
'modoboa.lib.middleware.CommonExceptionCatcher',
'modoboa.lib.middleware.RequestCatcherMiddleware')

Traceback

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
                response = self.process_exception_by_middleware(e, request)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
                response = wrapped_callback(request, _callback_args, *_callback_kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/utils/decorators.py" in inner
                return func(_args, *_kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/utils/decorators.py" in inner
                return func(_args, *_kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
            return view_func(request, _args, *_kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
            return view_func(request, _args, *_kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/views.py" in graphs
tplvars["graphs"] = gsets[gset].export(tplvars["domain"], start, end)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/graphics.py" in export
            "curves": graph.export(rrdfile, start, end)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/graphics.py" in export
        timestamp = int(row.find('t').text)
Exception Type: AttributeError at /stats/graphs/
Exception Value: 'NoneType' object has no attribute 'text'

Request Information

GET
Variable Value
gset
u'mailtraffic'
_
u'1474661219137'
POST
No POST data
FILES
No FILES data
COOKIES
Variable Value
csrftoken
'xxxxxxxxxxxxxxxxxxxxxxxxxxx'
sessionid
'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
ga
'xxxxxxxxxxxxxxxxxxxx'
__cfduid
'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
META
Variable Value
wsgi.multiprocess
True
HTTP_COOKIE
'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
UWSGI_SCRIPT
'instance.wsgi:application'
SCRIPT_NAME
u''
REQUEST_METHOD
'GET'
PATH_INFO
u'/stats/graphs/'
SERVER_PROTOCOL
'HTTP/1.1'
QUERY_STRING
'gset=mailtraffic&=1474661219137'
CONTENT_LENGTH
''
HTTP_USER_AGENT
'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2869.0 Safari/537.36'
HTTP_CONNECTION
'keep-alive'
SERVER_NAME
'srv.example.com'
REMOTE_ADDR
'123.123.123.123'
wsgi.url_scheme
'https'
SERVER_PORT
'443'
uwsgi.node
'srv'
DOCUMENT_ROOT
'/srv/modoboa/instance'
wsgi.input
<uwsgi.Input object at 0x7f21ce1d89a8>
HTTP_DNT
'1'
HTTP_HOST
'srv.example.com'
HTTPS
'on'
wsgi.multithread
False
HTTP_UPGRADE_INSECURE_REQUESTS
'1'
REQUEST_URI
'/stats/graphs/?gset=mailtraffic&=1474661219137'
HTTP_ACCEPT
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8'
wsgi.version
(1, 0)
wsgi.run_once
False
wsgi.errors
<open file 'wsgi_errors', mode 'w' at 0x7f21ce167930>
REMOTE_PORT
'56135'
HTTP_ACCEPT_LANGUAGE
'en-US,en;q=0.8'
REQUEST_SCHEME
'https'
UWSGI_APPID
'srv.example.com|'
uwsgi.version
'2.0.12-debian'
CONTENT_TYPE
''
wsgi.file_wrapper
''
CSRF_COOKIE
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
HTTP_ACCEPT_ENCODING
'gzip, deflate, sdch, br'

Django settings truncated since they are default besides debug and listener domain

Answer 20 · 2016-09-26T13:09:56.000Z

This issue has been definitely fixed by #13. I'll release a new version soon.