graphite-project/carbonate

Traceback on carbon-sync

obfuscurity opened this issue · 2 comments

Seeing this traceback on a specific whisper file. I can make the file available if it helps with diagnosis.

Traceback (most recent call last):
  File "/usr/local/bin/carbon-sync", line 9, in <module>
    load_entry_point('carbonate==0.2.0', 'console_scripts', 'carbon-sync')()
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 190, in main
    run_batch(metrics_to_sync, remote, args.storage_dir)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 119, in run_batch
    merge_elapsed = sync_batch(metrics_to_heal)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 60, in sync_batch
    heal_metric(staging, local)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 75, in heal_metric
    fill_archives(source, dest, time())
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 85, in fill_archives
    fill(src, dst, gapstart - step, start)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 50, in fill
    (timeInfo, values) = fetch(src, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 711, in fetch
    return file_fetch(fh, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 786, in file_fetch
    unpackedSeries = struct.unpack(seriesFormat, seriesString)
struct.error: unpack requires a string argument of length 1524

@obfuscurity, do you still have this issue?

We've seen a similar error from whisper-dump.py when the wsp file has been truncated by a full disk.

There also appears to be an open ticket in the graphite project with identical symptoms graphite-project/whisper#52 (comment)

I'm going to close this out since it seems to be a bug in whisper, not carbonate.