Bug: Failed parse data: Invalid element size detected

[mmaciejak]

What happened?

While trying to download data error happens.

Output from that snippet:

Starting download from 2022-11-17 11:30:00 to 2022-11-17 11:30:01 UTC
topics are: ['aufwickler-data', 'camera-1', 'cpu', 'damage-1', 'debug-camera-1', 'disk', 'kernel', 'mem', 'net', 'netstat', 'processes', 'rezeptmanager-data', 'swap', 'system', 'temp', 'trigger']
dict is: ['camera-1/1668684600200.dp', 'camera-1/1668684600402.dp', 'camera-1/1668684600603.dp', 'camera-1/1668684600805.dp']
image 0,  package: <drift_client.drift_data_package.DriftDataPackage object at 0x000001CD10E23D30> downloading
Failed parse data: Invalid element size detected
Traceback (most recent call last):
  File "E:\bbraun_onsite\bbraun_download.py", line 79, in <module>
    get_data(start_utc_time_stripped)
  File "E:\bbraun_onsite\bbraun_download.py", line 25, in get_data
    raw_img = package.as_np()
  File "E:\bbraun_onsite\.venv\lib\site-packages\drift_client\drift_data_package.py", line 16, in dec
    return func(self)
  File "E:\bbraun_onsite\.venv\lib\site-packages\drift_client\drift_data_package.py", line 120, in as_np
    return self.as_buffer().compose(scale_factor)
AttributeError: 'NoneType' object has no attribute 'compose'

Version

0.2.1

OS

Windows

Programming Language

No response

Snippet to repeat the bug

def get_data(start_time):
    client = DriftClient("tesa-1d4.local", os.getenv("DRIFT_PASSWORD"))
    topics = ["camera-1"]
    end_time = start_time + timedelta(seconds = 1)

    print(f'Starting download from {start_time} to {end_time} UTC')
    print(f'topics are: {client.get_topics()}')

    for topic in topics:
        packages = client.get_package_names(topic, start_time.strftime("%Y-%m-%d %H:%M:%S"), end_time.strftime("%Y-%m-%d %H:%M:%S"))
        print(f'dict is: {packages}')

        for i, package in enumerate(packages):
            package = client.get_item(str(package))
            if package.status_code == 0:
                print(f'image {i},  package: {package} downloading')
                raw_img = package.as_np()

Logs

No response

Attachments

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[atimin]

This is a bug in WaveletBuffer::Parse. I'll move the issue there.

[atimin]

The source of the data is a Linux machine which uses WaveletBuffer 0.4.0. Looks like this is an issue that the serialized packages aren't portable between Linux and Windows any more. It definitely worked for WaveletBuffer 0.2.0. @victor1234 do you have any ideas?

[victor1234]

Wavelet Buffer versions are the same?

[atimin]

Yep, the same. I'll assign the issue to you and send the "broken" package.

[atimin]

The “broken” packages contain an outdated version of WaveletBuffer. We don't repeat this for the latest version.

[atimin]

It is an actual error. The marshalling of WaveletBuffer is not platform independent. It happens when SfCompressor is off and we use blaze::Archive for serialisations of subbands.

[atimin]

We misuse blaze’s serialazer in WaveletBuffer::Parse and WaveletBuffer::Serialize methods:

 blaze_arch >> buffer->impl_->decompositions_;

We need to serialise each subband separately. this is a breaking change and we have to support previous versions.

SOLUTION:

We create different implementations of serialiser and use serialization_version for switching.

class ISerializer {
public:
 /**
   * Parses subbands from a blob of data and creates a new buffer
   * @param blob the blob of subbands
   * @return nullptr if it failed to parse the buffer
   */
  [[nodiscard]] virtual std::unique_ptr<WaveletBuffer> Parse(
      const std::string& blob) = 0;

  /**
   * Serialize a buffer into the blob for saving in a file or sending via
   * network
   * @param blob the blob to serialize
   * @param sf_compression - 0 - switch off, 16 - max compression(bfloat).
   * @return return true if it has no error
   */
  [[nodiscard]] virual bool Serialize(const WaveletBuffer& buffer,std::string* blob,
                               uint8_t sf_compression = 0) const = 0;
}

We make two implementations of ISerializer for current version (2) and new correct implementation for version (3). Also we need to create unit tests for both versions.