Integrate PSLab-Python with OSS-Fuzz for Continuous Fuzz Testing

Question

Integrate PSLab-Python with OSS-Fuzz for Continuous Fuzz Testing

Closed this issue a year ago · 18 comments

I hope this message finds you well! I’d like to propose integrating PSLab-Python with Google's OSS-Fuzz continuous fuzz testing platform designed to improve the stability and security of open-source projects.

Why Integrate PSLab-Python with OSS-Fuzz?

Improved Stability: OSS-Fuzz can automatically detect edge-case bugs, crashes, and security vulnerabilities in PSLab-Python’s codebase.
Enhanced Reliability: Continuous fuzzing ensures that untrusted inputs, such as data from sensors, communication protocols, or user-defined configurations, are handled robustly.
Proactive Bug Fixes: By identifying potential issues early, OSS-Fuzz helps maintain a stable and secure codebase.

Benefits for PSLab-Python

Continuous Testing: OSS-Fuzz runs automated fuzz tests continuously, providing ongoing feedback about the project’s stability.
Bug Reporting: OSS-Fuzz identifies and reports any issues directly to the maintainers, allowing for timely fixes.
Improved User Confidence: Ensuring robustness against unexpected inputs enhances trust in PSLab-Python’s reliability.

I’d be happy to work for setting up the integration. Let me know Can I move Forward with this And Create PR for Same
Looking forward to your feedback!
Best regards

Answer 1 · 2024-12-15T15:41:45.000Z

Hi @Shivam7-1. I'd actually never heard of fuzz testing before, but it seems like it could be useful. Go ahead and work on integrating OSS-Fuzz with pslab-python!

I'm curious how you plan to approach the issue of serial traffic emulation. Most of what pslab-python does involves talking to a pslab device over a serial bus, which makes testing a bit tricky. The current testing approach relies on replaying recorded serial traffic, but this has resulted in a very high maintenance overhead and I would like to move away from this approach.

Does OSS-Fuzz offer an alternative way to test?

Answer 2 · 2024-12-15T18:13:34.000Z

Hii @bessman Thanks For Responding
To address the serial traffic emulation challenge, I plan to create fuzz targets that mock PSLab hardware interactions like set_gain, analog_read, and send_data. This approach will emulate hardware behavior without relying on actual devices or recorded traffic, leveraging OSS-Fuzz to test software logic effectively.

Let me know if you have any suggestions!
Best regards,
Shivam

Answer 3 · 2024-12-16T07:47:05.000Z

Sounds good!

As a first step, I would suggest implementing a fuzz test for a single function, as a proof of concept. For example, triggering an oscilloscope read with pslab.oscilloscope.capture. This would allow me to better understand the concept and evaluate its suitability for pslab-python.

Answer 4 · 2024-12-16T09:00:39.000Z

Hii @bessman Thanks For Reply
here is below Code which could work

import sys
import atheris
import pslab

def TestOneInput(data):
    fdp = atheris.FuzzedDataProvider(data)

    try:
        # Initialize the communication interfaces
        i2c = pslab.i2c.I2C()

        # Fuzz I2C read/write operations
        address = fdp.ConsumeIntInRange(0, 127)  # Valid 7-bit I2C addresses
        register = fdp.ConsumeIntInRange(0, 255)  # Register range
        value = fdp.ConsumeIntInRange(0, 255)  # Data to write
        
        # Fuzz reading from an I2C device
        i2c.read(address, register)
        
        # Fuzz writing to an I2C device
        i2c.write(address, register, value)

        # Fuzz oscilloscope functionality
        oscilloscope = pslab.oscilloscope.Oscilloscope()
        channel = fdp.ConsumeIntInRange(1, 4)  # Channels 1-4
        timebase = fdp.ConsumeFloat()  # Timebase for capture
        samples = fdp.ConsumeIntInRange(1, 1000)  # Number of samples
        oscilloscope.capture(channel=channel, timebase=timebase, samples=samples)
    except Exception:
        pass

def main():
    atheris.Setup(sys.argv, TestOneInput)
    atheris.Fuzz()

if __name__ == "__main__":
    main()

How about this for pslab_fuzz.py file As in ossfuzz Continous fuzzing and auto Alert would be Given As well as its Scan whole Repo code so it would automaticallygive alerts if it finds anything missing
Thanks

Answer 5 · 2024-12-16T09:11:12.000Z

Here is Another varient of Code which we could also use

import atheris
import sys
from pslab import ScienceLab

def TestOneInput(data):
    fdp = atheris.FuzzedDataProvider(data)
    try:
        # Initialize the ScienceLab instance
        psl = ScienceLab()

        capacitance_range = fdp.ConsumeIntInRange(1, 5)  
        psl.multimeter.measure_capacitance(capacitance_range)

        channel = fdp.ConsumeIntInRange(1, 4)  
        samples = fdp.ConsumeIntInRange(100, 1000)  # Number of samples
        timebase = fdp.ConsumeFloat()  # Timebase for the capture
        psl.oscilloscope.capture(channel=channel, samples=samples, timebase=timebase)

        frequency = fdp.ConsumeIntInRange(1, 5000)  # Frequency in Hz
        amplitude = fdp.ConsumeIntInRange(1, 5)  # Amplitude in volts
        psl.waveform_generator.generate_wave(frequency=frequency, amplitude=amplitude)
    except Exception as e:
        # Log unexpected errors for debugging
        if "expected exception" not in str(e):
            raise

def main():
    atheris.Setup(sys.argv, TestOneInput)
    atheris.Fuzz()

if __name__ == "__main__":
    main()

As same oss fuzz gives Scan whole Repo code so it would automatically give alerts if it finds anything missing
if i am missing with parameters could you give me some like psl.oscilloscope.capture
Thanks

Answer 6 · 2024-12-16T09:17:01.000Z

That looks better. A couple of things:

It's pslab.Oscilloscope(), or pslab.instrument.Oscilloscope(), or pslab.instrument.oscilloscope.Oscilloscope(), not pslab.oscilloscope.Oscilloscope().
The parameters of Oscilloscope.capture are channels and timegap, not channel and timebase.
It's psl.waveform_generator.generate, not .generate_wave

I would also suggest splitting up the tests in different functions. One for I2C, one for oscilloscope, one for waveform generator, and so on.

How does one run these tests? If a test passes, what does that tell me? If it fails, what does that tell me?

Answer 7 · 2024-12-16T09:27:37.000Z

How this below code

import sys
import atheris
import pslab


def fuzz_i2c(data):
    fdp = atheris.FuzzedDataProvider(data)
    i2c = pslab.I2C()
    try:
        address = fdp.ConsumeIntInRange(0, 127)  # Valid 7-bit I2C addresses
        register = fdp.ConsumeIntInRange(0, 255)  # Register range
        value = fdp.ConsumeIntInRange(0, 255)  # Data to write

        # Fuzz reading from an I2C device
        i2c.read(address, register)

        # Fuzz writing to an I2C device
        i2c.write(address, register, value)
    except Exception:
        pass


def fuzz_oscilloscope(data):
    fdp = atheris.FuzzedDataProvider(data)
    try:
        oscilloscope = pslab.instrument.Oscilloscope()

        # Fuzz parameters for capture
        channels = [fdp.ConsumeIntInRange(1, 4) for _ in range(2)] 
        timegap = fdp.ConsumeFloat()  # Time gap between captures

        oscilloscope.capture(channels=channels, timegap=timegap)
    except Exception:
        pass


def fuzz_waveform_generator(data):
    fdp = atheris.FuzzedDataProvider(data)
    try:
        waveform_generator = pslab.waveform_generator.WaveformGenerator()

        # Fuzz parameters for waveform generation
        frequency = fdp.ConsumeFloatInRange(1.0, 100000.0)  # Frequency in Hz
        amplitude = fdp.ConsumeFloatInRange(0.1, 5.0)  # Amplitude in volts
        waveform_type = fdp.ConsumeString(8)  # Waveform type (e.g., 'sine', 'square', 'triangle')

        waveform_generator.generate(frequency=frequency, amplitude=amplitude, waveform_type=waveform_type)
    except Exception:
        pass


def TestOneInput(data):
    fuzz_i2c(data)
    fuzz_oscilloscope(data)
    fuzz_waveform_generator(data)


def main():
    atheris.Setup(sys.argv, TestOneInput)
    atheris.Fuzz()


if __name__ == "__main__":
    main()

Answer 8 · 2024-12-16T09:29:59.000Z

channels = [fdp.ConsumeIntInRange(1, 4) for _ in range(2)]

Why did channels become a list?

I am starting to suspect I am talking to an AI. Please carefully read the documentation and write code that will actually work, or I will close this issue.

Answer 9 · 2024-12-16T09:41:13.000Z

hii
i think here for Two channels for multi-capture i think i had mistake
sorry this is my first time integrating project

import sys
import atheris
import pslab


def fuzz_i2c(data):
    fdp = atheris.FuzzedDataProvider(data)
    i2c = pslab.I2C()
    try:
        address = fdp.ConsumeIntInRange(0, 127)  # Valid 7-bit I2C addresses
        register = fdp.ConsumeIntInRange(0, 255)  
        value = fdp.ConsumeIntInRange(0, 255)

        i2c.read(address, register)

        i2c.write(address, register, value)
    except Exception:
        pass


def fuzz_oscilloscope(data):
    fdp = atheris.FuzzedDataProvider(data)
    try:
        oscilloscope = pslab.instrument.Oscilloscope()

        channel = fdp.ConsumeIntInRange(1, 4)
        timegap = fdp.ConsumeFloat()  # Time gap between captures

        oscilloscope.capture(channel=channel, timegap=timegap)
    except Exception:
        pass


def fuzz_waveform_generator(data):
    fdp = atheris.FuzzedDataProvider(data)
    try:
        waveform_generator = pslab.waveform_generator.WaveformGenerator()

        frequency = fdp.ConsumeFloatInRange(1.0, 100000.0)  # Frequency in Hz
        amplitude = fdp.ConsumeFloatInRange(0.1, 5.0)  # Amplitude in volts
        waveform_type = fdp.ConsumeString(8)

        waveform_generator.generate(frequency=frequency, amplitude=amplitude, waveform_type=waveform_type)
    except Exception:
        pass


def TestOneInput(data):
    fuzz_i2c(data)
    fuzz_oscilloscope(data)
    fuzz_waveform_generator(data)


def main():
    atheris.Setup(sys.argv, TestOneInput)
    atheris.Fuzz()


if __name__ == "__main__":
    main()

i think now it good to go

Answer 10 · 2024-12-16T09:43:36.000Z

i think now it good to go

It is not. When you run this test (you do run the test yourself, right?) what is the result?

Answer 11 · 2024-12-16T09:51:59.000Z

Everything got build sucessfully
for this here i used python3 infra/helper.py build_fuzzers

Answer 12 · 2024-12-16T09:53:32.000Z

I'm not seeing a test result in that output. It looks like output from a build script to set up a testing environment, not from running the actual test.

Answer 13 · 2024-12-16T10:02:14.000Z

here is sample flow ossfuzz project integration dont even need any fuzz file it do automatically scan the code of listed project time to time and accordingly if they find fault or security issue in code it will directly report to mainitainer of it
for this i had create project.yaml file you can check this here google/oss-fuzz@858dd28

Answer 14 · 2024-12-16T10:09:31.000Z

dont even need any fuzz file it do automatically scan the code of listed project time to time and accordingly if they find fault or security issue in code it will directly report to mainitainer of it

How does it do that?

I do not understand how oss-fuzz works or what information it provides. For this issue to proceed, I need answers to the following:

What does a passing test look like?
What does it tell me?
What does a failing test look like?
What does it tell me?

Answer 15 · 2024-12-16T10:15:35.000Z

Simply ossfuzz is used to Scan Repo Continuously and find security Related issue in Code
If it find and security vulnerabilities or bug in code it will directly Report to the maintainer
It works for this Programming languages written code python,c++,java,Rust
Go

Answer 16 · 2024-12-16T10:17:33.000Z

That is too vague.

What does a passing test look like?

What does it tell me?

What does a failing test look like?

What does it tell me?

Answer 17 · 2024-12-16T10:26:46.000Z

Here is Answer
What does a passing test look like?
A passing test means the code runs without crashes, security vulnerabilities, or memory errors during fuzzing.
What does it tell me?
It tells that the code is secure and stable
What does a failing test look like?
A failing test occurs when the fuzzer detects crashes, memory issues, or other unexpected behavior
What does it tell me?
It indicates the presence of security flaws, crashes, or logic errors, providing detailed information to help maintainers fix the issues

Answer 18 · 2024-12-16T10:28:32.000Z

The benefit for pslab-python is not clear. Closing.