Prof Alex Groce
Why Embedded Testing Really Matters:
More to come, see syllabi here for overview of topics, flow of the class
Office hours Wednesday 8:00am-9:00am (SICCS 208)
First reading, to prepare yourselves: https://www.usenix.org/system/files/1311_05-08_mickens.pdf
CLASS 1: General Overview of Class
- We're going to torture software (automatically) in order to find bugs
- We are interested in code in C/C++, mostly
- Highly critical code, sometimes safety-critical
- But written in low-level, bug-prone languages
CLASS 2: Embedded Software, What is It?
- Embedded software:
-
- Software running on something that's not "primarily a computer"
-
- May not even have an operating system
-
- Lives in computing power/memory constrained environment
-
- "Talks to hardware, not people"
-
- In particular, sensors + actuators is often the model
-
- "Old school" physical world not social media land
-
- Examples: automotive, power plants, aerospace, robots, Internet of Things (controlled by the Russian mafia)...
CLASS 3: Read Mickens, particular problems of testing embedded world; break the code, break the "OS" and you can't debug/trace
- We looked at some actual example code
-
- https://github.com/TheSEGAProject/CP-Firmware (Paul Flikkema's group)
-
- https://github.com/nasa/fprime (NASA "boilerplate" embedded code for CubeSAT "cheapo" satellite missions)
CLASS 4: We will start talking about fuzzing
Things to look at:
-
Good overview of Bart Miller's work, includes links to the paper(s): https://blog.trailofbits.com/2018/12/31/fuzzing-like-its-1989/
-
The AFL website: http://lcamtuf.coredump.cx/afl/
CLASS 5: We talked more about fuzzing and ran AFL and libfuzzer
CLASS 6: We're finishing up the intro to fuzzing
ASSIGNMENT 1 went out
- Topic: command line options for fuzzers, esp. -use_value_profile=1
- Topic: fuzzer behavior depends on two things (well, lots of things, but):
-
- Notion of WHAT IS INTERESTING BEHAVIOR?
-
- How to modify/mutate files
- There is no "best fuzzer" -- need to run many, hope to get lucky!
CLASS 7: Sanitizers 1
-
Sanitizers are powerful tools for making your code crash more
-
Expose bad behavior that slips by, e.g. memory-safety issues
-
-fsanitize=address - AddressSanitizer (ASAN) is best known, most useful
-
Others: undefined (UBSAN), integer, memory (uninitialized values), thread
-
Sanitizers are SLOW: 2x up to much worse slowdown; in class we saw 4x
-
Slow is fine for unit tests, but not so hot for fuzzing
-
Sanitizers are VERBOSE -- real code produces lots of bogus warnings, often
-
Good general reading on the sanitizers from Google is at https://github.com/google/sanitizers -- in particular, note the top contributor, kcc : Kostya Serebryany (known as "Kostya" like "Prince" or "Madonna" to security/testing people) is also the libFuzzer Guy
CLASS 8: Sanitizers Big Picture / Odds and Ends
-
Well defined behavior: program only does "legal" things
-
Implementation-defined behavior: program does stuff where compilers can vary, but there is a requirement to have some meaning, often within constraints
- Undefined behavior: see https://blog.regehr.org/archives/213
-
Fuzzing crypto can be tough: no paths, in part to avoid timing/power attacks
-
Run fuzzing in a VM, to avoid a bug trashing your file system
-
radamsa is a "mess with this input" fuzzer
READINGS FOR NEXT WEEK:
- https://www.cs.princeton.edu/courses/archive/fall12/cos109/mars.rover.pdf
- https://spaceflight.nasa.gov/spacenews/releases/2000/mpl/mpl_report_1.pdf
CLASS 9:
- NASA Gossip, My Career
CLASS 10:
-
Test what you fly
-
Doing fixed operations with systematic or even random faults is not enough, need random operations (see tester.txt)
-
Rename's a beast
-
MER shows lots of GOOD practices (paranoia, systems that do predictable talking to home, heavy fault protection), and some weaknesses in testing
CLASS 11:
-
Mars Polar Lander
-
additional info: http://web.mit.edu/16.070/www/readings/Failures_MCO_MPL.pdf
-
If a test fails and you "fix" the bug, run the test again to make sure
-
Never assume requirements are complete!
-
Backdoors from compiler bugs: see https://www.alchemistowl.org/pocorgtfo/pocorgtfo08.pdf -- also, the classic article here is https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf (Ken Thompson's Turing Award speech)
-
Testing has grave responsibility (and, ok, often not that much power, sorry)
CLASS 12:
-
Fuzzing a simple library with afl (see stack_example directory)
-
Why DeepState -- see https://github.com/agroce/deepstate-stack
-
READING ASSIGNMENT FOR WEDNESDAY: https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/
CLASS 13:
-
We ran a bunch of fuzzers, and looked for pizza
-
DeepState lets you run many different fuzzers (and other tools) easily
-
DeepState is a unit testing library, but you can ask a tool to pick the values forr you
-
See deepstate_example
CLASS 14:
-
More DeepState, this time a look at the stack example
-
Introduced swarm testing (see https://agroce.github.io/issta12.pdf)
-
See file swarm.txt; we could find the stack "bug" in no time with swarm, not find it in hours without using it
-
DeepState lets us apply meta-heuristics, "extra tricks" on top of fuzzers like AFL/libFuzzer
-
See unittestsort.txt for a preview of Monday