Plugin fuzzer (for testing, not shoegazing!)

[Basslint]

Note: this is not about fuzz effects (as in Big Muff) but about fuzz testing.

As a package maintainer, I am overwhelmed by the idea of making sure all plugins work after they and/or their dependencies are updated.

The plugins' own unit tests (when present, which is not often) aren't 100% reliable for testing: the plugins can always crash when loaded into a host.

Some days ago I came across a fuzzer for the Godot libre game engine (Qarminer) and that might have influenced me to come up with this idea: why not use a fuzzer for the major plugins technologies (VST, VST3 and LV2) at the very least to discover internal crashes?

After all plugins should be an ideal setting for fuzzing: they are data-oriented, they have a bunch of discoverable parameters which are supposed to be automated, they are written using 3 major standardized interfaces.

Call this a research thread, mostly to put on (totally not) paper ideas from more skilled developers than myself.

• Are there any libre and open source plugin fuzzers around?
• How would you go about implementing a fuzzer targeting all major plugin technologies (LV2/VST/VST3)? (I thought about leveraging carla-devel)
• What do you think would be the most problematic aspects of implementing a fuzzer?

[Basslint]

@Kott told me about pluginval by Tracktion. I had even opened a thread for it, which shows the pitfalls of human memory. Still, I think some technical discussions could be interesting because pluginval can't host LV2s.

[SpotlightKid]

Are you aware of https://carlh.net/plugins/torture.php ?

[Kott]

Are you aware of https://carlh.net/plugins/torture.php ?

I was not. It's really cool, thank You.