For example, this line should be:
install(DIRECTORY ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/rnnoise.vst3 DESTINATION lib/vst3)
cmake's install instruction expects relative paths as DESTINAION, otherwise it installs into $PREFIX$PREFIX.
The same problem exists with LV2 plugin.