Unable to link to USB Host Shield Library

[getSurreal]

I'm using the USB Host Shield Library (https://github.com/felis/USB_Host_Shield_2.0) in my sketch and it compiles using the IDE, but cmake is unable to find it. The library has a lot of different files, but none that match the folder name like usual libraries. I have my libraries linked in CMakeLists.txt like so:

 link_directories(~/Arduino/libraries)

And it does compile other libraries

I tried adding an additional link directly to the diretory like so

 link_directories(~/Arduino/libraries/USB_Host_Shield_20)

But it still didn't find it.

What should I do in order to link to this library?