Review build setup

[willdurand]

It looks like the lib now contains wasm stuff and that's not what I want. I am not sure we can have different lib types with different dependencies in the same Cargo project like it is currently. I think I'd need to make a core lib (pure rust, no wasm stuff) and then use that for the wasm lib and the CLI.