What did you find problematic about Qt? You were using straight Widget-based Qt IIRC. Do you think QML would have solved some of those issues?
I'm thinking about writing an LV2 plugin with an interface made in QML which comunicates via OSC to the sound engine, so that the GUI and audio part are well separated. Your opinion as a much more experienced developer would be much appreciated
In the most of the cases this will work, and maybe you don't need to care too much. But if you want to be more strict about plugins than...
If the user wants to load different plugins that were based on different versions of Qt, than the host needs to load in its address space two different shared Qt libraries which in the most of the cases will cause the host/DAW to crash due to various reasons. In theory this should not happen but Qt is designed in a way that does not offer safely two instances of Qt to run in the same process address space. I saw that the Qt interface permits not to export the symbols (and I am not sure if this option was developed until the end), but in this case you'll need to build all Qt with this option, dynamic or statically... but this is too complicated, and probably not really possible for Qt, and I am afraid with QML module added will make things even worse.
In general every dependence you add to the plugin be sure that it will not behave in this way. If you add something be sure that it is used by the host/DAW too[*] and you are ABI compatible with that or you build/link it statically by turning off symbols export if it is possible. On the other hand, every plugin at the end will depend say of X11 API library, for example, shared libraries, but this library is only loaded by the host once (and only one version) because is base for every GUI that runs on the system. So, GUI toolkits for plugins better to make use only of "system wide" things.
Probably for these reasons there are also developed GUI toolkits for plugins like DISTRHO by GNU/Linux audio hackers, VST GUI by Steinberg, or maybe JUCE too. But be ware if you'll use one of these toolkits (or Redkite) instead of Qt than there will be a lot of limits you can do, the first one, you may not have a file browser that is native or that is mature and cross-platform, you many not have advanced dynamic layout features which often complicates things, cross-platform shortcuts support, things related to graphics backends rendering etc.
If you'll decide to use one of these toolkits and want to have a cross-platform and cross-format plugins support out of the box, than I recommend DISTRHO.
"OSC to the sound engine" - I don't know this engine but for LV2 and in general for plugins you don't need to use any sound engine, the host takes care of this, you need this only for standalone version. For standalone make sure that engine supports JACK as an option at least, because JACK is a GNU/Linux audio standard for music, you need to support it.
[*] Even in this case it is a not a guarantee, it needs that library also to be thread-safe, and not to depend of a global state