NEW! Safe Plugin Scanner version 2 with cache

MusE is a DAW for Linux with both MIDI and Audio editing. https://muse-sequencer.github.io

Moderators: MattKingUSA, khz, spamatica

Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

OK I think this is about as far as I can go with a plugin scanner for now.
There is some more stuff I can do with it, maybe in a wee bit.
Like replacing the redundant SimplePlugin library with this instead...

Please give it a whirl.
Remember you can always force a rescan as described below.

Available in git master now. From the ChangeLog:

* NEW! Safe Plugin Scanner version 2: (Tim)
- The plugin scans are cached now, in ~/.config/MusE/scanner.
This means faster startup time since it no longer has to
rescan each time.
- Deicsonze and SimpleDrums MESS synths load muuuch faster
now since each instance no longer has to scan plugins.
- New command line option -R forces rescan.
Also new 'Force rescan' checkbox under plugin paths tab
in Global Settings dialog.
- Automatically detects changes in any of the plugin paths
given in Global Settings dialog, and rescans.
- Added Ladspa RDF (lrdf) support. Initial support for now.
- More major rearrangements: Moved several module libraries
out of the muse tree into the libs tree, since they stand
on their own.
Created new libs folder 'plugin' containing all of the
plugin scanning and caching code.
Awl and AL libraries are now fully separated and isolated
from the main app. Awl is NO LONGER linked in with the
main app, only with the Deicsonze MESS synth. Yay.

[Edit:] Oh yeah, I forgot: I reworked the MusE splash screen a wee bit
to display some informative text about what's happening during startup,
you know like lots of other apps. Helps during those long plugin scans. Try 'er out.
alex stone
Established Member
Posts: 350
Joined: Fri Jun 06, 2008 7:39 am
Has thanked: 61 times
Been thanked: 53 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by alex stone »

Tim, pulled from github, removed build DIR, and started compiling.

It failed with the following error:


[ 11%] Building CXX object libs/plugin/CMakeFiles/plugin_cache_reader_module.dir/plugin_cache_reader.o
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp: In function ‘QStringList MusEPlugin::pluginGetLadspaDirectories(const QString&)’:
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:422:24: error: ‘qEnvironmentVariable’ was not declared in this scope
QString ladspaPath = qEnvironmentVariable("LADSPA_PATH");
^~~~~~~~~~~~~~~~~~~~
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:422:24: note: suggested alternative: ‘qEnvironmentVariableIsSet’
QString ladspaPath = qEnvironmentVariable("LADSPA_PATH");
^~~~~~~~~~~~~~~~~~~~
qEnvironmentVariableIsSet
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp: In function ‘QStringList MusEPlugin::pluginGetMessDirectories(const QString&)’:
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:445:22: error: ‘qEnvironmentVariable’ was not declared in this scope
QString messPath = qEnvironmentVariable("MESS_PATH");
^~~~~~~~~~~~~~~~~~~~
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:445:22: note: suggested alternative: ‘qEnvironmentVariableIsSet’
QString messPath = qEnvironmentVariable("MESS_PATH");
^~~~~~~~~~~~~~~~~~~~
qEnvironmentVariableIsSet
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp: In function ‘QStringList MusEPlugin::pluginGetDssiDirectories()’:
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:465:22: error: ‘qEnvironmentVariable’ was not declared in this scope
QString dssiPath = qEnvironmentVariable("DSSI_PATH");
^~~~~~~~~~~~~~~~~~~~
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:465:22: note: suggested alternative: ‘qEnvironmentVariableIsSet’
QString dssiPath = qEnvironmentVariable("DSSI_PATH");
^~~~~~~~~~~~~~~~~~~~
qEnvironmentVariableIsSet
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp: In function ‘QStringList MusEPlugin::pluginGetLinuxVstDirectories()’:
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:485:21: error: ‘qEnvironmentVariable’ was not declared in this scope
QString vstPath = qEnvironmentVariable("LINUX_VST_PATH");
^~~~~~~~~~~~~~~~~~~~
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:485:21: note: suggested alternative: ‘qEnvironmentVariableIsSet’
QString vstPath = qEnvironmentVariable("LINUX_VST_PATH");
^~~~~~~~~~~~~~~~~~~~
qEnvironmentVariableIsSet
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp: In function ‘QStringList MusEPlugin::pluginGetVstDirectories()’:
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:511:21: error: ‘qEnvironmentVariable’ was not declared in this scope
QString vstPath = qEnvironmentVariable("VST_PATH");
^~~~~~~~~~~~~~~~~~~~
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:511:21: note: suggested alternative: ‘qEnvironmentVariableIsSet’
QString vstPath = qEnvironmentVariable("VST_PATH");
^~~~~~~~~~~~~~~~~~~~
qEnvironmentVariableIsSet
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp: In function ‘QStringList MusEPlugin::pluginGetLv2Directories()’:
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:531:21: error: ‘qEnvironmentVariable’ was not declared in this scope
QString lv2Path = qEnvironmentVariable("LV2_PATH");
^~~~~~~~~~~~~~~~~~~~
/home/alex/github/muse/muse3/libs/plugin/plugin_cache_reader.cpp:531:21: note: suggested alternative: ‘qEnvironmentVariableIsSet’
QString lv2Path = qEnvironmentVariable("LV2_PATH");
^~~~~~~~~~~~~~~~~~~~
qEnvironmentVariableIsSet
libs/plugin/CMakeFiles/plugin_cache_reader_module.dir/build.make:61: recipe for target 'libs/plugin/CMakeFiles/plugin_cache_reader_module.dir/plugin_cache_reader.o' failed
make[2]: *** [libs/plugin/CMakeFiles/plugin_cache_reader_module.dir/plugin_cache_reader.o] Error 1
CMakeFiles/Makefile2:1282: recipe for target 'libs/plugin/CMakeFiles/plugin_cache_reader_module.dir/all' failed
make[1]: *** [libs/plugin/CMakeFiles/plugin_cache_reader_module.dir/all] Error 2
Makefile:151: recipe for target 'all' failed
make: *** [all] Error 2
alex@alextone:~/github/muse/muse3$



Any clues?


Alex.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Hi can you tell me what running the command 'qtdiag' says what version
of Qt you have?

The qEnvironment function in question was added in Qt 5.10
So if you have an earlier version I'll have to fix this. Checking for options...

Thanks.
Tim.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Hi. I changed it to use std::getenv instead of qEnvironmentVariable

That should work.
In git master now. Please try it again.

Lemme know if any more trouble!

Thanks.
Tim.
alex stone
Established Member
Posts: 350
Joined: Fri Jun 06, 2008 7:39 am
Has thanked: 61 times
Been thanked: 53 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by alex stone »

Yep, that worked, thanks.

qtdiag shows 5.9.5.

I keep this box off the interlink, but i may have to do an upgrade at some point. And it's running so well.........

Cheers,

Alex.
spamatica
Established Member
Posts: 573
Joined: Mon Feb 08, 2010 10:38 am
Has thanked: 80 times
Been thanked: 97 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by spamatica »

Awesome Tim, building now.

I've had some time to work with MusE since the first version, very stable.
There were some new issues with native UIs no longer showing up though, anything you've seen?

It pains me that I can't seem to find the time to try and root out a few bugs myself.. soon hopefully.
MusE DAW
spamatica
Established Member
Posts: 573
Joined: Mon Feb 08, 2010 10:38 am
Has thanked: 80 times
Been thanked: 97 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by spamatica »

Oh, a build error.
muse/muse3/libs/plugin/plugin_cache_writer.cpp:126:8: error: unused parameter ‘do_rdf’ [-Werror=unused-parameter]
bool do_rdf)

I'm guessing I'm missing some rdf library so I can probably get past it but might need some #ifdefs there somewhere.
MusE DAW
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Ah, gets me every time. Fixed in git master.
Thanks so much guys for testing!

It's been a blast making this scanner. We could make an actual run-time sandbox out of it.

Slightly off-topic but...

While I was in there, I had a look at the latest situation with...
VST3 !

Y'know, it's open source and Linux-friendly now.
I've been playing with the SDK, After some head-scratching and fixing, I'm able to run the examples.

Here's the really weird thing: Until just weeks ago, the ability to read 2.x plugins was included in the VST3 SDK.
UUIC Now that is gone, completely from the git SDK. IIUC it will be gone for good.

So I guess I'll keep a copy of this slightly older VST3 SDK with the 2.x support ???
Can you see how that statement makes the removal of 2.x support seem so illogical?

But at least this is balanced out - heavily - by the fact that anyone can use the SDK now
so there would be no excuse for not porting plugins to it... Unless you are unfortunate
to have plugins which CAN'T be ported, say because the plugin author no longer exists.
It's GPL3. Hm... Is that compatible with our GPL2?

Cheers.
Tim.
spamatica
Established Member
Posts: 573
Joined: Mon Feb 08, 2010 10:38 am
Has thanked: 80 times
Been thanked: 97 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by spamatica »

MusE license says "...as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version."

Which as I understand it (after some reading) means that if we add the VST3 sources the combination would automatically be under GPLv3.
Don't know if that poses any issue, don't think so.
MusE DAW
folderol
Established Member
Posts: 2069
Joined: Mon Sep 28, 2015 8:06 pm
Location: Here, of course!
Has thanked: 224 times
Been thanked: 400 times
Contact:

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by folderol »

It's only a problem if any parts of the code are GPL V2 strict.
The Yoshimi guy {apparently now an 'elderly'}
artofmusic
Established Member
Posts: 90
Joined: Mon Mar 17, 2014 8:01 pm

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by artofmusic »

I've been having trouble getting sforzando to work on my system using LinVst which is a shame as this is the only way to get automation support with the host (Muse), even carla doesn't support this. It has failed ever since the new scanner has been implemented. Below is a snippet of what I think is going on.

Code: Select all

Initializing Native VST support. Using VESTIGE compatibility implementation.
XML: level = 0
XML Parse Error at line 0 col 2
pluginScan FAILED: On readPluginScan(): file: /home/alex/.wine/drive_c/Program Files/Plogue/sforzando/VST/sforzando VST_x64.so
I have the corresponding .dll and everything works fine in muse 3.0.2 and in ardour/mixbus. I have built it from source on Archlinux using qt 5.12 and gcc 8.2.1.
If you can maybe create a way to disable the scanner globally or a way to exclude plugins from the scan on a case by case basis for the next stable release that would be great, thanks.

The full log is attached below in a zip file.
Attachments
debugscanneroutput.zip
output log of Muse sandbox plugin scanner.
(21.45 KiB) Downloaded 59 times
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Hello. sforzando is a Windows VST. MusE currently only supports Linux native VST.
That's why MusE skips the sforzando dll - MusE doesn't recognize it.

If you say that you have been previously able to use sforzando in MusE,
I can only imagine it must have been through LinVst somehow.
Unless MusE is being prevented from running further due to this skipping of this plugin,
you should still be able to use the plugin in MusE, nothing should have changed there.

Lemme know how it goes, if I can help further.
Tim.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Sorry, my mistake, was not paying attention.
I thought LinVst was just a VST host but it's more than that, eh?

Testing and checking LinVst and sforzando... I will reply later...

[Edit:]
Can you share with me any settings or scripts or configuration settings or Wine dll overrides for LinVst
that you may have used so I can duplicate your situation (hopefully quickly, since I've never used LinVst). Thanks.

Tim.
artofmusic
Established Member
Posts: 90
Joined: Mon Mar 17, 2014 8:01 pm

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by artofmusic »

BUILD INFO:
Here's the exact PKGBUILD 3.0.2 (plugin functions) I used for the build process https://aur.archlinux.org/cgit/aur.git/ ... ILD?h=muse
Here's the exact PKGBUILD git (plugin doesn't get detected) I used for the build process viewtopic.php?f=61&t=19274
LinVst Project (LinVst 2.4 latest commit branch): https://github.com/osxmidi/LinVst
I put linvst.so in /usr/bin instead of in /usr/lib/vst as ardour would pick it up, but a little error would popup and it would say linvst.so not found and it would add the plugin to ardour's plugin blacklist.

SYSTEM INFO:
Kernel 4.20 with the ZEN Patchset configured with 1000khz timer and default cpu governor of performance.
I'm using wine-staging-nine as my winelib host as regular wine has some font rendering issues with my sforzando plugin. sforzando plugin is version 1.933. https://www.plogue.com/products/sforzando.html
The hardware I'm using is a i5 3570k, with 32GB RAM and an AMD RX480 GPU.
Desktop XFCE4.

ADDED CONTEXT:
LinVst like carla is a host of-sorts (lighter than carla) and it's not a stand-alone plugin in a way. I know confusing. Just an FYI on Archlinux and Ubuntu 16.04LTS and greater I haven't got the dssi-vst scanner thingy to work in your software with sforzando, this is why I use LinVst as it seems more reliable (never crashed on me). Anyway, on ardour at least when I have the plugin loaded multiple times I get a weird md5 hash like string of charachters for the plugin(s) name that is loaded, on muse they show up all as sforzando as they should on 3.0.2.
Finally, I think the cause of this may have to do with the fact that maybe the dssi-vst scanner supplied with muse and LinVst's supplied dssi-vst scanner are somehow getting picked up by your new scanner and it may be a conflict with that, but I don't know really.

FEATURE REQUEST (MAYBE):
Possible feature request wink wink. Maybe removing the old dead dssi-vst thing and creating a Lin2Win plugin bridge section using the superior winelib based LinVst is the way to go from now on. It supports everything the vestige header supports (incl. proper automation and sample accurate data chunks), oh and thus you don't need the steinberg SDK to build it. An extra bonus would be to implement a scanner feature that would allow it to create a .so file for every corresponding .dll in the VST's residing directory. :)
Heres a little script in bash I wrote that does exactly this, but from the shell:

Code: Select all

if [ -f $linvst ]; then
	for file in *.[Dd][Ll][Ll]; do
    	cp "/usr/bin/linvst.so" "$(basename "$file" .dll).so"
	done
	rm *.[Dd][Ll][Ll].so 2> /dev/null
else
	echo "Please Install linvst.so"
fi
ME BEING A PLUGIN ZEALOT:
In Linux gaming terms
LinVst --> DXVK of linux audio
dssi-vst --> d3d to OpenGL Classic of linux audio

This is all I did no other configs or scripts were used and I didn't do any plugin path exports in the .bashrc file.
Everything is default otherwise.
Haha, hope this helps.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Gotcha!
LinVst is outputting some stuff on stdout which our scanner picks up as
extra junk before the real xml portion.

Here is a small portion of the terminal output of MusE running with -D option (and with a special
debug output line added by me to the code to print the stdout array returned by the scanner QProcess).

Here it is scanning a Zyn plugin which works just fine, and a LinVst plugin (sforzando) which fails.

Notice the extra copyrights and stuff in the LinVst stdout array output. Our xml reader freaks out on that.
The xml reader expects output similar to the Zyn output.
Don't be fooled by that DSSI-VST text below, that ain't our dssi-vst support code, that's LinVst which is based on dssi-vst.

Stay tooned... Will try to fix...

Code: Select all

Checking file: </usr/local/lib/vst/ZynReverb.so>
pluginScan: stdout array:
<?xml version="1.0"?>
<muse version="3.1">
  <plugin file="/usr/local/lib/vst/ZynReverb.so" label="ZynReverb">
    <type>16</type>
    <class>1</class>
    <uniqueID>1515745910</uniqueID>
    <description>Reverb</description>
    <version>3.768.3</version>
    <maker>ZynAddSubFX Team</maker>
    <apiVersionMajor>2400</apiVersionMajor>
    <pluginVersionMajor>3</pluginVersionMajor>
    <pluginVersionMinor>3</pluginVersionMinor>
    <portCount>15</portCount>
    <inports>2</inports>
    <outports>2</outports>
    <ctlInports>11</ctlInports>
    </plugin>
  </muse>


Checking file: </home/me/.wine/drive_c/Program Files/Common Files/VST2/Plogue Art et Technologie, Inc/sforzando VST_x64.so>

pluginScan: Output from scan:
002a:fixme:process:GetProcessWorkingSetSizeEx (0xffffffffffffffff,0x22d7f8,0x22d7f0,(nil)): stub
002a:fixme:process:GetProcessWorkingSetSizeEx (0xffffffffffffffff,0x22d7f8,0x22d7f0,(nil)): stub
RemoteVSTServer::terminate: setting exiting flag

pluginScan: stdout array:
DSSI VST plugin server v0.986
Copyright (c) 2012-2013 Filipe Coelho
Copyright (c) 2010-2011 Kristian Amlie
Copyright (c) 2004-2006 Chris Cannam
LinVst version 2.32
Loading  /home/me/.wine/drive_c/Program Files/Common Files/VST2/Plogue Art et Technologie, Inc/sforzando VST_x64.dll
<?xml version="1.0"?>
<muse version="3.1">
  <plugin file="/home/me/.wine/drive_c/Program Files/Common Files/VST2/Plogue Art et Technologie, Inc/sforzando VST_x64.so" label="sforzando">
    <type>16</type>
    <class>3</class>
    <uniqueID>1347176273</uniqueID>
    <description>sforzando</description>
    <version>0.0.0</version>
    <maker>Plogue Art et Technologie, Inc</maker>
    <apiVersionMajor>2400</apiVersionMajor>
    <pluginFlags>3</pluginFlags>
    <portCount>514</portCount>
    <outports>2</outports>
    <ctlInports>512</ctlInports>
    <requiredFeatures>4</requiredFeatures>
    <vstPluginFlags>2047</vstPluginFlags>
    </plugin>
  </muse>

XML: level = 0
XML Parse Error at line 0 col 2

pluginScan FAILED: On readPluginScan(): file: /home/me/.wine/drive_c/Program Files/Common Files/VST2/Plogue Art et Technologie, Inc/sforzando VST_x64.so


Post Reply