NEW! Safe Plugin Scanner version 2 with cache

MusE is a DAW for Linux with both MIDI and Audio editing. https://muse-sequencer.github.io

Moderators: MattKingUSA, khz, spamatica

artofmusic
Established Member
Posts: 90
Joined: Mon Mar 17, 2014 8:01 pm

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by artofmusic »

Code: Select all

pluginScan: Output from scan:
002a:fixme:process:GetProcessWorkingSetSizeEx (0xffffffffffffffff,0x22d7f8,0x22d7f0,(nil)): stub
002a:fixme:process:GetProcessWorkingSetSizeEx (0xffffffffffffffff,0x22d7f8,0x22d7f0,(nil)): stub
RemoteVSTServer::terminate: setting exiting flag

pluginScan: stdout array:
DSSI VST plugin server v0.986
Copyright (c) 2012-2013 Filipe Coelho
Copyright (c) 2010-2011 Kristian Amlie
Copyright (c) 2004-2006 Chris Cannam
LinVst version 2.32
That's the part it doesn't like. At least your DAW can open the GUI like all the others except qtractor. And now that you mention it this copyright thing also has something to do with my Focusrite Windows EQ plugins as well that I use seldomly. Hopefully, this can be fixed soon as a lot of people use this wrapper over carla.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Whew! That was tough. Cross-platform IPC is a pain.
And that stdout flushing thing, mentioned below, was tough.

Hopefully with this fix we bring back more plugins into the fold.
Another user in the version #1 announcement thread had quite a few
plugin rejections as well. Hopefully this will help with many of them.

Please lemme know if any further trouble. Tim.

Fixed in MusE git master now. From ChangeLog:
- Fix plugin scanner bug reported by artofmusic in forum.
Some plugins may output stuff on stdout. This was confusing
our xml scanner. So a method of separating our xml from the
rest of stdout was added. Tested OK with LinVst and sforzando.
Some notes I included:
// NOTE: For IPC I looked at the ideal QLocalSocket and QLocalServer.
// But they require the Qt network module.
// QSharedMemory is nice but it's not serial like we need.
// And its fixed created size must be known beforehand.
// (But we may use it for audio in a future runtime sandbox.)
// And the problem with stdout or stderr is the plugin itself
// can put stuff there as well. Thus an extractor is needed.
// I tried putting our content on stderr, thinking flushing
// was not needed with it. Didn't work! My marker lines were
// inter-mingled with the error output lines of some plugins
// instead of the expected order. Not sure why except
// possibly those plugins run another process and we're
// seeing the delayed (skewed) output of that.
// Or maybe it was some QProcess channel buffering interfering?
// So stdout is used for communicating our content, and
// flushing is enforced (it was required) immediately
// BEFORE and AFTER our markers are output during the scan.
// It may still be possible that some plugins might run another
// process and output on stdout and we'll see delayed output
// inter-mingled with ours, but so far the stdout method
// works with all tested plugins so far.
// Ultimately a better marking method would help but if we
// are to use xml it might require marking EACH xml line
// which would be difficult. Tim.
spamatica
Established Member
Posts: 573
Joined: Mon Feb 08, 2010 10:38 am
Has thanked: 80 times
Been thanked: 97 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by spamatica »

Hey Tim,
cool thing, will be testing shortly!

I'm curious about the IPC thing though. Not sure what the cross-platform IPC you mention is, or did you simply mean inter-process?

And furthermore, why do you need IPC? Is the external process long lived so you send commands to it? Launching the process for each plugin and adding as argument a filename (QTemporaryFile), which would receive the XML, would have avoided the problems, or?
MusE DAW
artofmusic
Established Member
Posts: 90
Joined: Mon Mar 17, 2014 8:01 pm

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by artofmusic »

It works as expected now. There are still a few ladspa plugins that fail, but all in all they are detected. here is the stderr output of muse using -DD and running the pulseaudio backend. Also, attached in the zip file is the scanner cache for LinuxVST. These are the only errors I see now on stdout.


Checking file: </usr/lib/ladspa/blop_files/parabola_1649_data.so>

pluginScan FAILED: stripped stdout array is empty. Plugin did not return any info ??? file: /usr/lib/ladspa/blop_files/parabola_1649_data.so


Checking file: </usr/lib/ladspa/blop_files/sawtooth_1641_data.so>

pluginScan FAILED: stripped stdout array is empty. Plugin did not return any info ??? file: /usr/lib/ladspa/blop_files/sawtooth_1641_data.so


Checking file: </usr/lib/ladspa/blop_files/square_1643_data.so>

pluginScan FAILED: stripped stdout array is empty. Plugin did not return any info ??? file: /usr/lib/ladspa/blop_files/square_1643_data.so


It duplicates this message later in the output. Thanks, again.
Attachments
scanner_cache_and_output.zip
(45.47 KiB) Downloaded 82 times
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Hm, Robert Yeah I will try QTemporaryFile.
I looked at using normal files but I guess I was concerned about leaving them open
if the process should crash.
But upon second looks, QTemporaryFile actually fits the bill
because it is safe and self-destructing.

I will definitely try it. Thanks. Stay tooned...

Tim.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Ah, that's more like it! Freedom.

Thanks for your strong guidance Robert.
I knew from the very start that using files was a good way but that sticking point was the being left open thing.
It really really clouded my path, eh?

Now there's no usage at all of stdout or stderr. You can still of course see it with our -D option.
And our Xml class got a big boost. I will try to clean up some weird pre-existing QByteArray stuff (you marked it as "deep copy ?").

In MusE git master now. ChangeLog:
- Scanner: Replaced silly stdout stuff with QTemporaryFile.
- Class Xml now supports directly reading and writing a QIODevice
and writing to a QString, instead of requiring a FILE*.
- Using the above, eliminated two usages of fdopen().
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

I will try to clean up some weird pre-existing QByteArray stuff (you marked it as "deep copy ?").
Done! Xml class cleaned up greatly.
I think the reason that messy QByteArray stuff was in there was that in the long journey to switch
to Qt4 which was really a major Qt change, these things were added to make it 'just work',
rather verbosely without looking deeply at the code.

It appears to be nice and streamlined now. Should hopefully be OK.
Tested OK with several song files, and re-saving them. Stepping through with debug to check as well.
(Funny, the older I get, the more I can enact sweeping changes in just a few hours that perplexed me for long times
ages ago. Qt4 really was a major change with its QString and QByteArray, sources of much confusion)

Try 'er out. Lemme know if any trouble with xml files such as song files and stuff.
In MusE git master now. ChangeLog:
- Xml class full makeover.
Removed all QByteArray stuff. Now it works directly with _s1 or _s2,
which cures concerns about deep copying the b.a. back to _s1/_s2,
stuff which was hastily added at Qt4 to make it work.
Tightened up several of the routines.
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

Hello, some news here of my changes of the last few days. In git master now.

Please give it a twirl.

Code: Select all

      - Plugin scanner: Redesign fixes problems with 'auto-rescan' trigger. (Tim)
        Significant changes: Previously folders were scanned for date/time differences
         compared to the cache file's date/time. This caused unwanted rescans if ANY
         file or folder inside changed. Now it is all *.so file centric. Now, ALL *.so
         files found are registered in our caches, and a new 'file timestamp' member
         was added, and a new "unknown_plugins.scan" cache file added. Then on next start
         it compares date/time of what exists in our cache with what is actually on the system.
         
      - Fixed remaining compile issues with various options disabled.
        Tested by turning off ALL cmake options. Revealed an error: We never linked
         pthreads into the core! With cross-platform in mind, this was followed:
        https://stackoverflow.com/questions/49768454/whats-the-differences-between
         -pthread-and-pthread-options-in-cmake (the 'modern' answer at bottom).
Tim E. Real
Established Member
Posts: 660
Joined: Sat Sep 15, 2012 12:36 am
Has thanked: 36 times
Been thanked: 105 times

Re: NEW! Safe Plugin Scanner version 2 with cache

Post by Tim E. Real »

D'oh!
And a bug was revealed with duplicate LinuxVST entries !

Observed. Sorry about that. Was simple neglect to store LinuxVST plugin name as well as label.
Fixed in git master now.

After pulling the fix, please force a rescan either with the -R command line switch or
via Settings > Global Settings > Plugin Paths > Force Rescan.
(The caches need to be rebuilt to newly store the LinuxVST plugin names.)
That should finally make the double entries go away.
Post Reply