New sfArk utility for Linux

Discuss anything new and newsworthy! See http://planet.linuxaudio.org and https://libreav.org/news for more Linux Audio News!

Announcements of proprietary software may fit better in the Marketplace.


Moderators: raboof, MattKingUSA, khz

j_e_f_f_g
Established Member
Posts: 2032
Joined: Fri Aug 10, 2012 10:48 pm
Been thanked: 360 times

New sfArk utility for Linux

Post by j_e_f_f_g »

This thread is about an sfark utility that I consider deprecated.

If you're looking for an sfArk to sf2 converter, I recommend you check out http://linuxmusicians.com/viewtopic.php?f=24&t=10661

-- Raboof
I was downloading some soundfont files from the internet and, to my horror, discovered that some moron(s) had compressed the files using the proprietary, totally undocumented, completely unnecessary 'sfArk' format instead of a much more sensible, useful, well-tested, established open standard such as zip. This was especially disheartening because I'm using Linux, and the misguided company (Melody Machine) that devised sfArk, had written only a windows closed-source version of their crappy sfArk decompression software. Well actually, they did once write an awful closed-source linux version, but that out-dated, abandoned version doesn't run on anything but ancient Intel CPUs running a 32-bit OS. Oh yeah, the windows version has also long been abandoned, so forget about sfArk support for Windows on something other than old Intel CPUs. One would hope that musicians have learned not to store music in proprietary, undocumented formats. Companies go out of business, leaving customers without support, and that appears to be the case with Melody Machine... deservedly so.

Anyway, being a programmer myself, I decided I had enough of sfArk, so I wrote an open-source linux utility to extract the soundfonts from sfArk files, both version 1 and 2. It uses a very basic Gnome interface that any musician should be able to handle. Click on the "Load" button, and a file dialog pops up to let you select some sfArk file. (Initially the dialog lists only those files whose names end in .sfark, but you can instead view all files by changing the button labeled 'Sfark' to 'All'). After you pick your sfark file, that dialog disappears, and another dialog appears asking you to enter the desired name for the soundfont being extracted. Initially, the dialog is filled in with the original name of the soundfont, for your convenience. After you finish with this dialog, the utility does its job. A colored bar in the window shows how it is progressing. If all goes well, text will appear at the bottom of the window, saying the soundfont is successfully extracted, and you'll find a new soundfont file on your drive. If there's a problem, a message box will pop up with an error message.

I don't do QT, so if you want a KDE version, you'll have to modify the source code. It should be a very simple job for any QT programmer.

I also don't use a 32-bit linux. I use 64-bit on Intel CPUs. I've compiled a 64-bit version of the utility in the 'linux' folder, which should run on any modern AMD64 linux (with Gnome libraries installed). For example, I use it with Linux Mint XFCE.

If you're using a 32-bit linux, you'll need to compile the source. First, you need the gnu C compiler. If using Debian, just install the package "build-essential". You also need the gnome dev package "libgtk-dev". From a command terminal, in the same folder as sfark.c, type (all on one line):

gcc -mfpmath=387 -o sfark sfark.c `pkg-config --cflags --libs gtk+-3.0` -lpthread -export-dynamic

This will produce the executable "sfark". Copy it somewhere along with the file "sfark.glade".

Some words about the sfArk format. It really is awful, and needs to be removed from the universe. First, it breaks up a soundfont into its non-audio data, and audio data. The non-audio data is zip compressed. The audio data is compressed using one of several 16-bit delta compression schemes, depending upon which "compression level" the musician chose when he created his sfark using Melody Machine's atrocious windows utility. At the highest compression level (which is the utility's default setting, so most sfark files use it) the totally archiac LPC compression is added (in addition to the delta compression). The inept scheme this guy came up with uses integer to float conversion, float calculations, and convert float back to int. Hello, can you say "rounding errors"?? And then this bozo actually does a checksum based on those rounding errors. In effect, those errors are in the sfark file itself. Here's the horrible implication: if you don't perform the exact same rounding errors when you extract the soundfont from the sfark, THEN THE SOUNDFONT WILL BE CORRUPT. And what are the details of those rounding errors? That bozo compiled his software specifically to run on the old 80387 math coprocessor for Intel's ancient 80386 CPU (before the Pentium even). If you don't use the 80387 float format with its internal 20-bit precision, then you won't get the same rounding errors when you extract the soundfont from the sfark, AND THE SOUNDFONT WILL BE CORRUPT. If you compile my utility to use your Intel Core2 or i7's modern, fast SSE instructions, or AMD's 3dnow, etc, THEN THE SOUNDFONT WILL BE CORRUPT. Fortunately, the gnu C compiler has an option (-mfpmath=387) that says "Don't use newer floating point instructions. If the CPU supports old 80387 instructions (and the current Intel CPUs still do), use those. If the CPU doesn't, compile an entire 80387 software emulation into this program". Now I don't how well, or even if, gnu's emulation works on CPUs other than intel/AMD. For example, if it doesn't work on ARM, then if you compile my utility on an ARM CPU, it will not produce an error-free soundfont. (It will actually produce a legitimate soundfont file, despite reporting a checksum error. But the soundfont's waveforms will not be identical to the original. The sound will be altered). This is because the sfark format is technically flawed. It was written by a programmer who didn't know what he was doing, and didn't know that one doesn't use floating point calcs, with int conversions, in a supposedly lossless format. That's inept. Oh yeah, after the zip, delta, and LPC compression schemes are applied, then the resulting mess is run through a bit-packing compression scheme, a checksum is calculated and stored in a truly dreadful 300 byte header that is slapped onto the start of the sfark file. Atrocious.

And that's why sfark must be removed from the universe.

And that's version 2. You don't want to know about version 1. You do? ok, first it applies a crummy 16-bit delta compression, and then for inexplicable reasons, chops up the compressed data into 2 separate files. Then it takes these 2 files and combines them into 1 using the utterly obsolete LsPack compression format, another fine example of a proprietary, closed-source, undocumented format whose product was abandoned by its corporate entity, leaving endusers out of luck. Folks, use closed-source programs if you must, but make sure a program saves your data in an open, documented format. Don't let your work be locked and lost forever by corporate abandon-ware.

Use my utility. Free soundfonts from the horror of sfark. And then kill the beast.

Someday I may tackle the stench (and more fine corporate abandon-ware) calling itself "sfpack". Fortunately, it appears that most musicians learned from the hideous mistake of sfark, and wisely eschewed sfpack. Unfortunately, there are a few sfpack files, proving that some people are too dumb to learn even from their own mistakes.

Here's the link to download an archive containing the source, and 64-bit compiled exe. It's an sfark file that has been sfpack'ed. Kidding! I'm not an idiot. I used ZIP.

http://home.roadrunner.com/~jgglatt/mid ... /sfark.zip

Author of BackupBand at https://sourceforge.net/projects/backupband/files/
My fans show their support by mentioning my name in their signature.

brummer

Re: New sfArk utility for Linux

Post by brummer »

:lol: :lol: funy read, many thanks, and welcome j_e_f_f_g
User avatar
raboof
Established Member
Posts: 1865
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 52 times
Been thanked: 80 times
Contact:

Re: New sfArk utility for Linux

Post by raboof »

j_e_f_f_g wrote:If you're using a 32-bit linux, you'll need to compile the source. First, you need the gnu C compiler. If using Debian, just install the package "build-essential". You also need the gnome dev package "libgtk-dev".
It's 'gtk-3-dev' on my Debian. Is there any particular reason you're including zlib?
From a command terminal, in the same folder as sfark.c, type (all on one line):

gcc -mfpmath=387 -o sfark sfark.c `pkg-config --cflags --libs gtk+-3.0` -lpthread -export-dynamic
Might be nice to stick this into a Makefile.

Might be nice to have this as a commandline tool as well as a GUI tool. Any chance on getting this out on github or something similar?
Some words about the sfArk format (...)
Very impressive. How the hell did you find out about all of those intricacies?
User avatar
autostatic
Established Member
Posts: 1994
Joined: Wed Dec 09, 2009 5:26 pm
Location: Beverwijk, The Netherlands
Has thanked: 32 times
Been thanked: 104 times
Contact:

Re: New sfArk utility for Linux

Post by autostatic »

Thanks for this j_e_f_f_g! This will help out a lot of users!
User avatar
linuxdsp
Established Member
Posts: 147
Joined: Sun Mar 01, 2009 12:40 pm
Location: Oxford, England
Contact:

Re: New sfArk utility for Linux

Post by linuxdsp »

What license is this released under? Is it GPL - if so there appears to be no license included in the source? The detail in the code implementation implies a deeper knowledge of the algorithms than one might assume could be gained from empirical inspection of compressed sfArk files, and yet the description in the readme / forum post implies / reveals some naivety about how the compression algorithm works (once all the unnecessary personal insults directed at the hapless (unkown) "bozo" etc developer who created the original code have been filtered out..) Is this original code? Was there intended to be a build for Windows too (there's a lot of #ifdef'd windows stuff in there) or was this originally code for Windows?

I mostly agree with the comments about open vs closed / poprietary file formats, and there could be some valid technical points in there too, but it's disappointing they are buried amongst so much unnecessary emotive language directed at the original developer's perceived failings
j_e_f_f_g
Established Member
Posts: 2032
Joined: Fri Aug 10, 2012 10:48 pm
Been thanked: 360 times

Re: New sfArk utility for Linux

Post by j_e_f_f_g »

raboof wrote:It's 'gtk-3-dev' on my Debian.
Ah ok.
raboof wrote:Is there any particular reason you're including zlib?
Only to make it easier for non-programmers to compile it without needing to search for additional sources that may not be packaged for their distro. Is zlib compiled into a dynamic lib and shipped with most distros?
Might be nice to stick this into a Makefile.
Was going to do a makefile, but then I figured it's probably easier for a non-programmer to cut/paste that one line from readme.txt to a terminal window.
Might be nice to have this as a commandline tool
It's almost there. Notice that if you define WIN32, the compiler creates a Windows cmd line version. Just change the one line before the main() function (right before the gtk stuff) from this:

Code: Select all

#ifdef WIN32
to

Code: Select all

#ifdef 1
Then delete all the code down to the line #else, and replace with this:

Code: Select all

int main(int argc, char * argv[])
{
	register char *	str;
	SFARKHANDLE		sfark;

	if (argc < 2)
		printf("Please specify the desired sfArk filename!\n");

	else if (!(sfark = SfarkAllocA()))
		printf("Not enough RAM!\n");
	else
	{
		int			errCode;

		errCode = 0;

		if (!(errCode = SfarkOpen(sfark, argv[1])))
		{
			if (argc > 2)
				str = argv[2];
			else
				str = SfarkGetBuffer(sfark);

			if (!(errCode = SfarkBeginExtract(sfark, str)))
			{
				unsigned char	dots;

				dots = 0;
				do
				{
					if (dots != SfarkPercent(sfark))
					{
						printf("*");
						++dots;
					}
				} while (!(errCode = SfarkExtract(sfark)));
				if (errCode > 0) errCode = 0;
				printf("\n");
			}
		}

		str = (char *)SfarkErrMsg(sfark, errCode);
		printf("\n%s\n", str);

		SfarkFree(sfark);
	}

 	return 0;
}
Compile. There you go.
Any chance on getting this out on github or something similar?
I've never used all those services. It's easiest for me to just put it on my own site. If you want to make it available elsewhere, go ahead and do it.
How the hell did you find out about all of those intricacies?
Reverse-engineering. It's a hobby.

Author of BackupBand at https://sourceforge.net/projects/backupband/files/
My fans show their support by mentioning my name in their signature.

j_e_f_f_g
Established Member
Posts: 2032
Joined: Fri Aug 10, 2012 10:48 pm
Been thanked: 360 times

Re: New sfArk utility for Linux

Post by j_e_f_f_g »

linuxdsp wrote:The detail in the code implementation implies a deeper knowledge of the algorithms than one might assume could be gained from empirical inspection of compressed sfArk files
Indeed. It's not a "clean room" implementation. Therefore no license. But there is original code there, and rewritting of algorithms where it was feasible.
linuxdsp wrote:a build for Windows too
I usually do coding and testing on Windows, and then port to linux. Frankly, I think MS Dev Studio 6 was the best IDE and debugging environment (ruined when MS went to .NET). I vastly prefer it to linux tools. So yes, it started as a basic Win32 "Console app" in MSVC6 for coding/testing.
linuxdsp wrote:so much unnecessary emotive language directed at the original developer's perceived failings
Sorry but that programmer made a major blunder that likely in the near future WILL MAKE IT IMPOSSIBLE TO EXTRACT SFARK FILES WITHOUT CORRUPTING THE AUDIO DATA (unless you keep an old pentium machine around). That's utterly awful. Not only do I consider sfark to be a totally unnecessary thing that would have been best never created, it's broken by design. I stand by my assessment 100%. It MUST die.

Author of BackupBand at https://sourceforge.net/projects/backupband/files/
My fans show their support by mentioning my name in their signature.

User avatar
raboof
Established Member
Posts: 1865
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 52 times
Been thanked: 80 times
Contact:

Re: New sfArk utility for Linux

Post by raboof »

j_e_f_f_g wrote:Is zlib compiled into a dynamic lib and shipped with most distros?
I'm pretty sure it is
j_e_f_f_g wrote:
raboof wrote:Any chance on getting this out on github or something similar?
I've never used all those services. It's easiest for me to just put it on my own site. If you want to make it available elsewhere, go ahead and do it.
https://github.com/raboof/unsfark - i butchered it a bit already.
j_e_f_f_g
Established Member
Posts: 2032
Joined: Fri Aug 10, 2012 10:48 pm
Been thanked: 360 times

Re: New sfArk utility for Linux

Post by j_e_f_f_g »

Can you do a KDE frontend for it?

Author of BackupBand at https://sourceforge.net/projects/backupband/files/
My fans show their support by mentioning my name in their signature.

User avatar
raboof
Established Member
Posts: 1865
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 52 times
Been thanked: 80 times
Contact:

Re: New sfArk utility for Linux

Post by raboof »

j_e_f_f_g wrote:Can you do a KDE frontend for it?
Nah, I suck at GUI's, plus I find the concept of creating both a GTK and a Qt version of the same app a bit silly.

I'm curious, in a couple of places I see you do

Code: Select all

&sumtable[0]
when passing an array to a function. Why? Seems like a roundabout way of doing things.
User avatar
linuxdsp
Established Member
Posts: 147
Joined: Sun Mar 01, 2009 12:40 pm
Location: Oxford, England
Contact:

Re: New sfArk utility for Linux

Post by linuxdsp »

Indeed. It's not a "clean room" implementation. Therefore no license. But there is original code there, and rewritting of algorithms where it was feasible.
I think (and I hope) its important to have some clarification regarding the license - it might be useful for someone to use in their own project, but without clarification about the originality / origin / permissions granted for re-use that isn't possible. If it's 100% original then why not release it as GPL (or any other form of license - public domain or otherwise). If it's not 100% original, where did it come from?, and do you (j_e_f_f_g) have permission from the copyright holders to publish it (normally this means in whole or in part). Regardless of the politics around proprietary code / file formats etc, the creator of the code normally has a copyright on it and I think its important to acknowledge that (in fact, copyright is an important part of the GPL too). If it's not clear what the license is, others put themselves at risk by taking the work or a derivative of it and publishing it.
User avatar
briandc
Established Member
Posts: 1442
Joined: Sun Apr 29, 2012 3:17 pm
Location: Italy
Has thanked: 58 times
Been thanked: 28 times
Contact:

Re: New sfArk utility for Linux

Post by briandc »

j_e_f_f_g wrote:I was downloading some soundfont files from the internet and, to my horror, discovered that some moron(s) had compressed the files using the proprietary, totally undocumented, completely unnecessary 'sfArk' format instead of a much more sensible, useful, well-tested, established open standard such as zip. This was especially disheartening because I'm using Linux, and the misguided company (Melody Machine) that devised sfArk, had written only a windows closed-source version of their crappy sfArk decompression software. Well actually, they did once write an awful closed-source linux version, but that out-dated, abandoned version doesn't run on anything but ancient Intel CPUs running a 32-bit OS. Oh yeah, the windows version has also long been abandoned, so forget about sfArk support for Windows on something other than old Intel CPUs. One would hope that musicians have learned not to store music in proprietary, undocumented formats. Companies go out of business, leaving customers without support, and that appears to be the case with Melody Machine... deservedly so.

Anyway, being a programmer myself, I decided I had enough of sfArk, so I wrote an open-source linux utility to extract the soundfonts from sfArk files, both version 1 and 2. It uses a very basic Gnome interface that any musician should be able to handle. Click on the "Load" button, and a file dialog pops up to let you select some sfArk file. (Initially the dialog lists only those files whose names end in .sfark, but you can instead view all files by changing the button labeled 'Sfark' to 'All'). After you pick your sfark file, that dialog disappears, and another dialog appears asking you to enter the desired name for the soundfont being extracted. Initially, the dialog is filled in with the original name of the soundfont, for your convenience. After you finish with this dialog, the utility does its job. A colored bar in the window shows how it is progressing. If all goes well, text will appear at the bottom of the window, saying the soundfont is successfully extracted, and you'll find a new soundfont file on your drive. If there's a problem, a message box will pop up with an error message.

I don't do QT, so if you want a KDE version, you'll have to modify the source code. It should be a very simple job for any QT programmer.

I also don't use a 32-bit linux. I use 64-bit on Intel CPUs. I've compiled a 64-bit version of the utility in the 'linux' folder, which should run on any modern AMD64 linux (with Gnome libraries installed). For example, I use it with Linux Mint XFCE.

If you're using a 32-bit linux, you'll need to compile the source. First, you need the gnu C compiler. If using Debian, just install the package "build-essential". You also need the gnome dev package "libgtk-dev". From a command terminal, in the same folder as sfark.c, type (all on one line):

gcc -mfpmath=387 -o sfark sfark.c `pkg-config --cflags --libs gtk+-3.0` -lpthread -export-dynamic

This will produce the executable "sfark". Copy it somewhere along with the file "sfark.glade".

Some words about the sfArk format. It really is awful, and needs to be removed from the universe. First, it breaks up a soundfont into its non-audio data, and audio data. The non-audio data is zip compressed. The audio data is compressed using one of several 16-bit delta compression schemes, depending upon which "compression level" the musician chose when he created his sfark using Melody Machine's atrocious windows utility. At the highest compression level (which is the utility's default setting, so most sfark files use it) the totally archiac LPC compression is added (in addition to the delta compression). The inept scheme this guy came up with uses integer to float conversion, float calculations, and convert float back to int. Hello, can you say "rounding errors"?? And then this bozo actually does a checksum based on those rounding errors. In effect, those errors are in the sfark file itself. Here's the horrible implication: if you don't perform the exact same rounding errors when you extract the soundfont from the sfark, THEN THE SOUNDFONT WILL BE CORRUPT. And what are the details of those rounding errors? That bozo compiled his software specifically to run on the old 80387 math coprocessor for Intel's ancient 80386 CPU (before the Pentium even). If you don't use the 80387 float format with its internal 20-bit precision, then you won't get the same rounding errors when you extract the soundfont from the sfark, AND THE SOUNDFONT WILL BE CORRUPT. If you compile my utility to use your Intel Core2 or i7's modern, fast SSE instructions, or AMD's 3dnow, etc, THEN THE SOUNDFONT WILL BE CORRUPT. Fortunately, the gnu C compiler has an option (-mfpmath=387) that says "Don't use newer floating point instructions. If the CPU supports old 80387 instructions (and the current Intel CPUs still do), use those. If the CPU doesn't, compile an entire 80387 software emulation into this program". Now I don't how well, or even if, gnu's emulation works on CPUs other than intel/AMD. For example, if it doesn't work on ARM, then if you compile my utility on an ARM CPU, it will not produce an error-free soundfont. (It will actually produce a legitimate soundfont file, despite reporting a checksum error. But the soundfont's waveforms will not be identical to the original. The sound will be altered). This is because the sfark format is technically flawed. It was written by a programmer who didn't know what he was doing, and didn't know that one doesn't use floating point calcs, with int conversions, in a supposedly lossless format. That's inept. Oh yeah, after the zip, delta, and LPC compression schemes are applied, then the resulting mess is run through a bit-packing compression scheme, a checksum is calculated and stored in a truly dreadful 300 byte header that is slapped onto the start of the sfark file. Atrocious.

And that's why sfark must be removed from the universe.

And that's version 2. You don't want to know about version 1. You do? ok, first it applies a crummy 16-bit delta compression, and then for inexplicable reasons, chops up the compressed data into 2 separate files. Then it takes these 2 files and combines them into 1 using the utterly obsolete LsPack compression format, another fine example of a proprietary, closed-source, undocumented format whose product was abandoned by its corporate entity, leaving endusers out of luck. Folks, use closed-source programs if you must, but make sure a program saves your data in an open, documented format. Don't let your work be locked and lost forever by corporate abandon-ware.

Use my utility. Free soundfonts from the horror of sfark. And then kill the beast.

Someday I may tackle the stench (and more fine corporate abandon-ware) calling itself "sfpack". Fortunately, it appears that most musicians learned from the hideous mistake of sfark, and wisely eschewed sfpack. Unfortunately, there are a few sfpack files, proving that some people are too dumb to learn even from their own mistakes.

Here's the link to download an archive containing the source, and 64-bit compiled exe. It's an sfark file that has been sfpack'ed. Kidding! I'm not an idiot. I used ZIP.

http://home.roadrunner.com/~jgglatt/mid ... /sfark.zip
As a non-programmer, this explanation is very educational and was even fun to read! :D

My question: wouldn't it just be easier to contact the one who provided the sfArk file and kindly ask them to compress the original in zip format? That would skip all the risks involved in the "rounding errors" and different computer set-ups.

I agree, sfArk is not nice. I've downloaded some in the past, wasn't able to unzip them, and decided (in my profound inexpertise) that they weren't worth my time. (kind of like the "sour grapes" story..)
But now I see that there are additional reasons to not like sfArk.

I'd be curious to try to compile your utility (I use 32-bit) though, just for the satisfaction of opening an sfArk file to be able to say "I did it!" :)

brian
Have your PC your way: use linux!
My sound synthesis biome: http://www.linuxsynths.com
j_e_f_f_g
Established Member
Posts: 2032
Joined: Fri Aug 10, 2012 10:48 pm
Been thanked: 360 times

Re: New sfArk utility for Linux

Post by j_e_f_f_g »

briandc wrote:Wouldn't it just be easier to contact the one who provided the sfArk file and kindly ask them to compress the original in zip format?
It would be ideal if all sfark files were gotten rid of. But unfortunately, there are still some out there, with no alternate format available.

Author of BackupBand at https://sourceforge.net/projects/backupband/files/
My fans show their support by mentioning my name in their signature.

UncleDoug
Posts: 1
Joined: Mon Oct 22, 2012 4:10 am

Re: New sfArk utility for Linux

Post by UncleDoug »

Thanks, very much, for this, Jeff!

I'm an old *nix geek and a sometime amateur acoustical musician who has recently started to play with my music in the digital world.

The first thing I had to make myself do was to actually *learn* (for the first time since Linux and I were much younger) how the very annoying and overlapping sound system components interact (I'm a sysadmin and a network guy -- never really had to know this before). OK, getting there.

Just in the past couple of weeks, as I've been exploring soundfonts, I've come across this truly braindead sfArk compression mess -- and discovered that a large percentage of the soundfont packages floating around have been mangled by it. Reverse engineering it might not have been beyond my ability, but it would have been damned difficult for me (I'm a systems guy more than a coder), and it would definitely have been beyond my patience (now that I'm old and grumpy). ;^)

If I hadn't stumbled upon your little utility, I would have just ignored everything in sfArk format. Thanks to you, I don't have to, and I've actually found some very nice and usable soundfonts.

You're a star in my book. Thanks again!

~Doug
User avatar
raboof
Established Member
Posts: 1865
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 52 times
Been thanked: 80 times
Contact:

Re: New sfArk utility for Linux

Post by raboof »

j_e_f_f_g wrote:
Any chance on getting this out on github or something similar?
I've never used all those services. It's easiest for me to just put it on my own site. If you want to make it available elsewhere, go ahead and do it.
Hi Jeff,

Some people have been trying out unsfark and ran into some issue on MacOSX (partly solved):
https://github.com/raboof/unsfark/issues/1 . Could you take a look at it?
Post Reply