Good compressor for mixbus

[merlyn]

To quote Frank Zappa mixing is about "getting all the instruments to sound as good as possible without flanging the piss out of them". Get all the instruments to sound as good as possible, and get them to sit together as well as possible. That's it. Making that your one task at this stage means you can concentrate on that, without having to deal with how loud, how, ahem, warm or how whizzy it sounds.

Now write that out as a 32 bit float file and import it into a new project. Now you can knock yourself out with loud (limiter), warm (saturator), and whizzy (enhancer).

[martibs]

GMac: Gold, thank you, exactly what I was hoping for!

[Jamesf]

I'm not sure exactly where mixing ends and mastering begins, so in that sense I guess do a bit of both as well. Grouping things the way @bluebell does is probably good practice anyhow.

As I've come to understand it, mixing is effectively an extension of arranging, while mastering takes the final mix as a whole and adds the tweaks that make it sound (hopefully) great in a particular medium.

So mixing is where you do panning, EQ, levels, compression etc. on individual channels to make them work well with all the others - and if you already arranged and performed it well, there won't be much to do beyond panning and maybe compression. It's also where effects are often added, especially delay, reverb, chorus/flanger/phaser etc.
Mastering is where you apply EQ and compression to the overall stereo (or 5.1, or whatever) mix to make it sound good on YouTube or CD.

I've learned a lot from Bobby Oswinski, Warren Huart and Ken Marshall

[Jamesf]

However, on the mixbus I struggle more. It's very easy to mess up the sound, and I'm also often not sure what qualities the compressor is supposed to add to the mix...

I feel like it would be easier to learn and experiment if I knew more what to expect from the compressor ... And maybe some words on what "kind" of compression the algorithm is based on?

The Ovnilab FAQ is a great source of information about the various types of compression, including some of the confusion people have about which does what. It's primarily about compression pedals for bass, but he describes it all in generally-applicable terms.

Following up on my previous comment about mixing vs mastering: what you're struggling with here is the mastering stage. I'm not a mastering engineer, nor do I play one on TV, but I can offer you the encouragement that every great mastering engineer I know of spend years struggling like you are now.
IIUC, it's not really supposed to add a character of its own; it's just there to match the dynamics to as well as possible to the delivery medium. Vinyl has one set of characteristics (like what happens when the grooves get too wide), while CD has another - if you're not already familiar with the Loudness War, that's a great example of where it was badly misused by many people.

Then there's the question of how much, and whether to just use one compression stage, or to chain several of them together with each doing just a tiny bit... yeah, I completely understand why you're confused and frustrated.