Laptop occasionally doesn't switch from speakers to headph.

[Broomy]

As stated in the title. I have this problem every now and then when using my onboard soundcard, but I can't lay my finger on it.
Sometimes I can switch to my headphones several times without any problem (by plugging them in) and sometimes the laptop keeps on playing audio through the build in speakers.
I've tried several headphones, restarting the laptop (with either the headphones already plugged in or not), using the ALSA mixer. But nothing works. And after while everything is fine again.

I use a usb-soundcard and the onboard soundcard on and off. Can this be an issue?