linuxaudio.org compromised - 2018-01-29
Moderators: MattKingUSA, khz
- autostatic
- Established Member
- Posts: 1994
- Joined: Wed Dec 09, 2009 5:26 pm
- Location: Beverwijk, The Netherlands
- Has thanked: 32 times
- Been thanked: 104 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
We're in the process of rebuilding everything on alternate servers as the forensics procedure at Virginia Tech simply takes too long. So hopefully tomorrow or beginning of next week we can flick the DNS switch.
Short recap, someone or something (this was probably an automated attack) probably got a reverse shell and exploited a local privilege escalation vulnerability, in this case Dirty COW. That's a somewhat older vulnerability which we could've mitigated by rebooting the server more often. The server was updated regularly but we were simply too sloppy with rebooting it as the linuxaudio.org is a hardware server sitting in some server room and there was some concern it wouldn't come back properly after a reboot.
The alternate servers are VM's so rebooting shouldn't be an issue anymore. They're also located in the EU on a fully open source cloud solution (OpenStack).
Regarding Twitter, unfortunately I have no access to that account. And we already have good backups and after the move that part is covered too.
Jeremy
Short recap, someone or something (this was probably an automated attack) probably got a reverse shell and exploited a local privilege escalation vulnerability, in this case Dirty COW. That's a somewhat older vulnerability which we could've mitigated by rebooting the server more often. The server was updated regularly but we were simply too sloppy with rebooting it as the linuxaudio.org is a hardware server sitting in some server room and there was some concern it wouldn't come back properly after a reboot.
The alternate servers are VM's so rebooting shouldn't be an issue anymore. They're also located in the EU on a fully open source cloud solution (OpenStack).
Regarding Twitter, unfortunately I have no access to that account. And we already have good backups and after the move that part is covered too.
Jeremy
- GraysonPeddie
- Established Member
- Posts: 661
- Joined: Sun Feb 12, 2012 11:12 pm
- Location: Altha, FL
- Been thanked: 6 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
OpenStack!!! That is something I'd like to learn in near future. This could be interesting for setting this up in my home environment even if people would tell me that is overkill as if an Ubiquiti UniFi 48-Port 500W PoE is overkill (48-port will be used for 4 video cameras, in-wall tablets, and a Doorbird powered by PoE, so I will have use of it when building a house in the near future).
OpenStack and Ubiquiti products (excluding AmpliFi) aren't designed for consumers in a home environment, but I'm more of a guy who likes having industrial-type products such as 1.5U custom-built servers and a 1U switch.
Anyway, good luck on getting the websites back online.
And yes, data forensics does take a while. It's important to preserve the data at all times for investigation and make sure all the access times are not updated upon touching the files in the filesystem.
PS: And yes, OpenStack does make sense for a large business environments as it's more for those who are looking to setup a hybrid cloud. I'm not certain if there are businesses out there that are using OpenStack internally as a private IaaS (Infrastructure as a Service) cloud.
OpenStack and Ubiquiti products (excluding AmpliFi) aren't designed for consumers in a home environment, but I'm more of a guy who likes having industrial-type products such as 1.5U custom-built servers and a 1U switch.
Anyway, good luck on getting the websites back online.
And yes, data forensics does take a while. It's important to preserve the data at all times for investigation and make sure all the access times are not updated upon touching the files in the filesystem.
PS: And yes, OpenStack does make sense for a large business environments as it's more for those who are looking to setup a hybrid cloud. I'm not certain if there are businesses out there that are using OpenStack internally as a private IaaS (Infrastructure as a Service) cloud.
--Grayson Peddie
Music Interest: New Age w/ a mix of modern smooth jazz, light techno/trance & downtempo -- something Epcot Future World/Tomorrowland-flavored.
Music Interest: New Age w/ a mix of modern smooth jazz, light techno/trance & downtempo -- something Epcot Future World/Tomorrowland-flavored.
- briandc
- Established Member
- Posts: 1442
- Joined: Sun Apr 29, 2012 3:17 pm
- Location: Italy
- Has thanked: 58 times
- Been thanked: 28 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
A big "Thank you!" to everyone involved in helping with this. I was glad to hear there were backups!
brian
brian
Have your PC your way: use linux!
My sound synthesis biome: http://www.linuxsynths.com
My sound synthesis biome: http://www.linuxsynths.com
- chaocrator
- Established Member
- Posts: 313
- Joined: Fri Jun 26, 2015 8:11 pm
- Location: Kyiv, Ukraine
- Been thanked: 1 time
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
it is usable as a private IaaS cloud, but requires some knowledge how to set it up with simpler network infrastructure, because that one in official openstack documentation is certainly overcomplicated.GraysonPeddie wrote:I'm not certain if there are businesses out there that are using OpenStack internally as a private IaaS (Infrastructure as a Service) cloud.
- GraysonPeddie
- Established Member
- Posts: 661
- Joined: Sun Feb 12, 2012 11:12 pm
- Location: Altha, FL
- Been thanked: 6 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
Even if I use conjure-up in Ubuntu?
--Grayson Peddie
Music Interest: New Age w/ a mix of modern smooth jazz, light techno/trance & downtempo -- something Epcot Future World/Tomorrowland-flavored.
Music Interest: New Age w/ a mix of modern smooth jazz, light techno/trance & downtempo -- something Epcot Future World/Tomorrowland-flavored.
- autostatic
- Established Member
- Posts: 1994
- Joined: Wed Dec 09, 2009 5:26 pm
- Location: Beverwijk, The Netherlands
- Has thanked: 32 times
- Been thanked: 104 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
Just like to add my thanks for all your hard work. Its often the case that we don’t fully appreciate what we have until its not there.
Cheers
Cheers
-
- Established Member
- Posts: 1067
- Joined: Mon May 12, 2014 7:11 am
- Has thanked: 15 times
- Been thanked: 36 times
Re: linuxaudio.org compromised - 2018-01-29
I read in IRC the server was hacked ( my layman's term) but at that time I didn't realize kx and lmp depended on that server too.
Anyway, many thanks for taking care of this!
Let us all be patient and let the guys do their work
Anyway, many thanks for taking care of this!
Let us all be patient and let the guys do their work
Re: linuxaudio.org compromised - 2018-01-29
Thank you for the hard work and great resources. I'd like to support your work with a small donation... Where does one go for that? It would be great to be able to do that on Liberapay!
-
- Established Member
- Posts: 36
- Joined: Sat Nov 19, 2016 4:45 am
- Has thanked: 2 times
- Been thanked: 2 times
Re: linuxaudio.org compromised - 2018-01-29
I would also be happy to support you through a small liberapay donation.chtfn wrote:Thank you for the hard work and great resources. I'd like to support your work with a small donation... Where does one go for that? It would be great to be able to do that on Liberapay!
- bluebell
- Established Member
- Posts: 1927
- Joined: Sat Sep 15, 2012 11:44 am
- Location: Saarland, Germany
- Has thanked: 113 times
- Been thanked: 122 times
Re: linuxaudio.org compromised - 2018-01-29
Thanks to all who contribute.
Linux – MOTU UltraLite AVB – Qtractor – http://suedwestlicht.saar.de/
- autostatic
- Established Member
- Posts: 1994
- Joined: Wed Dec 09, 2009 5:26 pm
- Location: Beverwijk, The Netherlands
- Has thanked: 32 times
- Been thanked: 104 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
So far the progress is slow. We have to deal with a timezone difference, I'm in CET while the current server and the Virginia Tech department hosting the server are in EST, and also the communication itself is not optimal. And then there's a another time issue, I can't put all my available time into restoring the server, I have a responsible day job, a family with two kids and several bands I rehearse with. We also lost some time over discussing whether linuxaudio.org should move away from the VT server or not.
Luckily I got some help for the mail services and the owner of the linuxaudio.org domain is standing by to change the DNS. And your kind words certainly help too!!! Many thanks for the support!
Jeremy
Luckily I got some help for the mail services and the owner of the linuxaudio.org domain is standing by to change the DNS. And your kind words certainly help too!!! Many thanks for the support!
Jeremy
- autostatic
- Established Member
- Posts: 1994
- Joined: Wed Dec 09, 2009 5:26 pm
- Location: Beverwijk, The Netherlands
- Has thanked: 32 times
- Been thanked: 104 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
First sites are starting to work again:
- kxstudio.linuxaudio.org
- kokkinizita.linuxaudio.org
- download.linuxaudio.org
- lac.linuxaudio.org/2018
-
- Established Member
- Posts: 2083
- Joined: Mon Sep 28, 2015 8:06 pm
- Location: Here, of course!
- Has thanked: 232 times
- Been thanked: 400 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
Great news!
The Yoshimi guy {apparently now an 'elderly'}
- autostatic
- Established Member
- Posts: 1994
- Joined: Wed Dec 09, 2009 5:26 pm
- Location: Beverwijk, The Netherlands
- Has thanked: 32 times
- Been thanked: 104 times
- Contact:
Re: linuxaudio.org compromised - 2018-01-29
There might be some issues with SSL certificates, I revoked them all and renewed a few ones. We weren't using HSTS yet so the sites that are up should be accessible to everyone.
Tomorrow we'll move on and hopefully we can also get the mailing lists back online again.
Tomorrow we'll move on and hopefully we can also get the mailing lists back online again.