I’m one of the admins who manage CrowdStrike at my company.

We have all automatic updates disabled, because when they were enabled (according to the CrowdStrike best practices guide they gave us), they pushed out a version with a bug that overwhelmed our domain servers. Now we test everything through multiple environments before things make it to production, with at least two weeks of testing before we move a version to the next environment.

This was a channel file update, and per our TAM and account managers in our meeting after this happened, there’s no way to stop that file from being pushed, or to delay it. Supposedly they’ll be adding that functionality in now.

Yes, CrowdStrike says they don’t need to do conventional AV definitions updates, but the channel file updates sure seem similar to me.

The file they pushed out consisted of all zeroes, which somehow corrupted their agent and caused the BSOD. I wasn’t on the meeting where they explained how this happened to my company; I was one of the people woken up to deal with the initial issue, and they explained this later to the rest of my team and our leadership while I was catching up on missed sleep.

I would have expected their agent to ignore invalid updates, which would have prevented this whole thing, but this isn’t the first time I’ve seen examples of bad QA and/or their engineering making assumptions about how things will work. For the amount of money they charge, their product is frustratingly incomplete. And asking them to fix things results in them asking you to submit your request to their Ideas Portal, so the entire world can vote on whether it’s a good idea, and if enough people vote for it they will “consider” doing it. My company spends a fortune on their tool every year, and we haven’t been able to even get them to allow non-case-sensitive searching, or searching for a list of hosts instead of individuals.

Speaking as someone who manages CrowdStrike in my company, we do stagger updates and turn off all the automatic things we can.

This channel file update wasn’t something we can turn off or control. It’s handled by CrowdStrike themselves, and we confirmed that in discussions with our TAM and account manager at CrowdStrike while we were working on remediation.

Depends on where you are.

I’m in the Midwestern United States now, where summer is often pretty frustrating due to the high humidity. But I’m originally from Phoenix, where I really enjoyed summer (in the shade), because I love the feeling of warmth soaking into my bones, and I never got sweaty.

Can’t speak for the person you’re replying to, but I’m a security engineer and stuff still makes its way to me that you would think would get filtered out by others (and isn’t my job to fix). It just takes the right person thinking “this is obviously a problem with $system, let’s just send it straight over to them so they can fix it quickly!” And then we get the fun job of proving it’s not us and has no relation to us.

We got a ticket today for packet loss between two systems, neither of which have any of our tools on them…

That’s great! Like I said, it’s dependent on your employment contract. But for people who aren’t as certain, separate work and personal devices as much as possible just to protect yourself.

Depending on where you work, your employer may be able to take that personal device you’re using for work in the event of a lawsuit against the company (where they need to retain anything that may be relevant to discovery), or in the event of a security incident (where they may need it for forensics).

I work in information security, and I practice strict isolation for that exact reason. Two laptops, two phones, because if anything ever happens they can and will take devices for analysis or evidence. If you are using an issued device, they’ll assign you a new one; if it’s a personal device you’ll get it back when they’re done with it, which could take years.

Edited to add this is dependent on your employment contract, but it’s better to be safe than sorry. Cover your camera and use your work computer.

I’m a security engineer, and encryption is great, but can be bypassed. Relying on encryption assumes it was implemented properly, that the system was shut down properly so all keys were flushed correctly, and the encryption algorithm doesn’t have weaknesses.

Generally if somebody dedicated enough can acquire physical access to a system, they can probably find a way into it given the right resources. Did that happen here? Probably not. Could it have? Absolutely. That’s why most enterprises or government hard drives are shredded rather than just relying on them being wiped or encrypted.

Encryption is part of the solution, but it’s not automatically the complete solution.

We don’t know what was on those servers, but it was apparently sensitive enough that the government redacted descriptions of the data in court filings.

The US government brief said the relocated servers were not wiped before being moved to a new data center. The type of data on the relocated servers was apparently so sensitive that it could not be described in the US court filing, which redacts the sentence that describes what the servers contained.

https://arstechnica.com/tech-policy/2023/09/us-government-slams-musk-in-court-filing-describing-chaotic-environment-at-x/

I run those calls through my own phone system, which I host on a system in my basement. There are a couple main options out there, I used FreePBX for a while but now I’m using 3CX. They don’t require a ton of computing power-- mine runs on a virtual server inside a larger system, but you could run one off of an inexpensive thin client from eBay if you wanted to.

I get my phone number from VoIP.ms, which is pretty inexpensive and has worked well for me for years.

For a phone, you can either use a soft phone (an app on your computer or smartphone), or use an older IP phone off eBay (which is what I do since I also have a Plantronics wireless headset that connects to it).

It’s pretty easy to get started, but you do need to make sure you’re configuring everything correctly since selfhosted services can open up security holes in your network if you don’t know what you’re doing.

This is why I always record calls with major corporations when I’m talking about money. I’ve never had to actually resort to sending them recordings, but I have used the “Well, every call made from this phone is recorded, so I can go back and pull the recording of what I’ve been told if you don’t have it in your system” line a couple times.