Hate to be that guy but if you automatically patch critical infrastructure or apply patches without reading their description first, you kinda did it to yourself. There’s a very good reason not a single Linux distribution patches itself (by default) and wants you to read and understand the packages you’re updating and their potential effects on your system
While you are generally correct, in this case the release notes labeled this as a security update and not an OS upgrade. The fault for this is Microsoft’s not the sysadmin.
Many distros (at least Ubuntu) auto-installs security updates, and here a mislabeled “security update” was auto-installed. This is not the fault of the sysadmins.
here a mislabeled “security update” was auto-installed.
To be fair, you would have to read all the way to the first paragraph to get this information from the article. Hard to blame people for not knowing this critical bit of information when it was buried so deep
There’s a lot of people out there running automation to keep their servers secure. Well I agree any automation out there should be able to flag and upgrade excluded, It would seem to me like Microsoft should own some of the blame for a full ass hard to uninstall OS update fed in with the same stream and without it interaction. I kind of expect my OS in stall pop up a window and say hey a****** this is going to upgrade your system, are you cool with that. I don’t know how it works these days but I know back in the day going between versions you would have to refresh your licensing on a large upgrade.
Unlike with other OSes Microsoft releases all of their patches on Tuesday at around the same time in one big batch. I spend my Tuesday morning reading the patch descriptions and selectively applying them. A method that hasn’t failed me once.
Yeah, I’m using Ninja on about 120 boxes. It’s set to auth critical only. If someone reports a problem, we’ll go ahead and blacklist that update temporarily while we sorted out even though it’s semi-automated they never happen all at once there’s always a couple of canaries that get up a little early.
Meanwhile I’ve still got customers who are running CentOS 6.
We have an app running on CentOS 6. The vendor of the app informed us they expect to have a new version that can run on RHEL 8 by the end of the year - 2025.
As is tradition
“Labeling error”
Lol, okay.
Why do my windows upgrades never run this smoothly?
Since rolling back to the previous configuration will present a challenge, affected users will be faced with finding out just how effective their backup strategy is or paying for the required license and dealing with all the changes that come with Windows Server 2025.
Accidentally force your customers to have to spend money to upgrade, how convenient.
Congratulation, you are being upgraded. Please do not resist. And pay while we are at it.
We are the Borg.
I have a message and a question.
A message from ESR and a question from me.
Since MS forced the upgrade, you should get 2025 for free. That would probably be really easy to argue in court
Ah, but did you read the article?
MS didn’t force it, Heimdal auto-updated it for their customers based on the assumption that Microsoft would label the update properly instead of it being labeled as a regular security patch. Microsoft however made a mistake (on purpose or not? Who knows…) in labeling it.
Then it’s still on Microsoft for pushing that update through what is essentially a patch pipeline
MS will be sued over this and they will lose. This is not an ambiguous case. They fucked up. It’s essentially an unconsentual/unilateral alteration to a contract, which kinda violates the principle of, you know, a contract.
It is, but they never forced anyone to take the update, so that might save their asses, or it might not
This would be no different to you ordering food in a restaurant, them bringing you the wrong meal, you refusing because you didn’t order it, then they tell you to go fuck yourself and charge you for it anyway.
If this argument is valid in your judicial system then you live in a clown world capitalist dictatorship.
Have you seen the state of the US? A “clown world capitalist dictatorship” is a pretty apt description
Why yes … I am aware 1+1=2
I’m saying they might send people the bill and then these people (well, companies) are going to have to fight it in court, where they’ll be right for sure, but Microsoft can make a lot of stupid arguments to prolong the whole thing, to the point where it’s cheaper to pay the license fee. For one they could say that continued use of the operating system constitutes agreement to licenses and pricing.
Either way this is server 2025 not windows 12. We’re talking about companies here, not people.
Yes, and I’m saying that the fact this could even be viewed by Microsoft as something that is worth going to trial, and being argued in court = hyper-capitalist dystopian dictatorship.
In a sane world not “by and for corporations”, this tactic would not even be in the realm of plausibility.
M$'s mistake creates no obligation to pay, either way. They cannot sue anyone for the extra money.
But some customers (depending on their legislation) might sue M$ to make broken systems running again, for example if these systems have stopped now with a ‘missing license’ error message.
Uh, if they didn’t ask for it, how is Microsoft going to make them pay for it?
Good luck arguing with Ms if you aren’t a giant company
Misleading title. It was installed by a third-party updater, Heimdall, but MS labeled a Windows 11 update wrong.
They labelled an OS version upgrade as a security update.
Yet another reason to not do auto-updates in an enterprise environment for mission-critical services.
In an enterprise environment, you rely on a service that tracks CVEs, analyzes which ones apply to your environment, and prioritizes security critical updates.
The issue here is that one of these services installed a release upgrade because Microsoft mislabelled it as security update.Should still be doing phased rollouts of any patches, and where possible, implementing them on pre-prod first.
Pre-prod is ideal, but a pipe dream for many. Lots of folks barely get prod.
We still stagger patching so things like this only wipe some of the critical infrastructure, but that still causes needless issues.
For security updates in critical infrastructure, no. You want that right away, in best case instant. You can’t risk a zero day being used to kill people.
Even security updates can be uncritical or supercritical. Consult the patch notes or get burned lol
Wrong.
Microsoft labelled the update as a security update
Do you know that’s not a mistake and done fully malicously knowing that? Please give me your source.
Read the fucking article.
The patch id couldnt be any clearer.
And you make absolutely no error?
Besides that:
Should MS have caught the errorenous ID (assuming it truly was errourneous and not knowingly falsely labeled)? Absolutely. Should the patch management team blindly release all updates that MS releases? No?
I’m truly, totally, completely shocked … that Windows is still being used on the server side.
We run a lot of Windows servers for specialized applications that don’t really have viable alternatives. It sucks, but it’s the same reason we use Windows clients.
Basically AD and the workstation management that uses it. Could all be run on a VM and snapshotted because you know it’s going to fuck up an update eventually. Perhaps SQL Server but that’s getting harder to justify the expense of anymore.
A bunch of enterprise services are Windows only. Also Active Directory is by far the best and easiest way to manage users and computers in an org filled with a bunch of end users on Windows desktops. Not to mention the metric shitload of legacy internal asp applications…
Linux does AD. Don’t let that stop you from switching.
No not really. It does the various services for the most part, but Active Directory is exclusively a Microsoft product. Group Policy in particular also does not have a drop in replacement that’s any sort of sane.
Yeah at work we do a lot of internal microsoft asp stuff, poweshell, AD, ms access, all that old legacy ms stuff
Is powershell “legacy”?
Windows Powershell sort of is legacy, but Powershell 7 definitely isn’t
I guess not actually but the amount of weird bugs I got from running a working script makes me think there’s something wrong with the way we have ours set up.
It must have been the same fun as when back in 2012 (or 2013?) McAfee (at least I think it was them) identified /system32 as a threat and deleted it :)
One of the few things that accursed software actually got right!
Haha, that’s great!
Do system administrators still exist? Honest question. I was one of those years ago and layoffs, forced back to office bullshit drove me away
yes, but we spend most of our time in meetings with cloud service vendors now.
I haven’t been inside the server room for a month.I only go in the server room to t-pose in front of the giant air conditioner to cool off.
I’m not necessarily talking about being in the server room, I’m talking about more like doing power shell stuff and the stuff you would think system administrators do. They are still teaching active directory in IT classes in college
Yes, this is still a crucial job role for most organizations.
I knew a guy with almost that exact resume, except he told me it was chickens. He worked in Lagos during the week and went back to his chickens in rural Nigeria on the weekend.
I think they call them devops now.
I still prefer sysop.
Same.
There are dozens of us (working for MSPs because in house doesn’t pay as well and companies are cheap and want to outsource that cost center)!
I switched from an MSP to a unionized in-house position, doubled my salary and my days of paid time off.
I worked for a classic MSP a while back, barely lasted 3 months. Such a toxic environment, tons of pressure to spread yourself thinner and thinner.
It was one of those places where you were expected to be there an hour early, stay an hour late, and work through your lunch.
Even though that’s illegal, it was never explicit, just one of those, wink wink type things. But the workload was always so heavy, you couldn’t stay on top of everything unless you were working 50+ hours a week.
And of course, all salary, no overtime or double time for weekend work.
I do internal IT now, much better. Trying to get my own one-person shop going to eventually be fully self-employed. Actually, it would be really cool to become a worker-owned co-op, but that’s still a faint dream.
Currently in an MSP. It’s all on the company culture as to if it’s shit or not. We’re fully wfh with no plans to move back to the office.
Overtime is never forced. If we have to work through lunch because all hell is breaking loose, we’re practically encouraged to leave an hour early unless the CEO is allowing ot and we want it. No pressure either direction.
If users are rude or generally hard to deal with, manager has our back in dealing with them.
Pay isn’t top dollar but there’s trade-offs
€ or good team, right?
I just accepted a job with a small MSP starting early next year. I kept a close ear out during the interview for signs of the classic MSP hell stuff that would chew through techs but it does look like I got a good one (small 8 or so man shop) but check in in about 3 months and we’ll see how I’m feeling haha
My longer term plan is to use this as a stepping stone to then move onto being in-house then figuring out my exit strategy before burnout takes me, which I’m thinking I’ll either be aiming to move into IT management or possibly moving into a business analytics or cloud administration type role. Technical sales probably wouldn’t be too bad either.
Nice! I’ve job hopped a few times and tripped my salary in 5 years and am at a unicorn msp with unlimited PTO and management that cares about employees.
I wish I could find a union IT shop, but nothing around that I’ve seen available. Happy to hear my first statement isn’t as universal as my experience suggests!
“Unlimited PTO” is a meaningless term, and a trap.
I have 42 days of PTO per year, plus 13 state holidays.
I have a right to take those days off, they can’t be denied by anyone.
And if I don’t take them, my team lead will have a talk with me in October at the latest, because the company would get in legal trouble if I didn’t get them.With “unlimited PTO” you have no such right to any amount of PTO.
Sure, you could try to schedule lots of PTO, but it can just be denied (“not possible right now”), or if you take too many, you’re just fired.Plus they don’t have to book the liability on the balance sheet!
You’ll let us know if they’re hiring, right? Right!?
That’s my job title.
What do you do now?
Idk dude, I got a redundancy about a year ago. There are still jobs out there but it feels like it’s dwindling.
What, do you think it’s all run by AI now?
No, just not many job postings for it. Go look on indeed with that exact title. Switch to remote, almost no jobs
So yes, they still exist.
I know this has nothing to do with my home computer, but this just further affirms my decision to switch to Linux earlier this year.
Copilot just forced itself onto my personal machines again so it’s just typical Windows fuckery all around.
When the OS becomes the virus
When reading comprehension is limited to the title.
MS mislabeled the update
Heimdal (apparently a patchmanagement) auto-installed the falsely labeled update.If OP (this was reported by a Redditor on r/sysadmin) and their company is unable to properly set grace periods for windows updates I can’t help them either.
IMHO you are supposed to manually review and release updates either on a WSUS or the management interface of your patching solution.
Not just “Hehe, auto install and see what happens”.
And if you do that shit, set a timeout for 14 days at least for uncritical rated updates.They said they believe it was a mislabeled update. MS didn’t respond. Before criticizing others for their reading comprehension, I think you could work on yourself too.
There is a world, and it may be ours, where MS purposefully pushes this out. As the end of the article makes clear, this will be only a minor issue for those with good backup (which they probably all should but they don’t), but for those who don’t they’ll be stuck with the new version and have to pay for the license of it. This is a large benefit to MS while they also get to pretend like it’s just a mistake and not having backups makes it your issue, not theirs.
Shouldnt you be able to just downgrade?
Dunno if that works on the server version.
I come to the comments for someone to summarise the article for me.
Reading (the TLDR) without complaining: Fine
Complaining while only reading the comments: Not fine
You thought you were in control?
Our server, comrade.
Crowdstrike moment
Of all the people MS doesn’t want to piss off.
deleted by creator
