Probably not as bad as some of the other examples here, but the company I currently work for has its 10tb shared drives backing up to a server that’s right next to it in the same cabinet. Those two servers, plus all of the networking hardware and a variety of ancillary devices are all plugged in to one socket via a bunch of extension cords.
Yes, the boss has been told to get it sorted, but he’s the kind of older guy who doesn’t give a shit.
Happened to us, they put the backups on a different device, away from the servers, but still in the same premise. Cryptolocker locked everything on the network, including the backups. No off-site backups.
TLDR - Boomer problems
Yeah, exactly that.
So I back up my own shit on my own external drive, just in case.
My current company has a script that runs and deletes files that haven’t been modified for two years. It doesn’t take into account any other factors, just modification date. It doesn’t aks for confirmation and doesn’t even inform the end user about.
Put all your files in a single zip file. No compression. Since Windows handles zip files like folders, you can work like normal. And the zip file will always have a recent time stamp.
That sounds like a lawyers dream… “can’t provide it if it doesn’t exist” … now granted, if they got a subpoena they’d have to save it going forward, but before then, if their not bound by something that forces data retention, the less random data laying around the better.
You should write a script to touch all the files before their script runs.
Thought about it but I use modification date for sorting to have the stuff I’ve recently worked on on top. I instead keep the files where the script isn’t looking. The downside is they are not backed up so I might potentially lose them but if I don’t do that, then I’ll lose them for sure…
Have you…called attention to this at all?
You don’t actually have to set all the modification dates to now, you can pick any other timestamp you want. So to preserve the order of the files, you could just have the script sort the list of files by date, then update the modification date of the oldest file to some fixed time ago, the second-oldest to a bit later, and so on.
You could even exclude recently-edited files because the real modification dates are probably more relevant for those. For example, if you only process files older than 3 months, and update those starting from "6 months old"1, that just leaves remembering to run that script at least once a year or so. Just pick a date and put a recurring reminder in your calendar.
1: I picked 6 months there to leave some slack, in case you procrastinate your next run or it’s otherwise delayed because you’re out sick or on vacation or something.
Change the date on all the files by scaling to fit the oldest file. Scale to 1 year as a safe maximum age. So if the oldest file is 1.5 years old, scale all files to be t/1.5 duration prior to now.
Have a script that makes a copy of all files that are 1.9 years old into a separate folder.
Create a series of folders labeled with dates. Every day copy the useful stuff to the new folder. Every night change modified dates on all files to current date.
What industry are you in. This could be compliance for different reasons. Retention is a very specific thing that should be documented in policies.
I know financial institutions that specifically do not want data just hanging around. This limits liability and exposure if there is a breach, and makes any litigation much easier if the data doesn’t exist by policy.
Should they be more choosy on what gets deleted, yea probably. But I understand why it’s there.
That’s the worst foresight I think I’ve ever heard of, you might as well make that 3 months if you’re just going to trash thousands of labor hours on those files.
No IT at all.
This want their fault and I feel for them.
Working in telecommunications and we get a call from a customer that they are moving premises in 3 months time. They want us to check the place out of we can get a service in there. All goes well, time table for install goes out and we rock up the day after they get the keys, only to find the previous tenant took ALL the copper cables with them. Now when I say all, I mean ALL. Data, telephony, and power were stripped out all the way back to the dmarc point of the building.
They were fucking pissed. The lease for the existing place ran out about 3 weeks after the got the keys to the new place.
I have worked as a lead developer for a major print shop with about 100 employees. The entire order workflow for all branches was shoehorned into one order management system that was initially hacked together for one or two users. It was built on a then already ancient OpenERP system and it had a PHP and smarty frontend for the actual order management. All was hosted on one old debian box which was a VM on a Windows server.
At some point in time, MT decided to slap a web shop onto this system, which was part of the main code base. User data were saved into the same database with plain text passwords. That was convenient for the support people: if somebody forgot their password, you could call support and they would read you your password over the phone.
Another thing that made my hair raise in fear, was that for every single order, any working file was retained indefinitely, even in the light of the then-looming GDPR laws. This amounted of terabytes of data, much of it very private.
I worked at the main branch. When a person walked in, there was a desktop computer at the counter. No password protection, an order management screen open by default. People could just walk in and start viewing orders at will. I am not sure whether they did, but we did push MT to at least have manadatory password protection on their PCs.
I was hired to implement a CRM for an insurance company to replace their current system.
Of course no documentation or functional requirements where provided, so part of the task was to reverse engineer the current CRM.
After a couple of hours trying to find some type of backend code on the server, I discovered the bizarre truth: every bit of business logic was implemented in Stored Procedures and Triggers on a MSSQL database. There were no frontend code either on the server, users have some ActiveX controls installed locally that accessed the DB.
every bit of business logic was implemented in Stored Procedures and Triggers on a MSSQL database.
Provided the SP’s are managed in a CVS and pushed to the DB via migrations (similar to Entity Framework), this is simply laborious to the devs. Provided the business rules are simple to express in SQL, this can actually be more performant than doing it in code (although it rarely ever is that simple).
There were no frontend code either on the server, users have some ActiveX controls installed locally that accessed the DB.
This is the actual WTF for me.
There was no version control at all. The company that provided the software was really shady, and the implementation was so bad that the (only) developer was there full time fixing the code and data directly in production when the users had any issue (which was several times a day).
This was 5 years ago at a usd200mil multinational…
The email system was pop3. There were no document backups. There was no collaboration tools. There was no IT security. You could basically copy company data out and no one would ever find out. The MS Office license was bought singly. Ahem!
Source control relying on 2 folders: dev/test and production. Git was prohibited due to the possibility of seeing the history of who did what. Which made sense in a twisted way since a previous boss used to single out people who made mistakes and harras them
When you lift up the red flag and there are more red flags underneath.
In every small problem are many large problems that want to come out.
Just share a git user, come on. Have everyone check in under the same name “development” or whatever, but no version control whatsoever?
Current company (Remote Desktop inception): Linux host machine -> Remote Desktop to windows machine -> Remote Desktop to Linux machine
Bad frame rates, modifier keys hardly ever work, super annoying to code. Windows machine resets all settings and files (besides desktop and one specific folder) each day. Each day I have to install a language pack, change display options, keyboard layout etc.
The recent Falcon cock up?
I actually disagree. I only know a little of Crowdstrike internals but they’re a company that is trying to do the whole DevOps/agile bullshit the right way. Unfortunately they’ve undermined the practice for the rest of us working for dinosaurs trying to catch up.
Crowdstrike’s problem wasn’t a quality escape; that’ll always happen eventually. Their problem was with their rollout processes.
There shouldn’t have been a circumstance where the same code got delivered worldwide in the course of a day. If you were sane you’d canary it at first and exponentially increase rollout from thereon. Any initial error should have meant a halt in further deployments.
Canary isn’t the only way to solve it, by the way. Just an easy fix in this case.
Unfortunately what is likely to happen is that they’ll find the poor engineer that made the commit that led to this and fire them as a scapegoat, instead of inspecting the culture and processes that allowed it to happen and fixing those.
People fuck up and make mistakes. If you don’t expect that in your business you’re doing it wrong. This is not to say you shouldn’t trust people; if they work at your company you should assume they are competent and have good intent. The guard rails are there to prevent mistakes, not bad/incompetent actors. It just so happens they often catch the latter.
IBM
/thread
One of my friends quit IBM not too long ago. From the stories he’s told me, it sounded like almost everyone there spends all of their time and energy blamimg others for failed projects and unhappy clients.
Exactly this. I don’t know anyone in the IT industry that would willingly buy IBM. They’re either locked in due to legacy reasons or government projects where most of them are incompetent.
Thankfully it’s changing, but slowly.
Freight shipping company still running on a custom AS400 application for dispatch. Time is stored as a 4-digit number, which means the nightside dispachers have their own mini Y2K bug to deal with every midnight.
On one hand, hooray for computer-enforced fucking-off every night. On the other hand, the only people who could fix an entry stuck in the system because of this were on dayside.
Apparently, this actually isn’t uncommon in the industry, which I think is probably the worst part to me.
Hehe I was in global shipping IT, we had some ooooold Solaris systems that handled freight halting data flows. Windows Server 98 servers that handled data for very large shippers. Every daylight savings time change something would break.
A behavioral health company with 25 iPads deployed to field employees as patient data collection devices all signed into the same iCloud account instead of using MDM or anything.
They all had the same screen lock PINs and though most of the data was stored in a cloud based service protected by a login, that app’s password was saved by default.
I was a backend developer for a startup company where:
- Windows servers without any firewall and security hardening.
- Docker swarm without WSL. We had to use 4 GB Windows base images for 50MB web apps.
- MSSQL without any replication and backups.
- Redis installed on Windows via 3rd-party tool that looked like a 2010 era keygen generator.
- A malware exploited the Redis * what a surprise * and kept killing processes to mine crypto on CPU…
- VPS provider forgot to activate new Windows Server on production and it kept restart for every 30 minutes until I checked the logs and notified them about the missing license.
I left there after 6 months.
The blind determination to use a desktop OS to do a server’s job.
Saw a mid size clinic where the server was also the personal desktop of the boss - who also used the domain admin user as his main user account. His reasoning was that he needed to see “everything” his employees did and that none must come “above him” IT wise.
And before I forget it:The machine was in his office where he still was seeing patients and where often patients were left unattended - without him locking the machine.
