Crashcart Start your free trial
Case study · Healthcare

"We had a guy."

A medical clinic followed the standard advice. Ransomware still cost it ten days and its patients' data.

Small medical practice, details changed 8 minute read
118,000
failed logins in 30 hours, with no one monitoring the alerts
~5years
since the last working backup
3days
before anyone told the clinic it was ransomware
~10days
until the clinic could work again

Monday morning

The front desk couldn't open the patient schedule. Then nobody could open anything. The clinic's IT contractor came in and told them the server had crashed.

It hadn't crashed. Next to the patient database was a ransom note. The clinic wouldn't hear about it until Thursday, when the FBI arrived.

The clinic is a small practice of about 15 people, including a physician, a nurse practitioner, nurses and office staff. It ran on one server in a closet that did nearly everything. It held the patient records, the scanned documents, every staff login and the fax system.

By Monday morning the attacker had been inside for more than two days.

On paper

"We were covered"

The clinic hadn't ignored security. It had done the four things a practice its size is usually told to do.

On paper

An IT person

An independent contractor had looked after the clinic's computers for years. The deal was a handshake. He would manage the systems, keep an eye on them and keep backups.

In reality

Things got fixed when they visibly broke. Nothing else was monitored.

On paper

Cyber insurance

If something went wrong, the policy was supposed to bring help.

In reality

The insurer didn't respond for days. Its eventual offer didn't fit the breach.

On paper

Backups

Backups were part of the IT deal. The server ran a backup job every day.

In reality

The job had failed every day for about five years.

On paper

Antivirus

Installed on every machine in the office.

In reality

It asked whether to let the encryptor run. The attacker said yes.

On paper, the clinic was covered four ways. No one in the front office could tell whether any of it worked.

In the logs

What the server logs showed

The handshake never defined "keeping an eye on things," and nothing was in writing. In practice, someone called when something visibly broke, and it got fixed. Nobody was looking at anything that hadn't broken yet.

The server's own logs show what that missed.

  • Remote desktop was open to the internet months

    Remote desktop, the tool IT people use to log in to a machine from somewhere else, was reachable from the whole internet. Its extra login check was turned off. So was the Windows firewall, on every network profile.

  • Password-guessing alerts went unanswered since July

    Automated attackers were guessing the administrator password around the clock, about 4,100 times an hour. In the 30 hours of log that survived, there were 118,000 failed logins. The server's built-in alert for password guessing fired 20 to 24 times a day, every day. Nobody was watching it.

  • The administrator stayed logged in since March

    The administrator account had been left logged in on the server for months. Whoever got in next would land on an open desktop with full control of the network.

  • Every backup was failing ~5 years

    The backup job failed every day with the same message: "no backup storage location found." As best anyone can tell, the last good backup was from around 2021. For roughly five years, the clinic had been relying on backups that didn't exist.

  • The server was past end of support 3 years

    Its version of Windows stopped getting security fixes in October 2023.

  • Old remote-access tools were still installed years

    Eight remote-access tools were installed on the server, most from vendors the clinic no longer used. One old scheduled task existed only to uninstall the antivirus.

  • There were no written plans ever

    HIPAA requires a practice that keeps patient records to do a risk analysis, keep a contingency plan for backups and disasters, and have a procedure for security incidents. No risk analysis had ever been done, and there was no recovery plan or incident plan to reach for.

None of this was hidden. Nobody was watching the server.

The attack

The weekend

  1. SATURDAY · 2:49 AM
    Someone logs in to the server over remote desktop as the administrator. They get the password right on the first try, and the open session from March is waiting for them.
  2. SATURDAY TO SUNDAY
    For a day and a half they work through the network. They unpack a toolkit, scan for other computers, move to at least four more machines, search for documents and open patient files.
  3. SUNDAY MORNING
    They launch the encryptor. The antivirus notices and asks whether to allow the program to run. The attacker is the one at the keyboard, so the attacker clicks "Allow."
  4. SUNDAY TO MONDAY
    Encryption runs for about 25 hours. A second virus spreads across the server's programs at the same time. Patient data is copied out of the clinic.
  5. MONDAY · EARLY MORNING
    The patient database is among the last things hit. The front desk can't open anything. The clinic is told the server crashed.
  6. THURSDAY
    The FBI arrives. It's the first time the clinic hears the word ransomware.

The response

The first days without a plan

The clinic needed every safeguard at once. None of them held up.

For three days, it was "a crash."

The clinic's only view into its own systems was what the IT contractor told them, and he described it as a server crash. Nobody said the word ransomware until the FBI showed up on Thursday. Those three days were when the clinic most needed to act: call the insurer, call a lawyer, keep every affected machine exactly as it was, and start the clock on a possible HIPAA breach. Instead they were waiting on a repair.

There was no backup to restore.

The job that was supposed to make one had been failing for about five years.

The insurance didn't answer.

The clinic called the insurer and heard nothing back for days. Staff didn't know what they were allowed to do in the meantime, or who else to call. When the insurer finally responded, it offered packaged services that didn't fit a breach that far along.

The machines were wiped to get people working again.

The IT contractor wiped and reinstalled the affected computers. The instinct is understandable. It was the wrong first move. Those machines held the evidence of what the attacker had touched and taken. Once they were wiped, nobody could say exactly which patient records had left the building. That makes the breach harder to scope and the notice to patients harder to write.

What page one of an incident plan says

Don't wipe anything yet. Call these people, in this order. Here is where the backups are. The clinic had none of it.

Recovery

Recovering the patient database

Crashcart's founder arrived that Thursday to assist with the response and work alongside the FBI.

From 118,000 log events he rebuilt the timeline: the first login, every machine the attacker reached, the moment the encryptor started, and the minute it reached the patient database. The attacker got in through the open remote desktop with the administrator password.

Then he went after the data. There was no backup, and the server's automatic snapshots, the last thing that might have held an older copy, had been deleted. What was left was the encrypted database itself. The encryptor had only reached the patient database at the very end, and it had scrambled the first part of each file before it stopped. The rest was still there.

He rebuilt the database by hand from the parts the encryptor hadn't touched. He kept checking it until the practice software could open it again. With a working backup, the same step is clicking "restore last night's copy."

It took about ten days from that Monday morning until the clinic could work again. With a working backup and a plan, the same recovery takes hours.

The damage

What it cost

  • About ten days of disrupted patient care, scheduling and billing.
  • Three days lost at the start, when the clinic believed it had a broken server, not a break-in.
  • Patient data in the hands of a criminal group, and a HIPAA breach the clinic now has to report and notify patients about.
  • Every computer the attacker touched, rebuilt from scratch, plus a new server.
  • A breach investigation that will ask for the risk analysis and policies the clinic never had.
  • Evidence that's gone for good, so some questions about what was taken will never have answers.

The clinic is open and working again. It did nothing reckless. It hired help, bought insurance, had backups on its list and ran antivirus, which is what small practices are told to do. No one checked that any of it worked, and the clinic had no way to check for itself.

What should have been there

Daily checks, reported to the owner

The clinic needed someone who knew its systems, checked them every day and told the owner the truth. Crashcart is built for that job.

Crashcart is run by an AI IT manager. When a practice signs up, it interviews the owner in plain language: what the business does, which computers hold patient records, who should be asked about what. Then it inspects every computer and builds its own record of the office.

It keeps a notebook on each practice with exceptions, owner decisions, old software kept on purpose and open problems. Its daily review of every change includes that notebook.

Every practice gets the same checks. The IT Manager interprets the results for each practice.

IT Manager notebook
Example clinic · reviewed this morning
  • Patient data

    SERVER-CLOSET holds the patient database. Backed up nightly, restore-tested on the 1st. Any problem here goes to the owner the same day.

  • Fixed

    Remote desktop on SERVER-CLOSET was open to the internet. Closed Tuesday, with the owner's OK.

  • Remote access

    Billing's remote-support tool is approved because the claims vendor uses it. Two more copies turned up on old PCs. Asked the owner about them.

  • Owner decision

    The lab analyzer PC stays on its old Windows because the analyzer's software needs it. Kept off the internet, watch only.

  • Waiting on owner

    The insurer's breach line, to finish the incident plan.

5 notesRead before every daily review

How Crashcart handles each failure

The clinic: Remote desktop open to the internet

Checked from outside, every day

Crashcart checks what the office exposes to the internet and flags remote desktop by name. It also checks every computer, every day, for remote desktop without its sign-in protection and for a firewall that's turned off. Each comes with a one-click fix.

The clinic: Unmonitored security alerts

Monitored around the clock

Every computer runs endpoint security, monitored around the clock by a security team. The attacker here spent about 30 hours inside before encrypting anything, logging in at 2:49 AM and moving between five machines. On a monitored network, that is 30 hours for the security team to notice and shut them out.

The clinic: Failed backups

Encrypted backups computers can't delete, tested weekly

Crashcart backs up every computer, every day. Each backup is encrypted before it leaves the office. A computer can add backups but never delete them, so an attacker can't delete or scramble them, even with the computer's administrator password. A restore test runs every week, a database restore test every month and a deeper test every quarter. Crashcart treats a failed test as urgent.

The clinic: The ransomware encryption

Decoy files and one-click isolation

Endpoint security plants decoy files on every computer. Ransomware that touches one sets off an alert to the security team. Crashcart also checks every backup for a sudden wave of changed files. If it sees one, the owner gets an email right away with an "Isolate now" button that cuts that computer off from the rest of the network.

The clinic: The server past end of support

Flagged before end of life

Crashcart names the system, gives the date security fixes stopped, and warns months before a version reaches end of life.

The clinic: Forgotten remote-access tools

Every remote-access tool, listed weekly

A weekly survey lists every remote-access tool on every computer and flags the ones that are running. The owner marks the tools in use. Crashcart asks about the rest.

The clinic: No written plans

Plans written with the practice

Crashcart works through the documents HIPAA expects with the practice: the risk analysis, the policies, a backup and recovery plan, an incident plan and the yearly review. It starts from what it already knows about the office, like which computers hold patient records and how they're backed up. Then it asks the owner what it doesn't know, like who the insurer is and which lawyer to call. A document can't be signed until those questions are answered. The incident plan opens with the first 15 minutes. Don't wipe or switch off the affected computer. Unplug it from the network. Don't pay anyone. Call the people on the list in order, starting with the insurer's breach line.

The clinic: Owner told three days late, secondhand

The owner sees it first

Every check is on the owner's dashboard, not hidden in an IT person's notes. A ransomware warning goes straight from Crashcart to the owner's inbox, not through anyone else first. Crashcart also emails other urgent problems, a summary every Monday and a report every month.

Crashcart doesn't replace a good IT person. It shows the owner whether the work is getting done.

For practice owners

Seven questions to ask your IT person this week

These questions need no technical knowledge. Insist on specific answers.

  1. When did we last restore a backup, and how long did it take?"We have backups" isn't an answer. A tested restore is.
  2. What can someone on the internet reach on our network?The answer should be a short list, and remote desktop shouldn't be on it.
  3. Who gets told when the security alerts go off, and what happened with the last one?
  4. How many failed login attempts did our server have in the last 24 hours?If nobody knows, nobody is looking. This clinic's server was getting about 4,100 an hour.
  5. Which of our computers can't get security updates anymore?
  6. Do we have a written HIPAA risk analysis and incident plan, and when were they last updated?
  7. If we got hit tomorrow, who do we call first, and is it written down somewhere I can find it?

Vague answers don't mean your IT person is doing a bad job. They mean nobody has written down what should happen. Attackers count on that gap.

The clinic did everything it was told. Nobody checked that any of it worked.

See what your logs show

Setup takes 10 minutes. Crashcart then checks every computer and reports what it finds in plain language.

Start your free trial