NetScaler under fire: CitrixBleed and a gateway DDoS
Being the only NetScaler admin when actively exploited vulnerabilities and a DDoS hit remote access, and how InfoSec and I worked through both.
- Where
- Rushmore Loan Management Services
- When
- 2017-2023
- My role
- Sole NetScaler administrator
The environment
I was the only NetScaler administrator at Rushmore, so the Citrix ADC appliances were mine to build, patch and fix (and to answer for when something broke). We ran a VPX high-availability pair in California and another in Texas, managed through Citrix ADM, both on-premises and in Citrix Cloud.
These appliances did a lot more than Citrix Gateway. They handled remote access for 1,200+ users, load balancing for a number of internal applications, and load-balanced LDAPS, which InfoSec required for directory authentication. If the NetScalers had a bad day, a lot of other things had a bad day with them.
Emergency patching in 2023
2023 was a rough year to own NetScalers. Citrix released fixes for vulnerabilities that were already being exploited, including CVE-2023-3519 and CitrixBleed (CVE-2023-4966). These weren’t patches that could wait for the next maintenance window. I applied the emergency firmware updates at both sites, and then worked with InfoSec going through the appliance logs, looking for any indicators of compromise.
We did find some log entries that looked suspicious. After digging in, we traced them back to our own ADM appliance’s management traffic, not an attacker (nothing like being spooked by your own monitoring). That might sound like a non-event, but it mattered. Being able to rule something out with evidence meant we could close the investigation instead of guessing.
A DDoS against remote access
Separately, Citrix Gateway came under a DDoS attack that started degrading sessions for our remote users. Working with InfoSec, I put region-based blocks in place at the firewalls for the regions the traffic was coming from. It wasn’t a permanent fix, but it kept remote access usable until Citrix shipped one.
What it taught me
Patching an internet-facing appliance isn’t the end of the job. You also need to know what normal looks like in its logs. Otherwise you can’t tell an attack from your own monitoring traffic, and that really muddies the waters when you’re in the middle of an incident.