So here's the story of the last week. Last Tuesday at 9am people at work went apeshit. The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed. Shit. I pull up the server dashboards. Everything looks normal. One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information. Now all the ape-shit is flying and fingers are pointing in my direction. Good times. And my guy is manually rebuilding indexes all over the place.
"Justin," I tell my guy. "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented. The job reports an error, but it's on a test database that is the last thing it rebuilds."
"Shit," says Justin.
So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem. Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave. So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months. There is no explanation for the slowdown in WELRS processing. The application wasn't changed, the database wasn't changed, the server wasn't changed. There's been significant data growth, but it was designed to handle that. I know because my late friend Scott built it. So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft. And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal. It returned to normal at 9am. Did you guys do something?"
"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I. "Did YOU do something?"
Nobody did anything. And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now. No explanation. No change in the code. No change in the SQL Server. No change in the database. At 9am last Tuesday the system started processing data at half speed. At 9am yesterday it returned to normal. Better than normal.
And this is what fucked up my weekend and the vacation I'm supposed to be on.
