Print

[razgueado]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

Sounds like whatever rain was holding up the issue finally let up. Sounds like a rough week, though.

Boy, howdy.  Probably took a year or two off my life.  I even smoked most of a pack of cigarettes.  By midnight Saturday night I was pretty convinced I was losing my mind.

[A Friend of Charlie]

The Tony Over/Under for today is 10.

I knew 12.

Well look at you!

12 here to, using the criteria I could describe them from memory.  I get 13 if you count that I've heard the guy from Collective Soul on the radio. 

This is a competition, right?

I didn't count Collective Soul dude.

[razgueado]

The Tony Over/Under for today is 10.

I knew 12.

Well look at you!

12 here to, using the criteria I could describe them from memory.  I get 13 if you count that I've heard the guy from Collective Soul on the radio. 

This is a competition, right?

I didn't count Collective Soul dude.

Well, I didn't either, because I have no clue what the guy looks like.  But I've heard him on the radio.

[bluecollar]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

Sounds like whatever rain was holding up the issue finally let up. Sounds like a rough week, though.

Boy, howdy.  Probably took a year or two off my life.  I even smoked most of a pack of cigarettes.  By midnight Saturday night I was pretty convinced I was losing my mind.

I hear ya brother. I'm on a 17 hour shift. There is more work than we can handle. I am down 5 guys from before Covid. Management replaced no one. It's now time to take action against my employer.

[A Friend of Charlie]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

That sounds like a very familiar story. When in doubt, blame the storage team.

[A Friend of Charlie]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

Sounds like whatever rain was holding up the issue finally let up. Sounds like a rough week, though.

Boy, howdy.  Probably took a year or two off my life.  I even smoked most of a pack of cigarettes.  By midnight Saturday night I was pretty convinced I was losing my mind.

I hear ya brother. I'm on a 17 hour shift. There is more work than we can handle. I am down 5 guys from before Covid. Management replaced no one. It's now time to take action against my employer.

The more you bend over backwards for them, the more they'll shit on you.

I got a text regarding a work matter yesterday. I told them that I determined it's not worth my vacation time and it could wait until I return. Haven't heard back since.

Edit: I doubt anything will come of it but I'm more than prepared to look for a new job if it comes to that.

[razgueado]

It appears Travellin' Dave's history review omitted a significant anniversary. 

On this day in 1977, Radio Shack unveiled the TRS-80 microcomputer.  I was living in Irving, Texas at the time, about to start 7th grade. 

The TRS-80, and a DECWriter VT-120 in the math lab at Sam Houston Junior High, were the parents of my career in Information Technology.  My five-year detour into ministry and social work, in the end, were a minor detour.

The grandparents of my career were Star Trek and the programmable accounting machines my dad sold for Olivetti in San Francisco in the very early 1970's.  I was born at the ass end of Silicon Valley.  There must have been something in the water. 

I still adore electronic things with buttons that manipulate electromagnetic energy.

[A Friend of Charlie]

It appears Travellin' Dave's history review omitted a significant anniversary. 

On this day in 1977, Radio Shack unveiled the TRS-80 microcomputer.  I was living in Irving, Texas at the time, about to start 7th grade. 

The TRS-80, and a DECWriter VT-120 in the math lab at Sam Houston Junior High, were the parents of my career in Information Technology.  My five-year detour into ministry and social work, in the end, were a minor detour.

The grandparents of my career were Star Trek and the programmable accounting machines my dad sold for Olivetti in San Francisco in the very early 1970's.  I was born at the ass end of Silicon Valley.  There must have been something in the water. 

I still adore electronic things with buttons that manipulate electromagnetic energy.

I was four years behind you with the IBM PCjr.

[razgueado]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

Sounds like whatever rain was holding up the issue finally let up. Sounds like a rough week, though.

Boy, howdy.  Probably took a year or two off my life.  I even smoked most of a pack of cigarettes.  By midnight Saturday night I was pretty convinced I was losing my mind.

I hear ya brother. I'm on a 17 hour shift. There is more work than we can handle. I am down 5 guys from before Covid. Management replaced no one. It's now time to take action against my employer.

Ouch.  You win.  I'm not down any members of my team, and it's exceedingly rare for me to work more than 15 hours.  Good luck to you, Rick.

[razgueado]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

That sounds like a very familiar story. When in doubt, blame the storage team.

Right?  And my new boss tells me in our first meeting, "I know that happens, and I've done it, but it won't happen anymore."

A week later he's pointing at me. 

I called him on it yesterday.  He apologized.

[razgueado]

It appears Travellin' Dave's history review omitted a significant anniversary. 

On this day in 1977, Radio Shack unveiled the TRS-80 microcomputer.  I was living in Irving, Texas at the time, about to start 7th grade. 

The TRS-80, and a DECWriter VT-120 in the math lab at Sam Houston Junior High, were the parents of my career in Information Technology.  My five-year detour into ministry and social work, in the end, were a minor detour.

The grandparents of my career were Star Trek and the programmable accounting machines my dad sold for Olivetti in San Francisco in the very early 1970's.  I was born at the ass end of Silicon Valley.  There must have been something in the water. 

I still adore electronic things with buttons that manipulate electromagnetic energy.

I was four years behind you with the IBM PCjr.

The PCjr was the first computer I ever bought.  I bought it from a friend in summer of 1988, when he got himself an IBM 50z.  That was the beginning of my pivot back to IT. 

That computer is still in a closet at my MIL's house in Tucson.  I should retrieve it someday. 

[LuvTooGolf]

Hazzuh!

[A Friend of Charlie]

It appears Travellin' Dave's history review omitted a significant anniversary. 

On this day in 1977, Radio Shack unveiled the TRS-80 microcomputer.  I was living in Irving, Texas at the time, about to start 7th grade. 

The TRS-80, and a DECWriter VT-120 in the math lab at Sam Houston Junior High, were the parents of my career in Information Technology.  My five-year detour into ministry and social work, in the end, were a minor detour.

The grandparents of my career were Star Trek and the programmable accounting machines my dad sold for Olivetti in San Francisco in the very early 1970's.  I was born at the ass end of Silicon Valley.  There must have been something in the water. 

I still adore electronic things with buttons that manipulate electromagnetic energy.

I was four years behind you with the IBM PCjr.

The PCjr was the first computer I ever bought.  I bought it from a friend in summer of 1988, when he got himself an IBM 50z.  That was the beginning of my pivot back to IT. 

That computer is still in a closet at my MIL's house in Tucson.  I should retrieve it someday.

I sold mine for $400 bucks almost 10 later.

I still have my Commodore Amiga in a box in the garage. It's probably making a nice home for squirrels.

[A Friend of Charlie]

Hazzuh!

Later, Dave.

[Travellin Dave]

So here's the story of the last week.  Last Tuesday at 9am people at work went apeshit.  The Washington Electronic Lab Reporting System suddenly was discovered to be processing data at half speed.  Shit.  I pull up the server dashboards.  Everything looks normal.  One of my guys notices that our twice-weekly index rebuild job on that server has been failing for weeks, and before I can stop him he shares that information.  Now all the ape-shit is flying and fingers are pointing in my direction.  Good times.  And my guy is manually rebuilding indexes all over the place.

"Justin," I tell my guy.  "The job didn't really fail. I monitor index fragmentation, and I know that all of the WELRS indexes are under 15% fragmented.  The job reports an error, but it's on a test database that is the last thing it rebuilds."

"Shit," says Justin. 

So now it's all our fault, and, among all the other tasks we're assigned, Justin and I have to now "fix" this, even though it's not our problem.  Except Justin isn't on the IMT, so he logs off on Friday night at 6pm, and he goes on leave.  So I fight with this bad boy all weekend, and every metric I dig through in SQL Server says the same thing - the server has been performing at the same level for months.  There is no explanation for the slowdown in WELRS processing.  The application wasn't changed, the database wasn't changed, the server wasn't changed.  There's been significant data growth, but it was designed to handle that.  I know because my late friend Scott built it.  So, yesterday morning, we scream for - I mean, file a support ticket with - Microsoft.  And we're waiting for them to answer when, lo and behold, about noon, one of the application guys for WELRS says, "Uh...I'm not sure what's happening here, but processing has returned to normal.  It returned to normal at 9am. Did you guys do something?"

"I didn't do shit, and neither did anyone on my team, I can guarantee it!" says I.  "Did YOU do something?"

Nobody did anything.  And by end of day, not only had processing returned to "normal," it had returned to the performance level it had when the data was half the size it is now.  No explanation.  No change in the code.  No change in the SQL Server.  No change in the database.  At 9am last Tuesday the system started processing data at half speed.  At 9am yesterday it returned to normal.  Better than normal.

And this is what fucked up my weekend and the vacation I'm supposed to be on.

So you single handedly improved the whole system.  Sounds like a promotion to me.  Bonus at least!