August 22nd, 2013


Is this evening over yet?

Well, that was an unexpectedly interesting evening.

I thought I was just going to the Davis Square outdoor contra dance. Which I did do.

Shortly after the dance ended, one of the dancers I know found her backpack missing while people were just hanging out talking. It had been there near the end of the dance, and then a few minutes later it wasn't. Several of us walked around looking for it and asking people if they'd seen it. I asked her if she had a phone in the backpack and she said she did, so she tried calling it - nobody answered. Then I asked if she had a "find my iPhone" or equivalent Android app on it, and she did, but didn't think she had a way of using the service from the middle of Davis Square. Fortunately I had both a little laptop, and a phone with tethering, so I got her online and she had it locate her phone...

... which was apparently right there in Davis Square, right where we were! Her friend who was more familiar with that app than us pointed out a button on the side of the page that let us cause the phone to make a noise, so I stayed and kept clicking it while they walked around. "Found it!" It was ringing in someone's pockets. [Edit to clarify: She was the one who yelled out "found it!" or something similar. I was sitting at the computer clicking, she and a friend of hers were walking around listening. When she heard the sound she asked the guy what it was, and he pulled her phone out of his pocket.]

Initially he denied having anything to do with taking a backpack and claimed he just found the phone, but some cops came over and asked him to show what was in his pockets, and when his pockets turned out to contain this girl's credit cards and IDs, his credibility was gone. Eventually the police got him to say that he left the backpack "behind Tedeschi's", and one of them found it in a dumpster. Along with what they got from his pockets and his bag, she ended up getting pretty much everything back, so when all that was done, we went to JP Licks for some ice cream.

That's when I noticed I'd missed a call on my own phone, from my boss. I texted him back, and he responded that we had an outage in production and even though I'm not on call and the on call people were dealing with it, this particular outage is in something that I'm an expect on, so if I wasn't too busy, could I help them out?

We were already about to split up and go home, so I said sure, got home 20 minutes later, and logged in. Indeed, help was needed. The outage was a problem we'd had variations of a few times in the past, and we had a documented workaround (which I had helped develop the first time this sort of thing happened), but this particular incident involved one piece that was different in an important way. The basic strategy of the workaround was still sound, but it had to be done in a different way, and figuring out how to modify it involved understanding details of this part of the system that I know off the top of my head but the other people had to delve in docs and config files for.

After a few false starts, we puzzled out a way to modify this workaround to make it work in this case, and the outage was patched!

And it was also almost midnight.