I once had a CD Pipeline which used powershell to execute some CLI. The CLI had a parameter for a password. The password was passed in via variable. I didn’t new the password had a ‘ which was interpreted by powershell…so the password passed into the CLI was wrong.
I was looking into a syntax error. It seemed to be taking issue with a line that looked fine to my eye. Copying and pasting the line resulted in the error, but manually typing it didn't. I started pasting fragments of the line about to find out what it took issue with and it was just a function name. Something like getResults. While I sat there scratching my head, I mindlessly tapped the left and right arrows on my keyboard over the characters and noticed that I'd have to press twice to get between two of the characters. This was my introduction to the zero-width space.
Similar bug: In PHP, our configuration wasn't working correctly. 100% of the code was fine... but one line had a space at the beginning. For some reason PHP treated the line as a comment and ignored it!
Errors are no problem. Silent errors, not so much.
I always have invisibles shown in my editor now after running into to many issues with single spaces in the wrong place, or copying code that mixes spaces and tabs, when the language doesn’t allow it.
Not my bug, but many years ago, I was trying to automate the Windows screen grabbing program SnagIt via VBS and the COM port, and this one command just wouldn't work, no matter what I tried. Out of frustration, I loaded the executable into a binary editor and discovered they'd literally spelled the command wrong (!), and of course never tested it. It may have been an "i before e, except after c" thing.
At a major datacenter in toronto I had some 10gbit internet links coming in. In my original design they went direct into a cisco firewall. They were "redundant" even though they literally sat next to each other in the cable management.
At some point after my design and implementation, junior sysadmin on my team added an inline, fail-open, security device in front of my firewall. It was a requirement from cyber security insurance. insurance claimed that it would always fail open.
Junior admin never told me, never updated documentation, never setup anything in network monitoring to indicate anything about that device. But we also didnt have any access to it. The only thing I knew was that it used the public ip of my firewall to transparently tunnel back to the insurance for management.
Months later i get a ticket about a 4 second outage. Literally nothing in my cisco logs; but i can confirm 4 second outage in network monitor. so i go to the datacenter operators who handle the transit and such. They come back and they dont see anything at all; even their network stats show nothing. Presumably not polling as often as we are.
It's 4 seconds, it didnt get much attention; until it became an intermittent issue about a week later. Not every day, not some specific time of day. Sometimes it happened in the middle of the night and virtually nobody noticed except our network monitor. Thin clients usually but not always reconnected within seconds. At most it was a like 10 second window when the organization, minimum hundreds of people stop working and then start working.
Cisco TAC gets on it and I'm getting a proper CCIE and we are doing caps wherein traffic leaves our firewall properly. It's just never coming back. The datacenter op on the otherhand had the same experience with their dell 10gbit switches. traffic was leaving our links. Link status never dropped on either side.
TAC is like, go check and see if a repeater between is failing. We have onsite hands do the check, but the DC had no repeaters. it was a solid fiber line in the same room into our rack. Onsite hands saw our tap in our rack but figured that was the firewall and failed to mention the device; just said there was a solid fiber line into the rack. The DC then closed the case and TAC didnt think it was on us. The firewall clearly sends the packets out the live link.
At this point we'd had maybe 60 seconds of outage; ~10 events.
A completely unrelated to me outage now happens with minutes of downtime. The DC is at fault now; but the last messaging to the CIO was that the DC wasnt at fault for the previous outages. Not to mention the DC hasnt said squat about any outage, which they always send out maintenance emails even if its unrelated. So I get about 4 hour meeting in the hot seat to address the problems with my network design. That if the problem isnt possible according to experts from DC and TAC... it's on me.
Top severity case with TAC next and the engineer is new as my current engineer isnt online. They find the outage was the DC; and I even get an email SUPER LATE(at least 12 hours late) saying they were done fixing the new outage.
Too late now... so TAC ends up being completely unhelpful. I make the 4 hour drive from Detroit to find a unmarked black box in front of my firewall. I had no idea what it was, I took it out of the loop causing a brief outage. Then i took it out of the rack and opened up to see if inside the box had anything. i took pictures thinking industrial espionage or something.
Obviously our problem was solved after this.
I later got in trouble for voiding the warranty on that rented device from the insurance and I get explained what the device was etc. All I could do was laugh.
> I make the 4 hour drive from Detroit to find a unmarked black box in front of my firewall. I had no idea what it was, I took it out of the loop causing a brief outage. Then i took it out of the rack and opened up to see if inside the box had anything. i took pictures thinking industrial espionage or something.
This is the most interesting part of the whole story, and to not include the details of what was actually causing the outage specifically makes reading this feel less informative than it might be otherwise.
I won't say pics or it didn't happen, but I do want to see those pictures all the same, or at least hear about them accurately described!
I once had a CD Pipeline which used powershell to execute some CLI. The CLI had a parameter for a password. The password was passed in via variable. I didn’t new the password had a ‘ which was interpreted by powershell…so the password passed into the CLI was wrong.
I was looking into a syntax error. It seemed to be taking issue with a line that looked fine to my eye. Copying and pasting the line resulted in the error, but manually typing it didn't. I started pasting fragments of the line about to find out what it took issue with and it was just a function name. Something like getResults. While I sat there scratching my head, I mindlessly tapped the left and right arrows on my keyboard over the characters and noticed that I'd have to press twice to get between two of the characters. This was my introduction to the zero-width space.
Similar bug: In PHP, our configuration wasn't working correctly. 100% of the code was fine... but one line had a space at the beginning. For some reason PHP treated the line as a comment and ignored it!
Errors are no problem. Silent errors, not so much.
I always have invisibles shown in my editor now after running into to many issues with single spaces in the wrong place, or copying code that mixes spaces and tabs, when the language doesn’t allow it.
cheers :D
Not my bug, but many years ago, I was trying to automate the Windows screen grabbing program SnagIt via VBS and the COM port, and this one command just wouldn't work, no matter what I tried. Out of frustration, I loaded the executable into a binary editor and discovered they'd literally spelled the command wrong (!), and of course never tested it. It may have been an "i before e, except after c" thing.
https://stackoverflow.com/questions/28891751/google-maps-bug...
Scotch tape on a motherboard near lower PCI slot traces causing Windows to BSOD on boot, but both memtest86/+ and Linux would test and boot fine.
At a major datacenter in toronto I had some 10gbit internet links coming in. In my original design they went direct into a cisco firewall. They were "redundant" even though they literally sat next to each other in the cable management.
At some point after my design and implementation, junior sysadmin on my team added an inline, fail-open, security device in front of my firewall. It was a requirement from cyber security insurance. insurance claimed that it would always fail open.
Junior admin never told me, never updated documentation, never setup anything in network monitoring to indicate anything about that device. But we also didnt have any access to it. The only thing I knew was that it used the public ip of my firewall to transparently tunnel back to the insurance for management.
Months later i get a ticket about a 4 second outage. Literally nothing in my cisco logs; but i can confirm 4 second outage in network monitor. so i go to the datacenter operators who handle the transit and such. They come back and they dont see anything at all; even their network stats show nothing. Presumably not polling as often as we are.
It's 4 seconds, it didnt get much attention; until it became an intermittent issue about a week later. Not every day, not some specific time of day. Sometimes it happened in the middle of the night and virtually nobody noticed except our network monitor. Thin clients usually but not always reconnected within seconds. At most it was a like 10 second window when the organization, minimum hundreds of people stop working and then start working.
Cisco TAC gets on it and I'm getting a proper CCIE and we are doing caps wherein traffic leaves our firewall properly. It's just never coming back. The datacenter op on the otherhand had the same experience with their dell 10gbit switches. traffic was leaving our links. Link status never dropped on either side.
TAC is like, go check and see if a repeater between is failing. We have onsite hands do the check, but the DC had no repeaters. it was a solid fiber line in the same room into our rack. Onsite hands saw our tap in our rack but figured that was the firewall and failed to mention the device; just said there was a solid fiber line into the rack. The DC then closed the case and TAC didnt think it was on us. The firewall clearly sends the packets out the live link.
At this point we'd had maybe 60 seconds of outage; ~10 events.
A completely unrelated to me outage now happens with minutes of downtime. The DC is at fault now; but the last messaging to the CIO was that the DC wasnt at fault for the previous outages. Not to mention the DC hasnt said squat about any outage, which they always send out maintenance emails even if its unrelated. So I get about 4 hour meeting in the hot seat to address the problems with my network design. That if the problem isnt possible according to experts from DC and TAC... it's on me.
Top severity case with TAC next and the engineer is new as my current engineer isnt online. They find the outage was the DC; and I even get an email SUPER LATE(at least 12 hours late) saying they were done fixing the new outage.
Too late now... so TAC ends up being completely unhelpful. I make the 4 hour drive from Detroit to find a unmarked black box in front of my firewall. I had no idea what it was, I took it out of the loop causing a brief outage. Then i took it out of the rack and opened up to see if inside the box had anything. i took pictures thinking industrial espionage or something.
Obviously our problem was solved after this.
I later got in trouble for voiding the warranty on that rented device from the insurance and I get explained what the device was etc. All I could do was laugh.
> I make the 4 hour drive from Detroit to find a unmarked black box in front of my firewall. I had no idea what it was, I took it out of the loop causing a brief outage. Then i took it out of the rack and opened up to see if inside the box had anything. i took pictures thinking industrial espionage or something.
This is the most interesting part of the whole story, and to not include the details of what was actually causing the outage specifically makes reading this feel less informative than it might be otherwise.
I won't say pics or it didn't happen, but I do want to see those pictures all the same, or at least hear about them accurately described!