Dev.SN ♥ developers
Sections
Dev.SN
martyb writes:
"Remember that one bug that had you tearing your hair out and banging your head against the wall for the longest time? And how it felt when you finally solved it? Here's a chance to share your greatest frustration and triumph with the community.
One that I vividly recall occurred back in the early 90's at a startup that was developing custom PBX hardware and software. There was the current development prototype rack and another rack for us in Quality Assurance (QA). Our shipping deadline for a major client was fast approaching, and the pressure level was high as development released the latest hardware and software for us to test. We soon discovered that our system would not boot up successfully. We were getting all kinds of errors; different errors each time. Development's machine booted just fine, *every* time. We swapped out our hard disks, the power supply, the main processing board, the communications boards, and finally the entire backplane in which all of these were housed. The days passed and the system still failed to boot up successfully and gave us different errors on each reboot.
What could it be? We were all stymied and frustrated as the deadline loomed before us. It was then that I noticed the power strips on each rack into which all the frames and power supplies were plugged. The power strip on the dev server was 12-gauge (i.e. could handle 20 amps) but the one on the QA rack was only 14-gauge (15 amps). The power draw caused by spinning up the drives was just enough to leave the system board under-powered for bootup.
We swapped in a new $10 power strip and it worked perfectly. And we made the deadline, too!
So, fellow Soylents, what have you got? Share your favorite tale of woe and success and finally bask in the glory you deserve."
(Score: 5, Interesting) by clone141166 on Friday March 07 2014, @11:10PM
Some of the worst bugs I have had to chase down have been in OpenGL GLSL shader programs. They don't have any kind of workable debugger available for them (there is one on windows that requires a subscription to NVidia's developer program, and only one or two that run on linux, neither of which I have been able to get to run reliably). There is no easy way to generate print statements - your only options are to output to a buffer and then read the buffer values on the CPU-side or write out colour values to pixels and then look at the colour of the pixels.
On top of this the shaders have a parallel execution structure where a vertex shader executes many times but at a 1:N ratio to the pixel shader. The most difficult to track down bugs I have had in GLSL programs have pertained to floating point values changing slightly between the vertex and pixel shader programs. Depending upon how the shader code is written, this in turn can cause the logic of the program to execute differently between the vertex and pixel shader.
Moral of the story being never, ever, ever trust the accuracy of floating point values!
(Score: 0, Flamebait) by paddym on Friday March 07 2014, @11:22PM
That's very interesting. How did you know there were bugs there in the first place? Do you see artifacts at a high level?
(Score: 2) by clone141166 on Friday March 07 2014, @11:37PM
Yes, essentially the pixel shading of the final geometry wasn't matching up with the geometry that was being produced (there was actually a geometry shader in the mix too iirc). It actually looked like multiple triangles were being rendered at the same position - in essence it looked exactly like z-fighting between two polygons.
I spent ages thinking the geometry shader was somehow producing two polygons when it should have been only producing one. But in actuality it was just because the pixel shader was shading differently for each pixel based on a floating point value that was fluctuating above/below a particular value. The fault was entirely mine, but it took quite a long time to figure out exactly why.
(Score: 4, Interesting) by istartedi on Saturday March 08 2014, @12:40AM
Moral of the story being never, ever, ever trust the accuracy of floating point values!
This reminds me of when I used to play with VRML back in the 90s. I had a torus with a seam that wouldn't go away. Fix? Assert that the last point is *exactly* equal to the first point. If you don't, floating point can generate a difference between sin(0) and sin(2*pi). That tiny difference was enough to make the engine not close the shape properly.