My favourite one of this kind is the Rockchip RK808 RTC, where the engineers thought that November had 31 days, needing a Linux kernel patch to this day that translates between Gregorian and Rockchip calendars (which are gradually diverging over time).
It's always November, isn't it? I've once made a log collection system that had a map of month names to months (had to create it because Go date package didn't support that specific abbreviation for month names).
As you might've guessed, it lacked November, but no one noticed for 4+ months, and I've left the company since. It created a local meme #nolognovember and even got to the public (it was in Russia: https://pikabu.ru/story/no_log_november_10441606)
That hardware real time clocks keep time in date and time drives me batty. And no one does the right thing which is just a 64 bit counter counting 32khz ticks. Then use canned tested code to convert that to butt scratching monkey time.
Story my old boss designed an STD Bus RTC card in 1978 or something. Kept time in YY:MM:DD HH:MM:SS 1/60 sec. And was battery backed. With shadow registers that latched the time. Couple of years later redesigned it as a 32 bit seconds counter. With a 32khz sub seconds counter. Plus a 48 bit offset register. What was a whole card was now a couple of 4000 series IC's on the processor card. He wrote 400 bytes of Z80 assembly to convert that to date and time. He said was tricky to get right but once done was done.
I would guess that’s because those chips were designed for use in systems that didn’t have a CPU, but reading the data sheet, it doesn’t look as if you can easily hook up this thing to 7-segment LEDs, so maybe this is a matter of “this is how we always did it, and if it ain’t broke, don’t fix it”, and then ‘fix’ it, anyways?
I had some interactions with the guy responsible for the code that made our system do the right thing around Daylight Saving. Listening to him talk out loud as he thought about bugs was fascinating. He was clearly one of the smartest people I've met and I would quickly fall behind as he rationalized problems to himself. What a marvelous mind.
The Renesas RTC divides the 32khz clock by 256. And after waking up doesn't update the shadow registers till the next tick. So if you wake out of deep sleep you don't know what the time is for 8ms.
I know of one that draws 0.5uA in normal mode but 12uA in binary counter mode.
> the characters ’n’ and ‘o’ differ by only one bit; an unpredictable error that sets that bit could change GenuineIntel to GenuineIotel.
On a QWERTY keyboard, the O key is also next to the I key. It's also possible someone accidentally fat-fingered "GenuineIontel" , noticed something was off, and moved their cursor between the "o" and "n", and accidentally hit Delete instead of Backspace.
Maybe an unlikely set of circumstances, but I imagine a random bit flip caused at the hardware-level is rare since it might cause other problems, if something more important was bit-flipped.
I like this theory - I can totally imagine some big spreadsheet of processor model names where someone copy/pastes the model name to some janky firmware-programming utility running on an off-the-shelf mini PC on the manufacturing floor, implemented as a "temporary fix" 5 years ago, every time the production line changes CPU model.
I am reminded of the old AMD CPUs with "unlockable" extra cores, which would when unlocked change the model name to something unusual.
"GenuineIotel" is definitely odd, but difficult to research more about; I suspect these CPUs might actually end up being collector's items sometime in the future.
because inserting no-op instructions after them prevents the issue.
Some of the 386 bugs described there sound to me like the classic kind of "multiple different subsystems interact in the wrong way" issue that can slip through the testing process and get into hardware, like this one:
> For example, there was one bug that manifested itself in incorrect instruction decoding if a conditional branch instruction had just the right sequence of taken/not-taken history, and the branch instruction was followed immediately by a selector load, and one of the first two instructions at the destination of the branch was itself a jump, call, or return.
Even if you write up a comprehensive test plan for the branch predictor, and for selector loads, and so on, it might easily not include that particular corner case. And pre silicon testing is expensive and slow, which also limits how much of it you can do.
80386 (1985) did not have a branch predictor, which was used first only in Intel Pentium (1993).
Nevertheless, the states of the internal pipelines, which were supposed to be stopped, flushed and restarted cleanly by taken branches, depended on whether the previous branches had been taken or not taken.
Ah, thanks for that correction -- I jumped straight from "depends on the history of conditional branches" to "branch predictor" without stopping to think that that would have been unlikely in the 386.
Before having branch predictors, most CPUs that used any kind of instruction pipelining behaved like a modern CPU where all the branches are predicted as not taken.
Thus on an 80386 or 80486 CPU not taken branches behaved like predicted branches on a modern CPU and taken branches behaved as mispredicted branches on a modern CPU.
The 80386 bug described above was probably caused by some kind of incomplete flushing of some pipeline after a taken branch, which leaved it in a state partially invalid, which could be exposed by a specific sequence of the following instructions.
This sort of bug, especially in and around pipelines are always hard to find. In chips I've built we've had one guy who built a system that would build random instruction streams to try and trigger as many as we possibly could
Yeah, I think random-instruction-sequence testing is a pretty good approach to try to find the problems you didn't think of up front. I wrote a very simple tool for this years ago to help flush out bugs in QEMU: https://gitlab.com/pm215/risu
Though the bugs we were looking to catch there were definitely not the multiple-interacting-subsystems type, and more just the "corner cases in input data values in floating point instructions" variety.
The revenge of the MIPS delay slot (the architecture simply didn't handle certain aspects of pipelining, so NOPs were required and documented as such).
The GenuineIotel thing fascinates me because I can't fully grasp how it could happen. I can imagine a physical defect causing a permanent wrong-bit in a specific piece of silicon, but it seems more widespread than that. Perhaps some kind of bug in the logic synthesis process?
I came across a CPU bug that prevented Linux from booting on 3rd gen i3/i5/i7 CPUs. Did a bunch of printf debugging until I was right before the freeze. Then found something relevant in the CPU errata. It could be "fixed" by passing in noapic. I had a decent writeup on the old CentOS forums, but they're gone, and I don't have a copy of my writeup anymore.
I had to deal with Intel Quark SoC X1000 on a Galileo board years ago, where the LOCK prefix instruction caused segfaults. Since the SoC is single threaded, the lock prefix could just be patched out from resulting binaries, before the compiler/build system was patched.
There's a difference in effort of several orders of magnitude between "change a setting so the compiler doesn't emit multiplies" and "convince GCC/LLVM to add a special-case flag for one very rare chip, or maintain your own fork". The vendor's workaround is the "ideal" solution, but disabling multiplies is a lot more practical if you don't need the performance.
They also mention in the next sentence that they adopted the "correct" workaround (by providing a multiplication library function for the compiler to call).
The company selling the chip can create a fork. They are typically the ones providing all of the sdks for you to use in order to use it, flash it, debug it, etc.
The issue is that it's no longer actually RISC-V M at the point, you're changing the instruction set. If you're compiling RISC-V M code, that doesn't need the extra NOP.
That being said, the disabling of MUL is being done at a software project level here, not by the CPU vendor. It's in the same linked commit that added in the NOP instructions to the arithmetic routines.
If your software runs on any chip and your chip runs any software, you have a problem, but in embedded cases, you know which chip runs which software, because you designed them together.
This is very true and why I'm not liking that Xilinx is trying to go the other way. It really gets in the way and doesn't work. I know what's connected to what and how, but their system device tree generator doesn't and it yells really loud about that. And I don't even need a device tree, just xparameters.h
Wasn't there also a Pentium division bug of some sort. I didn't pay much attention to microcomputers back then (being a mainframe programmer at the time), but I remember hearing about it from the mainstream news.
> To me, this issue doesn’t seem as embarrassing as Intel’s wrong CPUIDs. Pipelined CPUs are hard to build
I disagree. Misspelling a name in the CPUID is kind of easy to do, somewhat awkward to test (in a non-tautological way), and pretty easy to work around.
Having `mul ...; lw ...;` fail show that they've done very little testing of the chip. Any basic randomised pipeline testing would hit that trivial case.
Essentially all CPUs are pipelined today. In-order pipelined CPU execution semantics are not particularly hard to test. Even some open source testing systems could detect this bug, e.g. TestRig or RISCV-DV.
Also one of my favourite kernel patch messages: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
As you might've guessed, it lacked November, but no one noticed for 4+ months, and I've left the company since. It created a local meme #nolognovember and even got to the public (it was in Russia: https://pikabu.ru/story/no_log_november_10441606)
That hardware real time clocks keep time in date and time drives me batty. And no one does the right thing which is just a 64 bit counter counting 32khz ticks. Then use canned tested code to convert that to butt scratching monkey time.
Story my old boss designed an STD Bus RTC card in 1978 or something. Kept time in YY:MM:DD HH:MM:SS 1/60 sec. And was battery backed. With shadow registers that latched the time. Couple of years later redesigned it as a 32 bit seconds counter. With a 32khz sub seconds counter. Plus a 48 bit offset register. What was a whole card was now a couple of 4000 series IC's on the processor card. He wrote 400 bytes of Z80 assembly to convert that to date and time. He said was tricky to get right but once done was done.
I know of one that draws 0.5uA in normal mode but 12uA in binary counter mode.
>.< haha i remember this
On a QWERTY keyboard, the O key is also next to the I key. It's also possible someone accidentally fat-fingered "GenuineIontel" , noticed something was off, and moved their cursor between the "o" and "n", and accidentally hit Delete instead of Backspace.
Maybe an unlikely set of circumstances, but I imagine a random bit flip caused at the hardware-level is rare since it might cause other problems, if something more important was bit-flipped.
"GenuineIotel" is definitely odd, but difficult to research more about; I suspect these CPUs might actually end up being collector's items sometime in the future.
because inserting no-op instructions after them prevents the issue.
The early 386s were extremely buggy and needed the same workaround: https://devblogs.microsoft.com/oldnewthing/20110112-00/?p=11...
> For example, there was one bug that manifested itself in incorrect instruction decoding if a conditional branch instruction had just the right sequence of taken/not-taken history, and the branch instruction was followed immediately by a selector load, and one of the first two instructions at the destination of the branch was itself a jump, call, or return.
Even if you write up a comprehensive test plan for the branch predictor, and for selector loads, and so on, it might easily not include that particular corner case. And pre silicon testing is expensive and slow, which also limits how much of it you can do.
Nevertheless, the states of the internal pipelines, which were supposed to be stopped, flushed and restarted cleanly by taken branches, depended on whether the previous branches had been taken or not taken.
Thus on an 80386 or 80486 CPU not taken branches behaved like predicted branches on a modern CPU and taken branches behaved as mispredicted branches on a modern CPU.
The 80386 bug described above was probably caused by some kind of incomplete flushing of some pipeline after a taken branch, which leaved it in a state partially invalid, which could be exposed by a specific sequence of the following instructions.
Though the bugs we were looking to catch there were definitely not the multiple-interacting-subsystems type, and more just the "corner cases in input data values in floating point instructions" variety.
https://en.wikipedia.org/wiki/Intel_Quark#Segfault_bug
That is not the workaround in the documentation that was just linked.
This seems reasonable for your compiler vendor to implement without getting rid of multiplication altogether.They also mention in the next sentence that they adopted the "correct" workaround (by providing a multiplication library function for the compiler to call).
That being said, the disabling of MUL is being done at a software project level here, not by the CPU vendor. It's in the same linked commit that added in the NOP instructions to the arithmetic routines.
I disagree. Misspelling a name in the CPUID is kind of easy to do, somewhat awkward to test (in a non-tautological way), and pretty easy to work around.
Having `mul ...; lw ...;` fail show that they've done very little testing of the chip. Any basic randomised pipeline testing would hit that trivial case.
Essentially all CPUs are pipelined today. In-order pipelined CPU execution semantics are not particularly hard to test. Even some open source testing systems could detect this bug, e.g. TestRig or RISCV-DV.
Microcode errata re-writes to GPR, compiling low level "mul," and "output," CPU RISC V to system archictecture.