426
you are viewing a single comment's thread
view the rest of the comments
[-] ShortFuse@lemmy.world 3 points 1 year ago

I just recently worked on fixed point 8.8 and basically the way fractional values work, you take an integer and say that integer is then divided by another one. So you represent the number in question with two numbers not one. 0.3 can be presented in a number of ways, like 30 % 10, or 6 % 20.

The problem is the way 0.1 is represented and 0.2 represented don't jive when you add them, so the compiler makes a fractional representation of 0.3 based on how 0.1 and 0.2 were expressed that just comes out weird.

That's also why 0.3 + 0.3 is fine. When you wrote 0.3, the compiler/runtime already knew how to express 0.3 without rounding errors.

this post was submitted on 17 Sep 2023
426 points (96.5% liked)

Programmer Humor

19564 readers
1479 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 1 year ago
MODERATORS