LoginLogin

Very weird rounding error

Root / Programming Questions / [.]

12Me21Created:
12Me21#1
INPUT X
WHILE ROUND(X)!=X
  X=X*10
  PRINT X
WEND
This is supposed to multiply a number by 10 until it is an integer, but try it with 1.88. It goes on forever And even weirder, at one point X does equal 188, but it is not detected... I would expect this for big numbers, but not something as simple as 1.88. EDIT: if you replace X=X*10 with 
INC X,9*X
'or
X=9*X+X
then it works...
snail_#2
Probably just double-precision math errors internally. I'd suspect FLOOR() will do a better job, but as my 3DS is busted I can't do any testing.
12Me21#3
Probably just double-precision math errors internally. I'd suspect FLOOR() will do a better job, but as my 3DS is busted I can't do any testing.
I tried FLOOR and CEIL, and they didn't work. EDIT: I just tried doing 
WHILE INSTR(STR(X),".")+1
(which I thought I did before but I guess not) and it worked! I guess because it just looks for a decimal point rather than doing math. I'm sure it's much slower though.
SquareFingers#4
at one point X does equal 188
No, it doesn't actually. You are experiencing rounding errors (and not particularly weird ones), plus the fact that PRINT does not print the full precision of stored values. EDIT: And neither does STR, which is why your other approach behaves the way it does.
12Me21#5
I give up trying to make a calculator. The EXTREME limitations of both STR$(random scientific notation for no reason) and FORMAT$(I think the maximum length is like 9 digits or something) are just too much. There just isn't a reliable way to convert a number to a string... WAIT! I know... PRINT and CHKCHR! 
CLS
PRINT N
X=0
WHILE CHKCHR(X,0)
S$=S$+CHR$(CHKCHR(X,0))
INC X
WEND
Wow! It actually works! It can convert numbers between 0.00000001(because of PRINT rounding (probably added to hide rounding errors)) and 999999999999999(because rounding errors) for comparison: STR$: depends on whether it wants to or not FORMAT$: only integers less than 100000000 Does anyone know why STR$ gives scientific notations in some situations? It seems to only happen with really small numbers, and even then only sometimes.
SquareFingers#6
What exactly are you looking for in number-to-string conversion? No scientific notation ever? 15 digits max? Including negative sign? Including the decimal point? Always, or just when there's something nonzero to display on the other side?
12Me21#7
I want 
PRINT number
'and
PRINT conversion$(number)
to look the same, basically. Something that I can use in my number->2 digit number characters converter. I'm making a calculator with a 10 digit display that will be 5 chraracters long.
SquareFingers#8
There's no negative sign, and there's no decimal point, in the 'condensed digits'. Just sayin'.
12Me21#9
I know :( I'm using GPOINT for the decimal point, and the - sign is displayed at the left side of the screen.
snail_#10
At this point, FORMAT$() is all you need for flexible number-to-string conversion. You shouldn't even touch STR$(), like ever.