[Skywise]

[quote='Roger Hunter' pid='36' dateline='1388459292']
Since I'm programming in TrueBasic I don't have that "f" option so I did it this way;

fmag = 62
mag = 6.3
imag = mag*10
n = int(imag-fmag)

So n = 1

[quote]

Yep. How you implement a calculation can affect the outcome when precision is involved. This can be a particular issue in strongly typed languages such as XB or C/C++ where commands which mix data types have an implied type conversion.

You may not have explicit types in TB, but TB has to translate things into code the CPU can use, and the CPU *does* have explicit types.

Most of the time it's not a issue and small errors won't affect the outcome, or are acceptable. Obviously you found an exception.

But your solution technically isn't the same calculation. You changed one of the input values from 6.2 to 62 and rearranged the formula to compensate.

I was able to implement code that assigned the numbers to variables, and then used those variables in the calculation and got the expected result. But I was also playing around with data types to get it to work.

x# = 6.3
y# = 6.2
z = (x#-y#)*10
q = INT(z)

x# and y# are doubles (double precisions floating point). z is an integer (in this case, 32 bit signed). In the calculation there is an implicit type conversion of the result of the calculation into an integer because I'm assigning the answer to z which is an integer, which rounded off the floating point result.

Of course, if I do,

q = INT((x#-y#)*10)

I get the 'wrong' answer.

What I did in the first case with multiple lines of code would be equivalent to,

q = INT(SLONG((x#-y#)*10)))

SLONG converts to a signed long integer.

Not sure if you can easily see what's happening with so much nesting.

Brian