Quantcast
Channel: Common Language Runtime Internals and Architecture forum
Viewing all articles
Browse latest Browse all 1710

why does .net suck at simple math???

$
0
0

take the simple subtraction 43.709719 - 43.7097189

which should yield 0.0000001, however when i type in that problem into the watch bar(or have a pair of values with sufficient precision i get a result like this:

WTF?!?!?!

how am I supposed to build a program that will handle high precision values if the language compiler can't even handle subtraction!  both visual studio 2010 and 2012 show this inability to handle subtraction of precision numbers

while i know for this example i could use a decimal declaration to handle them since both have fewer than 8 decimal places, i have values that are accurate to 14 decimal places that i need to work with.

Edit:

also trying things of a lower precision... from Watch 1

        1.0 - 0.9    0.099999999999999978    Double


Viewing all articles
Browse latest Browse all 1710

Latest Images

Trending Articles



Latest Images