Controlling rounding

Posts   
 
    
wvnoort
User
Posts: 96
Joined: 06-Jan-2005
# Posted on: 18-Jul-2005 17:42:20   

Hi all,

At the moment I am working on a part of our system that requires a lot of calculations with different currency and percentages and i am feeling a little uncertain about the rounding that occurs.

In the database, currencies are represented as NUMBER with 6 digits after the decimal point, most other monetary values have two decimals. This translates to Decimal in the generated entities.

Has anyone encountered problems with rounding, especially in calculations like:


Decimal total = 0;
foreach(OrderEntity order in selectedOrders)
{
     total += Quantity * Price / CurrencyRate;
}

What is the best strategy to avoid loss of digits in intermediate calculations.

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39933
Joined: 17-Aug-2003
# Posted on: 18-Jul-2005 18:28:06   

the best you can do I think is use double as intermediate type, and then store the end value again as decimal. As you have a fixed number of fraction digits, rounding will occur, you can't avoid that.

Frans Bouma | Lead developer LLBLGen Pro
wvnoort
User
Posts: 96
Joined: 06-Jan-2005
# Posted on: 19-Jul-2005 09:51:27   

Otis wrote:

the best you can do I think is use double as intermediate type, and then store the end value again as decimal. As you have a fixed number of fraction digits, rounding will occur, you can't avoid that.

Using doubles for intermediate results is in my opinion not the best option. As amounts never exceed the 99 million, the decimal type is large enough to hold the intermediate results and it is more precise than the floating point types. See the remarks on the decimal type in the MSDN lib:

Contrary to the float and double data types, decimal fractional numbers such as 0.1 can be represented exactly in the decimal representation. In the float and double representations, such numbers are often infinite fractions, making those representations more prone to round-off errors.

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39933
Joined: 17-Aug-2003
# Posted on: 19-Jul-2005 10:32:34   

Ok, point taken smile

I was thinking of storing a fractional intermediate result of say 20 digits in a double, so you won't get intermediate rounding errors, but indeed, it might just do more harm in other situations...

Frans Bouma | Lead developer LLBLGen Pro