Comparing numbers

[Doug Easterbrook]

hi Xavier.

I wouldn’t call it a bug in $total.     But it is the oldest numeric precision issue that exists in digital computers and it will catch you.

Omnis is only reporting what the computer returns (and the number can be different on mac, windows and 32 vx 64 bit machines).



It takes me back to when I used to write in Fortran (on a huge IBM bigger than my bedroom with only 8k of memory :) ) as a high school student

In those old days we had to be very cognizant that storage of floating point numbers in base 2 was never guaranteed to have perfect mathematical precision..  Thats because we had 16 bit numbers and 6 digits of precision.    However, it doesn't matter if you were using 8 bit, 16 bit, 32 bit, or 64 bit machines — the problem still exists, only the significant digits gets slightly larger.



there are a a whole host of articles online about imprecision in real numbers.    wikipedia, you name it .. its well studied

https://www.sciencedirect.com/topics/computer-science/precision-number


The problem becomes more acute if lvVencimientos.importe is a different number of decimal places that ivFFACTURA.FR_TOT.  and for that matter, the problem can occur even if they are the same precision.



you’ll see the problem if you  

define var1 as floating decimal places and then do.


Calculate var1 as 0.44444444+0.0000000055555555


you won’t get what you think because the arithmetic units in computers can only deal with so many significant digits.


you can make show it up worse by making the significant digits spread out further.

Calculate var1 as 44444444444444.00+0.0000000055555555

The number of significant digits that are handled in 64 bit numbers is 16 (I think).    if you add a large number and a small number, the small number becomes insignificant and wont show up in the total.


so, if your list has a variety of numbers before and after the decimal point, the largest number will determine how many of the smaller numbers will not seem to be counted.

even then, rnd() may not help, so you need to code for it.









Doug Easterbrook
doug at artsman.com
Phone (403) 650-1978

> On Feb 17, 2025, at 7:15 AM, IT <it at plastipol.com> wrote:
> 
> Hi,
> 
> I have put an rnd() for list $total and now works properly.
> 
> rnd(lvVencimientos.$cols.importe.$total();2)<>ivFFACTURA.FR_TOT
> 
> This worries me that I can have some holes in my code regarding comparations. I can understand that different defined numbers could fail when comparing but I’m comparing same defined numbers and it mustn't fail.
> Seem more a bug of $total than other thing.
> 
> Thank you all for your assistance.
> 
> xavier
> 
> 
>> El 17 feb 2025, a las 15:01, Nigel Hughes <nigel at rnamh.co.uk> escribió:
>> 
>> Xavier
>> 
>> I seem to remember having to use the ‘rnd’ command to force a $total to be 2dp as behind the scenes it wasn’t rounding to 2dp
>> 
>> May help !
>> 
>> Nigel
>> 
>> 
>> 
>>> On 17 Feb 2025, at 12:40, IT <it at plastipol.com> wrote:
>>> 
>>> Hi all
>>> 
>>> I have an invoicing process that calculate due dates and imports depending on customer payment terms.
>>> 
>>> 
>>> Do lvoVencimientos.$calcular_vencimientos(some parameters,,,) Returns lvVencimientos
>>> If lvVencimientos.$cols.importe.$total()<>ivFFACTURA.FR_TOT
>>> 	OK message  {Error [ivFFACTURA.FR_NUMERO]//Total due date import:[lvVencimientos.$cols.importe.$total()]//Invoice total:[ivFFACTURA.FR_TOT]}
>>> 	Quit all methods
>>> End If
>>> 
>>> 
>>> This 'If' sometimes fail.
>>> 
>>> For example lvVencimientos.$cols.importe.$total() is 9087,71 and ivFFACTURA.FR_TOT is 9087,71 but Omnis shows the error message.
>>> Both, ‘importe' and ‘FR_TOT' are defined same, kNumber, k2dpShortnum.
>>> 
>>> 
>>> 
>>> Is there any known bug that Omnis fail comparing numbers in certain circumstances?
>>> 
>>> regards
>>> 
>>> xavier
>>> 
>>> _____________________________________________________________
>>> Manage your list subscriptions at https://lists.omnis-dev.com
>>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>> 
>> _____________________________________________________________
>> Manage your list subscriptions at https://lists.omnis-dev.com
>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
> 
> _____________________________________________________________
> Manage your list subscriptions at https://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com