Comparing numbers

[Andy McVeigh]

We also have had that problem mainly with numbers that are a result of a calculation 2 weeks rent at $450 per week would end up as $899.999999 if you made it a D6 number etc so we always use rnd when doing the calculation and also when comparing two values

Andrew McVeigh
Surfway Real Solutions
Phone 02 44412679 Mobile 0418428016
www.surfway.com.au
www.berrarabeach.com.au


 <http://www.surfway.com.au/>
 <http://www.surfway.com.au/>
> On 18 Feb 2025, at 12:35 pm, Doug Easterbrook via omnisdev-en <omnisdev-en at lists.omnis-dev.com> wrote:
> 
> hi Xavier.
> 
> I wouldn’t call it a bug in $total.     But it is the oldest numeric precision issue that exists in digital computers and it will catch you.
> 
> Omnis is only reporting what the computer returns (and the number can be different on mac, windows and 32 vx 64 bit machines).
> 
> 
> 
> It takes me back to when I used to write in Fortran (on a huge IBM bigger than my bedroom with only 8k of memory :) ) as a high school student
> 
> In those old days we had to be very cognizant that storage of floating point numbers in base 2 was never guaranteed to have perfect mathematical precision..  Thats because we had 16 bit numbers and 6 digits of precision.    However, it doesn't matter if you were using 8 bit, 16 bit, 32 bit, or 64 bit machines — the problem still exists, only the significant digits gets slightly larger.
> 
> 
> 
> there are a a whole host of articles online about imprecision in real numbers.    wikipedia, you name it .. its well studied
> 
> https://www.sciencedirect.com/topics/computer-science/precision-number
> 
> 
> The problem becomes more acute if lvVencimientos.importe is a different number of decimal places that ivFFACTURA.FR_TOT.  and for that matter, the problem can occur even if they are the same precision.
> 
> 
> 
> you’ll see the problem if you  
> 
> define var1 as floating decimal places and then do.
> 
> 
> Calculate var1 as 0.44444444+0.0000000055555555
> 
> 
> you won’t get what you think because the arithmetic units in computers can only deal with so many significant digits.
> 
> 
> you can make show it up worse by making the significant digits spread out further.
> 
> Calculate var1 as 44444444444444.00+0.0000000055555555
> 
> The number of significant digits that are handled in 64 bit numbers is 16 (I think).    if you add a large number and a small number, the small number becomes insignificant and wont show up in the total.
> 
> 
> so, if your list has a variety of numbers before and after the decimal point, the largest number will determine how many of the smaller numbers will not seem to be counted.
> 
> even then, rnd() may not help, so you need to code for it.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Doug Easterbrook
> doug at artsman.com
> Phone (403) 650-1978
> 
>> On Feb 17, 2025, at 7:15 AM, IT <it at plastipol.com> wrote:
>> 
>> Hi,
>> 
>> I have put an rnd() for list $total and now works properly.
>> 
>> rnd(lvVencimientos.$cols.importe.$total();2)<>ivFFACTURA.FR_TOT
>> 
>> This worries me that I can have some holes in my code regarding comparations. I can understand that different defined numbers could fail when comparing but I’m comparing same defined numbers and it mustn't fail.
>> Seem more a bug of $total than other thing.
>> 
>> Thank you all for your assistance.
>> 
>> xavier
>> 
>> 
>>> El 17 feb 2025, a las 15:01, Nigel Hughes <nigel at rnamh.co.uk> escribió:
>>> 
>>> Xavier
>>> 
>>> I seem to remember having to use the ‘rnd’ command to force a $total to be 2dp as behind the scenes it wasn’t rounding to 2dp
>>> 
>>> May help !
>>> 
>>> Nigel
>>> 
>>> 
>>> 
>>>> On 17 Feb 2025, at 12:40, IT <it at plastipol.com> wrote:
>>>> 
>>>> Hi all
>>>> 
>>>> I have an invoicing process that calculate due dates and imports depending on customer payment terms.
>>>> 
>>>> 
>>>> Do lvoVencimientos.$calcular_vencimientos(some parameters,,,) Returns lvVencimientos
>>>> If lvVencimientos.$cols.importe.$total()<>ivFFACTURA.FR_TOT
>>>> 	OK message  {Error [ivFFACTURA.FR_NUMERO]//Total due date import:[lvVencimientos.$cols.importe.$total()]//Invoice total:[ivFFACTURA.FR_TOT]}
>>>> 	Quit all methods
>>>> End If
>>>> 
>>>> 
>>>> This 'If' sometimes fail.
>>>> 
>>>> For example lvVencimientos.$cols.importe.$total() is 9087,71 and ivFFACTURA.FR_TOT is 9087,71 but Omnis shows the error message.
>>>> Both, ‘importe' and ‘FR_TOT' are defined same, kNumber, k2dpShortnum.
>>>> 
>>>> 
>>>> 
>>>> Is there any known bug that Omnis fail comparing numbers in certain circumstances?
>>>> 
>>>> regards
>>>> 
>>>> xavier
>>>> 
>>>> _____________________________________________________________
>>>> Manage your list subscriptions at https://lists.omnis-dev.com
>>>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>>> 
>>> _____________________________________________________________
>>> Manage your list subscriptions at https://lists.omnis-dev.com
>>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>> 
>> _____________________________________________________________
>> Manage your list subscriptions at https://lists.omnis-dev.com
>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
> 
> _____________________________________________________________
> Manage your list subscriptions at https://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com