Odd problem with calculation result

[reganc]

We had an odd problem a while ago and I could not work out why it was occurring.

We have a field whose value allows an extra column to appear on a picklist. Originally this field held a single 3 digit integer, this being a numeric stock location code.

After a while, a 2nd stock location was needed to be shown. So it was decided, for simplicity, that we would change the numeric field to have 3 decimal places and the 2nd location would be held as the 'thousandth' value, ie .003 would return the code for a 2nd stock location of 3.

The below was therefore used to extract this 2nd location:

Code: Select all

   int(100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR)))

But we found that this simply did not work. It would often return a meaningless value, eg a 2 when the value should have been 3.

Changing the code that gets the 2nd stock branch to

Code: Select all

   val(ntrim(100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR))))

worked.

But can anyone tell me why it did not work in the first place?

[Piotr D]

Hi Regan,
this is typicall bahavior. This is difference of representation between integer and real value. For example, value 2 (integer) is not the same as 2.0 (real). With real value, it can be f.e. 1.9999999999999999.... and in machine code is translate to 2.0. In thic case, you must use the same precision. It is possible, that 2=2*(1.0) return false. You must use round(2,0)=round(2*(1.0),0).
In your situation, int(100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR))) may return not 3, but 2, in case that 100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR)) is less than 3 (f.e. 2.99999999999999999...). But when you change this code like:
int(round(100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR))),0) it will be work correctly.

Piotr

[reganc]

Hi Regan,
this is typicall bahavior. This is difference of representation between integer and real value. For example, value 2 (integer) is not the same as 2.0 (real). With real value, it can be f.e. 1.9999999999999999.... and in machine code is translate to 2.0. In thic case, you must use the same precision. It is possible, that 2=2*(1.0) return false. You must use round(2,0)=round(2*(1.0),0).
In your situation, int(100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR))) may return not 3, but 2, in case that 100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR)) is less than 3 (f.e. 2.99999999999999999...). But when you change this code like:
int(round(100*(MY_CTRL->BROWSE_BR-int(MY_CTRL->BROWSE_BR))),0) it will be work correctly.

Piotr

Thanks, Piotr. That does make sense. I had seen that there were some instances where calculations in Xbase++ can return the wrong answer. I just did not know when it might occur.

It is a pity a simple calculation like this can cause so much trouble... I shall have to be more careful in future.