Ticket #234 (closed defect: fixed)
Code: fixed_xte_fact.m_nVal==1 if FIXED_MATH
| Reported by: | max | Owned by: | jwharington@… |
|---|---|---|---|
| Priority: | normal | Milestone: | XCSoar 6.0.x |
| Component: | Other | Version: | |
| Keywords: | Cc: |
Description
The resolution of the "fixed" type is too small to represent the constant "fixed_xte_fact" (Earth.cpp), which is calculated by the formula:
1.0 / (fixed_rad_to_deg * fixed_earth_r)
or:
1.0 / 365031411
The number 365031411 needs 28 bits to be stored, which happens to be exactly the accuracy of the "fixed" data type. The division algorithm seems to round this up, resulting in a very inaccurate result "m_nVal==1". Let's compare:
1.0 / 365031411 = 0.73537632080654014 * (2-28)
our result = 1.0 * (2-28)
This is an error of more than 35%, and is not even remotely usable for real calculations.
My new division algorithm (which is more than 3x faster) always rounds down, and its result is "0".







Is there a reason why it is not just #defined?!