rounding errors

• Subject: rounding errors
• From: "Dominic R. Scales" <Dominic.Scales(at)aerosensing.de>
• Date: Fri, 27 Apr 2001 08:51:24 +0000
• Newsgroups: comp.lang.idl-pvwave
• Organization: DLR
• Xref: news.doit.wisc.edu comp.lang.idl-pvwave:24709

```HELP!

What gives? Is there any numerical math guy/gal out there
who can tell me how this happens? It seems to me, that
the accuracy of the second/third cast ist WAY off.

a=double('2.56989')
b=double( 2.56989 )
c=double(float('2.56989'))

print,a,b,c,format='(d)'

2.5698900000000000 <---- this is what i want to have
2.5698900222778320
2.5698900222778320

So, the question is: why can the cast from a string give a much
more accurate result than a cast from a literal constant (or a
float variable, for that matter)? Do I really have to cast with:

d=double(2.56989*1000000L)/1000000.
print,d,format='(d)'

2.5698900000000000

or even
d=double(string(2.56989))
print,d,format='(d)'

2.5698900000000000

Ah yes, and while I'm at it... Have you ever compared
5.2e-6, 5.2*1e-6, 5.2*10*1e-7 ?

print, 5.2e-6, 5.2*1e-6, 5.2*10*1e-7, format='(e20.10)'
5.2000000323e-06
5.1999995776e-06
5.2000000323e-06

Cheers,
Dominic

--
Dipl. Phys. Dominic R. Scales | Aero-Sensing Radarsysteme GmbH
Tel:  +49 (0)8153-90 88 90    | c/o DLR Oberpfaffenhofen
Fax:  +49 (0)8153-908 700     | 82234 Wessling, Germany
WWW:  aerosensing.de          | email: Dominic.Scales@aerosensing.de
```