[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
rounding errors
- Subject: rounding errors
- From: "Dominic R. Scales" <Dominic.Scales(at)aerosensing.de>
- Date: Fri, 27 Apr 2001 08:51:24 +0000
- Newsgroups: comp.lang.idl-pvwave
- Organization: DLR
- Xref: news.doit.wisc.edu comp.lang.idl-pvwave:24709
HELP!
What gives? Is there any numerical math guy/gal out there
who can tell me how this happens? It seems to me, that
the accuracy of the second/third cast ist WAY off.
a=double('2.56989')
b=double( 2.56989 )
c=double(float('2.56989'))
print,a,b,c,format='(d)'
2.5698900000000000 <---- this is what i want to have
2.5698900222778320
2.5698900222778320
So, the question is: why can the cast from a string give a much
more accurate result than a cast from a literal constant (or a
float variable, for that matter)? Do I really have to cast with:
d=double(2.56989*1000000L)/1000000.
print,d,format='(d)'
2.5698900000000000
or even
d=double(string(2.56989))
print,d,format='(d)'
2.5698900000000000
Ah yes, and while I'm at it... Have you ever compared
5.2e-6, 5.2*1e-6, 5.2*10*1e-7 ?
print, 5.2e-6, 5.2*1e-6, 5.2*10*1e-7, format='(e20.10)'
5.2000000323e-06
5.1999995776e-06
5.2000000323e-06
Cheers,
Dominic
--
Dipl. Phys. Dominic R. Scales | Aero-Sensing Radarsysteme GmbH
Tel: +49 (0)8153-90 88 90 | c/o DLR Oberpfaffenhofen
Fax: +49 (0)8153-908 700 | 82234 Wessling, Germany
WWW: aerosensing.de | email: Dominic.Scales@aerosensing.de