[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

rounding errors



HELP!

  What gives? Is there any numerical math guy/gal out there 
  who can tell me how this happens? It seems to me, that 
  the accuracy of the second/third cast ist WAY off.

   a=double('2.56989')
   b=double( 2.56989 )  
   c=double(float('2.56989'))

   print,a,b,c,format='(d)'

       2.5698900000000000 <---- this is what i want to have
       2.5698900222778320
       2.5698900222778320

  So, the question is: why can the cast from a string give a much
  more accurate result than a cast from a literal constant (or a 
  float variable, for that matter)? Do I really have to cast with:

  d=double(2.56989*1000000L)/1000000.
  print,d,format='(d)'

       2.5698900000000000

  or even 
  d=double(string(2.56989))
  print,d,format='(d)'

       2.5698900000000000

  Ah yes, and while I'm at it... Have you ever compared
       5.2e-6, 5.2*1e-6, 5.2*10*1e-7 ?

  print, 5.2e-6, 5.2*1e-6, 5.2*10*1e-7, format='(e20.10)'
    5.2000000323e-06
    5.1999995776e-06
    5.2000000323e-06


  Cheers,
         Dominic

-- 
Dipl. Phys. Dominic R. Scales | Aero-Sensing Radarsysteme GmbH
Tel:  +49 (0)8153-90 88 90    | c/o DLR Oberpfaffenhofen
Fax:  +49 (0)8153-908 700     | 82234 Wessling, Germany
WWW:  aerosensing.de          | email: Dominic.Scales@aerosensing.de