[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

*Subject*: Re: rounding errors*From*: "Dominic R. Scales" <Dominic.Scales(at)aerosensing.de>*Date*: Fri, 27 Apr 2001 11:38:39 +0000*Newsgroups*: comp.lang.idl-pvwave*Organization*: DLR*References*: <3AE9330C.29D059E9@aerosensing.de> <3AE94F58.45E4BA83@pet.mpin-koeln.mpg.de>*Xref*: news.doit.wisc.edu comp.lang.idl-pvwave:24711

Alex Schuster wrote: > You just have to define the 2.56989 as a double: b = 2.56989d :) > > Remember that the floating point value 2.56989 is not the same as the > real number 2.56989000000000000..., but just some value around > 2.5698900. Converting it to a double introduces more digits to the > right, but they are pretty random. > > Alex > -- > Alex Schuster Wonko@weird.cologne.de PGP Key available > alex@pet.mpin-koeln.mpg.de Danke Alex :) the prints were more of an example. In the 'real' world, the data is read in from a file with readu and is an array of float variables. I want to perform the following maths in double but with the cast to double I already introduce 'pretty random digits to the right', as you say. I'ld really like to avoid calling something like double(string(a)) for some large array... Any ideas? Schönes langes Wochenende, Dominic [ Tuesday is a holiday, so I'll take Monday off :) ] -- Dipl. Phys. Dominic R. Scales | Aero-Sensing Radarsysteme GmbH Tel: +49 (0)8153-90 88 90 | c/o DLR Oberpfaffenhofen Fax: +49 (0)8153-908 700 | 82234 Wessling, Germany WWW: aerosensing.de | email: Dominic.Scales@aerosensing.de

**Follow-Ups**:**Re: rounding errors***From:*Randall Skelton

**Re: rounding errors***From:*Craig Markwardt

**References**:**rounding errors***From:*Dominic R. Scales

**Re: rounding errors***From:*Alex Schuster

- Prev by Date:
**Re: rounding errors** - Next by Date:
**Re: rounding errors** - Prev by thread:
**Re: rounding errors** - Next by thread:
**Re: rounding errors** - Index(es):