[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

*Subject*: Re: Roundoff error in SMOOTH*From*: thompson(at)orpheus.nascom.nasa.gov (William Thompson)*Date*: 26 May 1999 23:12:25 GMT*Newsgroups*: comp.lang.idl-pvwave*Organization*: NASA Goddard Space Flight Center -- Greenbelt, Maryland USA*References*: <26MAY199915495025@stars.gsfc.nasa.gov>*Xref*: news.doit.wisc.edu comp.lang.idl-pvwave:14925

landsman@stars.gsfc.nasa.gov (Wayne Landsman) writes: >I was recently surprised to discover that applying the SMOOTH function to a >non-negative array could yield an array with negative numbers. I give an >example below. This problem is evidently due to some sort of roundoff error, >since it does not occur when using double precision. But it is not obvious to >me how averaging 9 non-negative numbers (for a 3x3 box smooth) could yield a >negative number, even allowing for roundoff error. >Although not obvious from my simple example, this has nothing to do with edge >effects -- I originally found the problem when 3x3 smoothing a 1024 x 1024 >array. (rest deleted) My theory is that the SMOOTH function probably uses Fourier transforms to speed up the convolution, and this is where the round-off error is sneaking in. Bill Thompson

**References**:**Roundoff error in SMOOTH***From:*Wayne Landsman

- Prev by Date:
**Re: Roundoff error in SMOOTH** - Next by Date:
**Re: Temporary variables still checked out** - Prev by thread:
**Re: Roundoff error in SMOOTH** - Next by thread:
**Re: Roundoff error in SMOOTH** - Index(es):