[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Roundoff error in SMOOTH
email@example.com (Wayne Landsman) writes:
>I was recently surprised to discover that applying the SMOOTH function to a
>non-negative array could yield an array with negative numbers. I give an
>example below. This problem is evidently due to some sort of roundoff error,
>since it does not occur when using double precision. But it is not obvious to
>me how averaging 9 non-negative numbers (for a 3x3 box smooth) could yield a
>negative number, even allowing for roundoff error.
>Although not obvious from my simple example, this has nothing to do with edge
>effects -- I originally found the problem when 3x3 smoothing a 1024 x 1024
My theory is that the SMOOTH function probably uses Fourier transforms to speed
up the convolution, and this is where the round-off error is sneaking in.