Sometimes we all need to deal with the floating point weights. A normalized values between 0.0 and 1.0.
Every time I deal with it I think about normalizing it to percents 0%-100% instead of 0.0-1.0. Why?
Because human beings, including myself, always think linear. I think: 0-100 has a better precision than 0-1. Our floating point numbers are discrete. There are more possible values between 0 and 100 than between 0 and 1. So our calculations are 100 times more precise with the percents. Right?
Wrong. Absolutely wrong. The IEEE-754 floating point format does not put numbers uniformly. This is not an even distribution. The distance between two floating point values depends on the module. For example, if we use a 32-bit float, the closest number to 100.0 is (100.0 + 2 ^ (-17)) . The closest number to 1.0 is (1.0 + 2 ^ (-23)).
The density around 1.0 is 64 times higher than the density around 100. The distance between closest floating numbers grows exponentially.
Even though there is more numbers between 0 and 100, the precision for a reasonable numbers like 50,60 etc is better when they are normalized to 1.
The moral is - don't be afraid to normalize. Normalization never cause loosing precision. More than that, if you are doing extensive calculations with large floating point numbers, it makes sense to normalize them, make calculations, and de-normalize back at the end. There is a big chance that the accumulated error is lower in this case.
