The int type cast should be on the whole expression
passed to the mod operator. Otherwise it won't work
when/if c0/c1 is unsigned.
Making c0/c1 unsigned makes it possible to use 5802
as MODX value.
@@ -91,7 +91,7 @@ fletcher_checksum(u_char * buffer, const size_t len, const uint16_t offset)
/* The cast is important, to ensure the mod is taken as a signed value. */
- x = ((int)(len - offset - 1) * c0 - c1) % 255;
+ x = (int)((len - offset - 1) * c0 - c1) % 255;
if (x <= 0)
x += 255;