Article 7ED3 Long division, part two

Long division, part two

by
ericlippert
from Fabulous adventures in coding on (#7ED3)

This is a sequel to my 2009 post about division of long integers.

I am occasionally asked why this code produces a bizarre error message:

Console.WriteLine(Math.Round(i / 6000000000, 5));

Where i is an integer.

The error is:

The call is ambiguous between the following methods: 'System.Math.Round(double, int)' and 'System.Math.Round(decimal, int)'

Um, what the heck?

What is going on here is: division of an int by a long converts the int to a long and produces a long as its result. That's the fundamental problem here: the user wants a fraction rounded to five decimal places, but what they are getting is not even a fraction; integer arithmetic rounds automatically. So this code is just wrong.

Unfortunately the compiler is allowing the bogus division to pass without warning, and then informing the developer that there is no overload of Round that takes a long. There are applicable overloads that take double and decimal but there is no basis upon which to deduce which one the developer wants, so that's an error.

The real danger here is that the developer will attempt to make the fix by attacking the stated error, rather than attacking the actual conceptual error. I have seen this "fix":

Console.WriteLine(Math.Round((double)(i / 6000000000), 5));

Again, that does the integer arithmetic rounding first, and then converts the result to double, which will already be rounded off to a whole number, not to five decimal places.

The right way to do this is to not use integers:

Console.WriteLine(Math.Round(i / 6000000000.0, 5)); // doubleConsole.WriteLine(Math.Round(i / 6000000000.0m, 5)); // decimal

We see this sort of error so often that we've actually made a checker for it in the Coverity code quality analyzer; this turns up more frequently than you might think in real-world code.


2626 b.gif?host=ericlippert.com&blog=67759120
External Content
Source RSS or Atom Feed
Feed Location http://ericlippert.com/feed
Feed Title Fabulous adventures in coding
Feed Link https://ericlippert.com/
Reply 0 comments