Classic WTF: Crazy Like a Fox(Pro)

It's Labor Day in the US. We're busy partaking in traditional celebrations, which depending on who you ask, is either enjoying one of the last nice long weekends before winter, or throwing bricks at Pinkertons. So we dig back into the archives, for a classic story about databases. Original --Remy

Database portability" is one of the key things that modern data access frameworks try and ensure for your application. If you're using an RDBMS, the same data access layer can hopefully work across any RDBMS. Of course, since every RDBMS has its own slightly different idiom of SQL, and since you might depend on stored procedures, triggers, or views, you're often tied to a specific database vendor, and sometimes a version.

And really, for your enterprise applications, how often do you really change out your underlying database layer?

Well, for Eion Robb, it's a pretty common occurrence. Their software, even their SaaS offering of it, allows their customers a great deal of flexibility in choosing a database. As a result, their PHP-based data access layer tries to abstract out the ugly details, they restrict themselves to a subset of SQL, and have a lot of late nights fighting through the surprising bugs.

The databases they support are the big ones- Oracle, SQL Server, MySQL, and FoxPro. Oh, there are others that Eion's team supports, but it's FoxPro that's the big one. Visual FoxPro's last version was released in 2004, and the last service pack it received was in 2007. Not many vendors support FoxPro, and that's one of Eion's company's selling points to their customers.

The system worked, mostly. Until one day, when it absolutely didn't. Their hosted SaaS offering crashed hard. So hard that the webserver spinlocked and nothing got logged. Eion had another late night, trying to trace through and figure out: which customer was causing the crash, and what were they doing?

Many hours of debugging and crying later, Eion tracked down the problem to some code which tracked sales or exchanges of product- transactions which might not have a price when they occur.

$query .= odbc_iif("SUM(price) = 0", 0, "SUM(priceact)/SUM(" . odbc_iif("price != 0", 1, 0) . ")") . " AS price_avg ";

odbc_iif was one of their abstractions- an iif function, aka a ternary. In this case, if the SUM(price) isn't zero, then divide the SUM(priceact) by the number of non-zero prices in the price column. This ensures that there is at least one non-zero price entry. Then they can average out the actual price across all those non-zero price entries, ignoring all the free" exchanges.

This line wasn't failing all the time, which added to Eion's frustration. It failed when two very specific things were true. The first factor was the database- it only failed in FoxPro. The second factor was the data- it only failed when the first product in the resultset had no entries with a price greater than zero.

Why? Well, we have to think about where FoxPro comes from. FoxPro's design goal was to be a data-driven programming environment for non-programmers. Like a lot of those environments, it tries its best not to yell at you about types. In fact, if you're feeding data into a table, you don't even have to specify the type of the column- it will pick the correct" type by looking at the first row.

So, look at the iif again. If the SUM(price) = 0 we output 0 in our resultset. Guess what FoxPro decides the datatype must be? A single digit number. If the second row has an average price of, say, 9.99, that's not a single digit number, and FoxPro explodes and takes down everything else with it.

Eion needed to fix this in a way that didn't break their database agnostic" code, and thus would continue to work in FoxPro and all the other databases, with at least predictable errors (that don't crash the whole system). In the moment, suffering through the emergency, Eion changed the code to this:

$query .= "SUM(priceact)/SUM(" . odbc_iif("price != 0", 1, 0) . ")") . " AS price_avg ";

Without the zero check, any products which had no sales would trigger a divide-by-zero error. This was a catchable, trappable error, even in FoxPro. Eion made the change in production, got the system back up and their customers happy, and then actually put the change in source control with a very apologetic commit message.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.