John D. Cook

This post is an expansion of something I wrote on Twitter: Data scientists often complain that the bulk of their work is data cleaning. But if you see data cleaning as the work, not just an obstacle to the work, it can be interesting. You could think of it as data pathology, a kind of […]The post Data pathology first appeared on John D. Cook.

It’s been eight years since I started my consulting business. Two of the things I love about having my own business are the stability and the reduced stress. This may sound like a joke, but I’m completely serious. Having a business is ostensibly less stable and more stressful than having a salaried job, but at […]The post More stability, less stress first appeared on John D. Cook.

If y is a quaternion, how many solutions does x² = y have? That is, does every quaternion have a square root, and if so, how many square roots does it have? A quaternion can have more than two roots. There is a example right in the definition of quaternions: i² = j² = k² […]The post Quaternion square roots first appeared on John D. Cook.

Sometimes you don’t have all the math functions available that you would like. For example, maybe you have a way to calculate natural logs but you would like to calculate a log base 10. The Unix utility bc is a prime example of this. It only includes six common math functions: sine cosine arctangent natural […]The post Bootstrapping a minimal math library first appeared on John D. Cook.

The smaller the base you write numbers in, the smaller their digits sums will be on average. This need not be true of any given number, only for numbers on average. For example, let n = 2021. In base 10, the sum of the digits is 5, but in base 2 the sum of the […]The post Average sum of digits first appeared on John D. Cook.

A few days ago I wrote about the expected number of roots in a random polynomial where each coefficient is drawn from a standard normal, i.e. a Gaussian distribution with mean 0 and variance 1. Another class of random polynomials, one that comes up in applications to physics, draws each coefficient from a different distribution. […]The post Random polynomials revisited first appeared on John D. Cook.

If you want to add ∞ to the real numbers, should you add one infinity or two? The answer depends on context. This post gives several examples each of when its appropriate to add one or two infinities. Two infinities: relativistic addition A couple days ago I wrote about relativistic addition, where the sum of […]The post One infinity or two? first appeared on John D. Cook.

The radio spectrum is conventionally [1] divided into frequency bands that seem arbitrary at first glance. For example, VHF runs from 30 to 300 MHz. All the frequency band boundaries are 3 times a power of 10. Why all the 3’s? Two reasons: 3 is roughly the square root of 10, and the speed of […]The post Radio Frequency Bands first appeared on John D. Cook.

These have been the most popular posts here this year. Pretending OOP never happened Worst tool for the job Banned math book Divisibility by 13 Memento ComplexitatisThe post Most popular posts of 2020 first appeared on John D. Cook.

Let c be a positive constant and define a new addition operation on numbers in the interval (-c, c) by This addition has several interesting properties. If x and y are small relative to c, then x ⊕ y is approximately x + y. But the closer x or y get to c the more […]The post Relativistic addition first appeared on John D. Cook.

William J. Milne [1] attributes the following theorem to Galileo: The area of a circle is a mean proportional between the areas of any two similar polygons, one of which is circumscribed about the circle and the other is isoparametric with the circle. So imagine a polygon P and let S be an inscribed circle, that […]The post Galileo’s polygon theorem first appeared on John D. Cook.

The previous post looked at finding the expected number of real zeros of high degree polynomials. If you wanted to see how many real roots a particular high degree polynomial has, you run into several difficulties. If you use something like Descartes’ rule of signs, you’re likely to greatly over-estimate the number of roots. It […]The post Visualizing real roots of a high degree polynomial first appeared on John D. Cook.

Suppose you create a 100th degree polynomial by picking coefficients at random from a standard normal. How many real roots would you expect? There are 100 complex roots by the fundamental theorem of algebra, but how many would you expect to be real? A lot fewer than I would have expected. There’s a theorem due […]The post Expected number of roots first appeared on John D. Cook.

A matrix is said to have a saddlepoint if an element is the smallest element in its row and the largest element in its column. For example, 0.546 is a saddlepoint in the matrix below because it is the smallest element in the third row and the largest element in the first column. In [1], […]The post Large matrices rarely have saddlepoints first appeared on John D. Cook.

David Shelupsky [1] suggested a generalization of sine and cosine based on solutions to the system of differential equations with initial conditions αs(0) = 0 and βs(0) = 1. If s = 2, then α(t) = sin(t) and β(t) = cos(t). The differential equations above reduce to the familiar fact that the derivative of sine […]The post A generalization of sine and cosine first appeared on John D. Cook.

You can define a topology on the positive integers by choosing as an open basis sets the series of the form an + b where a and b are relatively prime positive integers. Solomon Golumb defines this topology in [1] and proves that it is connected. But that paper doesn’t prove that proposed basis really […]The post A connected topology for the integers first appeared on John D. Cook.

You can sometimes make code more readable by using names for characters rather than the characters themselves or their code points. Python and Perl both, for example, let you refer to a character by using its standard Unicode name inside \N{}. For instance, \N{SNOWMAN} refers to Unicode character U+2603, shown at the top of the […]The post Accessing characters by name first appeared on John D. Cook.

The numbers that make up today’s date—12, 16, and 20—form a Pythagorean triple. That is, 12² + 16² = 20². There won’t be any such dates next year. You could confirm this by brute force since there are only 365 days in 2021, but I hope to do something a little more interesting here. The […]The post Pythagorean dates first appeared on John D. Cook.

The goal this post is to show how to call Perl and Python one-liners from a shell. We want to be able to use bash on Linux, and cmd or PowerShell on Windows, ideally with the same code in all three shells. Bash Bash interprets text inside double quotes but not in single quotes. So […]The post Shells, quoting, and one-liners first appeared on John D. Cook.

Pick a positive integer k and take the product of k consecutive integers greater than k. Then the result is divisible by a prime number greater than k. This theorem was first proved 128 years ago [1]. For example, suppose we pick k = 5. If we take the product 20*21*22*23*24 then it’s obviously divisible […]The post Factors of consecutive products first appeared on John D. Cook.

Logic vs C I recently made a mistake because I interpreted the symbol ^ in the wrong context. I saw an expression of the form a ^ b and thought of the ^ as AND, as in logic, but the author’s intent was XOR, as in C. How else might someone misinterpret the ^ operator? […]The post Four meanings of ^ first appeared on John D. Cook.

I first heard the hymn “I Vow to Thee, My Country” while watching the first season of The Crown [1]. I assume the hymn is familiar in the UK, but it is not in America as far as I know. When I say I first heard the hymn, I mean that I first heard it […]The post \The Crown and The Planets\ first appeared on \John D\. Cook\.

Suppose you want to estimate the number of patients who respond to some treatment in a large population of size N and what you have is data on a small sample of size n. The usual way of going about this calculates the proportion of responses in the small sample, then pretends that this is […]The post Predictive probability for large populations first appeared on John D. Cook.

Applications of statistics often require working with for large x and fixed α and β. The simplest approximation to this ratio of gamma functions is You can do a little better [1] with where c = (α + β – 1)/2. To get more accuracy we’ll need more terms. Tricomi and Erdélyi worked out the […]The post Asymptotic series for gamma function ratios first appeared on John D. Cook.

You can uniquely represent a large number by its remainders when divided by smaller numbers, provided the smaller numbers have no factors in common, and carry out arithmetic in this representation. Such a representation is called a Residue Number System (RNS). In the 1960’s people realized RNSs could be useful in computer arithmetic. The original […]The post Redundant Residue Number Systems first appeared on John D. Cook.

The largest known primes are Mersenne primes, in part because the Lucas-Lehmer algorithm makes it more efficient to test Mersenne numbers for primality than to test arbitrary numbers. These numbers have the form 2n – 1. The Great Internet Mersenne Prime Search (GIMPS) announced today that they have tested whether 2n – 1 is prime […]The post Ruling out gaps in the list of Mersenne primes first appeared on John D. Cook.

Formal verification of theorems takes a lot of work. And it takes more work where it is least needed. But the good news is that it takes less effort in contexts where it is needed most. Years of effort have gone into formally verifying the proofs of theorems that no one doubted were correct. For […]The post When are formal proofs worth the effort? first appeared on John D. Cook.

Can you define something like cross product of three-dimensional vectors in other dimensions? That depends on what you mean by a product being “like” the familiar cross product. In [1] Walsh lists several properties of the cross product, and proves that there exists a product with these properties only in dimensions 1, 3, and 7. […]The post A cross product in 7 dimensions first appeared on John D. Cook.

An integer n is said to be square-free if no perfect square (other than 1) divides n. Similarly, n is said to be cube-free if no perfect cube (other than 1) divides n. In general, n is said to be k-free if it has no divisor d > 1 where d is a kth power […]The post Density of square-free numbers, cube-free numbers, etc. first appeared on John D. Cook.

A digitally delicate prime is a prime number such that if any of its digits are changed by any amount, the result is not prime. The smallest such number is 294001. This means that variations on this number such as 394001 or 291001 or 294081 are all composite. A paper [1] posted last week gives […]The post Digitally delicate primes first appeared on John D. Cook.

Suppose you’re tasked with designing a Christmas countdown figure that uses numbered cube faces. How would you put the numbers on the cubes? Suppose you want the calendar to say “00” on Christmas Day. If we were accustomed to counting in base six, you could label the sides of each cube 0 through 5, and […]The post Cubic calendars first appeared on John D. Cook.

I recently discovered quiver, a tool for drawing commutative diagrams. It looks like a nice tool for drawing diagrams more generally, but it’s designed particularly to include the features you need when drawing the kinds of diagrams that are ubiquitous in category theory. You can draw diagrams using the online app and export the result […]The post Drawing commutative diagrams with Quiver first appeared on John D. Cook.

Let π(x) be the number of primes less than x. The simplest form of the prime number theorem says that π(x) is asymptotically equal to x/log(x), where log means natural logarithm. That is, This means that in the limit as x goes to infinity, the relative error in approximating π(x) with x/log(x) goes to 0. […]The post Refinements to the prime number theorem first appeared on John D. Cook.

The previous post looked at all Boolean expressions on three or four variables and how much they can be simplified. The number of Boolean expressions on n variables is and so the possibilities explode as n increases. We could do n = 3 and 4, but 5 would be a lot of work, and 6 […]The post Minimizing random Boolean expressions first appeared on John D. Cook.

In the previous post we looked at how to minimize Boolean expressions using a Python module qm. In this post we’d like to look at how much the minimization process shortens expressions. Witn n Boolean variables, you can create 2^n terms that are a product of distinct variables. You can specify a Boolean function by […]The post How much can Boolean expressions be simplified? first appeared on John D. Cook.

This post will look at how to take an expression for a Boolean function and look for a simpler expression that corresponds to the same function. We’ll show how to use a Python implementation of the Quine-McCluskey algorithm. Notation We will write AND like multiplication, OR like addition, and use primes for negation. For example, […]The post Minimizing boolean expressions first appeared on John D. Cook.

Linear logic uses an unusual symbol, an ampersand rotated 180 degrees, for multiplicative disjunction. The symbol is U+214B in Unicode. I was looking into how to produce this character in LaTeX when I found that the package cmll has two commands that produce this character, one semantic and one descriptive: \parr and \invamp [1]. This […]The post Rotating symbols in LaTeX first appeared on John D. Cook.

Suppose you want to find the smallest number with 5 divisors. After thinking about it a little you might come up with 16, because 16 = 24 and the divisors of 16 are 2k where k = 0, 1, 2, 3, or 4. This approach generalizes: For any prime q, the smallest number with q divisors […]The post The smallest number with a given number of divisors first appeared on John D. Cook.

Both Pfizer and Moderna have announced recently that their SARS-COV2 vaccine candidates reduce the rate of infection by over 90% in the active group compared to the control (placebo) group. That’s great news. The vaccines may turn out to be less than 90% effective when all is said and done, but even so they’re likely […]The post Good news from Pfizer and Moderna first appeared on John D. Cook.

The other day I saw an article about some math test and thought “I bet I’d blow that away now.” Anyone who has spent a career using some skill ought to blow away an exam intended for people who have been learning that skill for a semester. However, after thinking about it more, I’m pretty […]The post I think I'll pass first appeared on John D. Cook.

A couple years ago I wrote a blog post looking at how close the quaternions come to commuting. That is, the post looked at the average norm of xy – yx. A related question would be to ask how often quaternions do commute, i.e. the probability that xy – yx = 0 for randomly chosen […]The post Probability of commuting first appeared on John D. Cook.

There are simple rules for telling whether a number is divisible by 2, 3, 4, 5, and 6. A number is divisible by 2 if its last digit is divisible by 2. A number is divisible by 3 if the sum of its digits is divisible by 3. A number is divisible by 4 if […]The post Test for divisibility by 13 first appeared on John D. Cook.

This evening I ran across a paper on an unusual coordinate system that creates interesting graphs based from simple functions. It’s called “circular coordinates,” but this doesn’t mean polar coordinates; it’s more complicated than that. [1] Here’s a plot reproduced from [1], with some color added (the default colors matplotlib uses for multiple plots). The […]The post Some mathematical art first appeared on John D. Cook.

Let T(N) be the number of distinct (non-congruent) triangles with integer sides and perimeter N. For example, T(12) = 3 because there are three distinct triangles with integer sides and perimeter 12. There’s the equilateral triangle with sides 4 : 4 : 4, and the Pythagorean triangle 3 : 4 : 5. With a little […]The post Counting triangles with integer sides first appeared on John D. Cook.

I ran across a paper [1] this morning on the differential equation y‘ = sin(xy). The authors recommend having students explore numerical solutions to this equation and discover theorems about its solutions. Their paper gives numerous theorems relating solutions and the hyperbolas xy = a: how many times a solution crosses a hyperbola, at what […]The post Ripples and hyperbolas first appeared on John D. Cook.

When the rule for stopping an experiment depends on the data in the experiment, the results could be biased if the stopping rule isn’t taken into account in the analysis [1]. For example, suppose Alice wants to convince Bob that π has a greater proportion of even digits than odd digits. Alice: I’ll show you […]The post Informative stopping first appeared on John D. Cook.

California’s CCPA regulation has been amended to say that data considered deidentified under HIPAA is considered deidentified under CCPA. The amendment was proposed last year and was finally signed into law on September 25, 2020. This is good news because it’s relatively clear what deidentification means under HIPAA compared to CCPA. In particular, HIPAA has […]The post Expert determination for CCPA first appeared on John D. Cook.

I imagine most programmers who develop an interest in category theory do so after hearing about monads. They ask someone what a monad is, and they’re told that if they really want to know, they need to learn category theory. Unfortunately, there are couple unnecessary difficulties anyone wanting to understand monads etc. is likely to […]The post Category theory for programmers made easier first appeared on John D. Cook.

The previous post looked at random Fibonacci sequences. These are defined by f1 = f2 = 1, and fn = fn-1 ± fn-2 for n > 2, where the sign is chosen randomly to be +1 or -1. Conjecture: Every integer can appear in a random Fibonacci sequence. Here’s why I believe this might be […]The post Is every number a random Fibonacci number? first appeared on John D. Cook.

The Fibonacci numbers are defined by F1 = F2 = 1, and for n > 2, Fn = Fn-1 + Fn-2. A random Fibonacci sequence f is defined similarly, except the addition above is replaced with a subtraction with probability 1/2. That is, f1 = f2 = 1, and for n > 2, fn = […]The post Random Fibonacci numbers first appeared on John D. Cook.