John D. Cook

The article Deming, data and observational studies by S. Stanley Young and Alan Karr opens with Any claim coming from an observational study is most likely to be wrong. They back up this assertion with data about observational studies later contradicted by prospective studies. Much has been said lately about the assertion that most published results are false, particularly […]

It’s hard to transfer intellectual property. When I was managing software projects, it would take months to fully transfer a project from one person to another. This was with full access to and encouragement from the original developer. This was a transfer between peers, both part of the same environment with all its institutional memory. […]

The Insight 2015 conference highlighted some impressive applications of big data: predicting the path of hurricanes more accurately (as we saw with hurricane Patricia), improving the performance of athletes, making cars safer, etc. These applications involve large amounts of data. But more importantly they involve new data, not simply greater quantities of data we’ve had before. […]

A/B testing, or split testing, is commonly used in web marketing to decide which of two design options performs better. If you have so many visitors to a site that the number of visitors used in a test is negligible, conventional randomization schemes are the way to go. They’re simple and effective. But if you […]

A few weeks ago I got a message on Twitter saying that IBM’s Watson had identified me as an “influencer” and invited me to the company’s Insight 2015 conference. So that’s where I am this week. I had a brief interview last night. Someone took this photo as we were setting up.

The distance between the Earth and Mars depends on their relative positions in their orbits and varies quite a bit over time. This post will show how to compute the approximate distance over time. We’re primarily interested in Earth and Mars, though this shows how to calculate the distance between any two planets. The planets […]

You may expect that a burst of input will cause a burst of output. Sometimes that’s the case, but often a burst of input results in a long, smoothly decreasing succession of output. You may not get immediate results, but long-term results. This is true of life in general, but it’s also true in a precise sense of differential equations. […]

Suppose a test asks you to place 10 events in chronological order. Label these events A through J so that chronological order is also alphabetical order. If a student answers BACDEFGHIJ, then did they make two mistakes or just one? Two events are in the wrong position, but they made one transposition error. The simplest way […]

Electric lighting has changed the way we sleep, encouraging us to lose sleep by staying awake much longer after dark than we otherwise would. Or maybe not. A new study of three contemporary hunter-gatherer tribes found that they stay awake long after dark and sleep an average of 6.5 hours a night. They also don’t nap […]

Here’s an unusual formula for pi based on the product and least common multiple of the first m Fibonacci numbers. Unlike the formula I wrote about a few days ago relating Fibonacci numbers and pi, this one is not as simple to prove. The numerator inside the root is easy enough to estimate asymptotically, […]

Energy efficiency improvements can pay for themselves in the long run. Financing can make the improvements immediately cash-flow positive, but only if the loan tenor can match the useful life of the equipment. This enables the payments to be low enough that the projected energy savings exceeds the payments. PACE, which stands for Property Assessed Clean Energy, is […]

I help people make decisions in the face of uncertainty. Sounds interesting. I’m a data scientist. Not sure what that means, but it sounds cool. I study machine learning. Hmm. Maybe interesting, maybe a little ominous. I’m into big data. Exciting or passé, depending on how many times you’ve heard the term. Even though each […]

Here’s an unusual formula for π. Let Fn be the nth Fibonacci number. Then As mysterious as this equation may seem, it’s not hard to prove. The arctangent identity shows that the sum telescopes, leaving only the first term, arctan(1) = π/4. To prove the arctangent identity, take the tangent of both sides, use the addition law for tangents, and […]

When I was growing up, I was told that you could never become fluent in a second language, and I believed it. I had no reason not to. I didn’t know anybody who had become fluent at a second language, and I could think of plenty of people who had learned English as a second […]

The other day I ran across the fact that 23! has 23 digits. That made me wonder how often n! has n digits. There can only be a finite number of cases, because n! grows faster than 10n for n > 10, and it’s reasonable to guess that 23 might be the largest case. Turns out it’s […]

John Tukey preferred the term “data analysis” over “statistics.” In his paper Data Anaysis, Computation and Mathematics, he explains why. My title speaks of “data analysis” not “statistics”, and of “computation” not “computing science”; it does not speak of “mathematics”, but only last. Why? … My brother-in-squared-law, Francis J. Anscombe has commented on my use of […]

There are huge opportunities to take technology that is well-known and undervalued in one context and apply it in another where it is unknown but valuable. You could call this technical arbitrage, analogous to financial arbitrage, taking advantage of the price difference of something in two markets. As with financial arbitrage, the hard part is […]

In the novel Enchantment, the main character, Ivan, gives a bitter assessment of his choice of an academic career, saying it was for “men who hadn’t yet grown up.” The life he had chosen was a cocoon. Surrounded by a web of old manuscripts and scholarly papers, he would achieve tenure, publish frequently, teach a group of […]

A function f is periodic if there exists a constant period ω such that f(x) = f(x + ω) for all x. For example, sine and cosine are periodic with period 2π. There’s only one way a function on the real line can be periodic. But if you think of functions of a complex variable, […]

This is a post about letting go of something you think you need. It starts with an illustration from programming, but it’s not about programming. Bob Martin published a dialog yesterday about the origin of structured programming, the idea that programs should not be written with goto statements but should use less powerful, more specialized […]

One of my clients is writing software in Julia so I’m picking up the language. I looked at Julia briefly when it first came out but haven’t used it for work. My memory of the language was that it was almost a dialect of Python. Now that I’m looking at it a little closer, I […]

This is reprint of Nick Higham’s post of the same title from the Princeton University Press blog, used with permission. Color is a fascinating subject. Important early contributions to our understanding of it came from physicists and mathematicians such as Newton, Young, Grassmann, Maxwell, and Helmholtz. Today, the science of color measurement and description is […]

It would be hard to think of two programming languages more dissimilar than Haskell and R. Haskell was designed to enforce programming discipline; R was designed for interactive use. Haskell emphasizes correctness; R emphasizes convenience. Haskell emphasizes computer efficiency; R emphasizes interactive user efficiency. Haskell was written to be a proving ground for programming language theorists. R was written to be […]

The most direct way to compute a Fourier transform numerically takes O(n2) operations. The Fast Fourier Transform (FFT) can compute the same result in O(n log n) operations. If n is large, this can be a huge improvement. James Cooley and John Tukey (re)discovered the FFT in 1965. It was thought to be an original discovery […]

Each Fibonacci number is the sum of its two predecessors. My previous post looked at generalizing this to the so-called Tribonacci numbers, each being the sum of its three predecessors. One could keep going, defining the Tetrabonacci numbers and in general the n-Fibonacci numbers for any n at least 2. For the definition to be complete, […]

Take an n × n matrix A and a vector x of length n. Now multiply x by A, then multiply the result by A, over and over again. The sequence of vectors generated by this process will converge to an eigenvector of A. (An eigenvector is a vector whose direction is unchanged when multiplied […]

I got a call this afternoon from someone who records audio books for the blind. He wanted to know the name of a symbol he didn’t recognize. He then asked me if the equation was real. Here’s the equation in context, from the book Michael Vey 4: Hunt for Jade Dragon. The context is as follows. Suddenly math […]

Yesterday on Twitter I said I was thinking about writing the names of each of my clients and leads on balls so I could literally juggle them. I was only half joking. I didn’t write my clients and leads on balls, but I did write them on index cards. And it helped a great deal. It’s […]

A while back I wrote about a method to test whether a number is divisible by seven. I recently ran across another method for testing divisibility by 7 in Martin Gardner’s book The Unexpected Hanging and Other Mathematical Diversions. The method doesn’t save too much effort compared to simply dividing by 7, but it’s interesting. It looks […]

The previous post stated a formula for f(n), the nth square triangular number (i.e. the nth triangular number that is also a square number): ((17 + 12√2)n + (17 – 12√2)n – 2)/32 Now 17 – 12√2 is 0.029… and so the term (17 – 12√2)n approaches zero very quickly as n increases. So the f(n) […]

Of course a triangle cannot be a square, but a triangular number can be a square number. A triangular number is the sum of the first so many positive integers. For example, 10 is a triangular number because it equals 1+2+3+4. These numbers are called triangle numbers because you can form a triangle by having a row of […]

In the preface to his book Strength of Materials, J. P. Den Hartog says After the alphabet and the tables of multiplication, nothing has proved quite so useful in my professional life as these six little expressions. The six expressions he refers to are nicknamed the vergeet-me-nietjes in Dutch, which translates to forget-me-nots in English. They are also known […]

You’ve probably heard someone ask someone else what books they would take to a deserted island. It’s usually implied that you’re bringing books for pleasure, not books that would help you survive on the island or leave it. People often answer the question with a list of their favorite books, perhaps skewed in favor of long […]

This afternoon I helped someone debug a financial spreadsheet. One of the reasons spreadsheets can be so frustrating to work with is that assumptions are hard to see. You have to click on cells one at a time to find formulas, then decode cell coordinates into their meanings. The root problem turned out to be […]

Earlier this week I had a chance to talk with Anthony Scopatz and Katy Huff about their new book, Effective Computation in Physics. JC: Thanks for giving me a copy of the book when we were at SciPy 2015. It’s a nice book. It’s about a lot more than computational physics. KH: Right. If you think of […]

In my previous post, we looked at the largest known prime, P = 257885161 – 1, and how many digits it has in various bases. This post looks at how to find the last digit of P in base b. We assume b < P. (If b = P then the last digit is 0, and if […]

The largest known prime at the moment is P = 257885161 – 1. This means that in binary, the number is a string of 57,885,161 ones. You can convert binary numbers to other bases that are powers of two, say 2k, by starting at the right end and converting to the new base in blocks of […]

If you write out a sequence of Fibonacci numbers, you can see that the last digits repeat every 60 numbers. The 61st Fibonacci number is 2504730781961. The 62nd is 4052739537881. Since these end in 1 and 1, the 63rd Fibonacci number must end in 2, etc. and so the pattern starts over. It’s not obvious that the cycle […]

David Morice rewrote Edgar Allen Poe’s poem The Raven in words of one syllable. Here is the first stanza: Once at twelve on one night’s drear, ’twas while I, weak and tired thought here On the words in lots of quaint and odd old tomes of mind’s lost lore, While I dozed, so near a nap, there […]

Allen Wirft-Brock gave the following defense of OOP a few days ago in a series of six posts on Twitter: A young developer approached me after a conf talk and said, “You must feel really bad about the failure of object-oriented programming.” I was confused. I said, “What do you mean that object-orient programming was […]

Ten life lessons from differential equations: Some problems simply have no solution. Some problems have no simple solution. Some problems have many solutions. Determining that a solution exists may be half the work of finding it. Solutions that work well locally may blow up when extended too far. Boundary conditions are the hard part. Something […]

Harmonic numbers The nth harmonic number, Hn, is the sum of the reciprocals of the integers up to and including n. For example, H4 = 1 + 1/2 + 1/3 + 1/4 = 25/12. Here’s a curious fact about harmonic numbers, known as Wolstenholme’s theorem: For a prime p > 3, the numerator of Hp-1 […]

Numerically integrating functions of many variables almost always requires Monte Carlo integration or some variation. Numerical analysis textbooks often emphasize one-dimensional integration and, almost as a footnote, say that you can use a product scheme to bootstrap one-dimensional methods to higher dimensions. True, but impractical. Traditional numerical integration routines work well in low dimensions. (“Low” […]

Zawinski’s law of software development: Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. The folks behind Mathematica and Wolfram Alpha announced yesterday now they too can read email.

Scientific computing in Python is expanding and maturing rapidly. Last week at the SciPy 2015 conference there were about twice as many people as when I’d last gone to the conference in 2013. You can get some idea of the rapid develop of the scientific Python stack and its future direction by watching the final […]

My contact information arranged into a diagram: Yesterday at SciPy 2015 Allen Downey did something similar for his contact info and gave me the idea for the image above. LinkedIn doesn’t quite fit; you have to know that LinkedIn profiles stick linkedin.com/in/ before the user name.

A certain question has the following possible answers. All of the below None of the below All of the above One of the above None of the above None of the above Which answer is correct? Source

If you raise any integer to the fifth power, its last digit doesn’t change. For example, 25 = 32. It’s easy to prove this assertion by brute force. Since the last digit of bn only depends on the last digit of b, it’s enough to verify that the statement above holds for 0, 1, 2, […]

Periodically someone on Twitter will suggest that one of my Twitter accounts is a bot. Others will reply in the second person plural, suggesting that there’s a group of people behind one of the accounts. These accounts aren’t run by a bot or a committee, just me. I do use software to schedule my tweets […]

From The Book of Strange New Things: … I said that if science could come up with something like the Jump it could surely solve a problem like that. Severin seized hold of that word, “science.” Science, he said, is not some mysterious larger-than-life force, it’s just the name we give to bright ideas that […]