John D. Cook

I recently ran across the article Something weird is happening with LLMs and chess. One of the things it mentions is how the a minor variation in a prompt can have a large impact on the ability of an LLM to play chess. One extremely strange thing I noticed was that if I gave a [...]The post Practical consequences of tokenization details first appeared on John D. Cook.

I mentioned in the previous post that not every Unicode character corresponds to a token in ChatGPT. Specifically I'm looking at gpt-3.5-turbo in tiktoken. There are 100,256 possible tokens and 155,063 Unicode characters, so the pigeon hole principle says not every character corresponds to a token. I was curious about the relationship between tokens and [...]The post ChatGPT tokens and Unicode first appeared on John D. Cook.

Large language models operate on tokens, not words, though tokens roughly correspond to words. A list of words would not be practical. There is no definitive list of all English words, much less all words in all languages. Still, tokens correspond roughly to words, while being more flexible. Words are typically turned into tokens using [...]The post The difference between tokens and words first appeared on John D. Cook.

The GELU (Gaussian Error Linear Units) activation function was proposed in [1]. This function is x (x) where is the CDF of a standard normal random variable. As you might guess, the motivation for the function involves probability. See [1] for details. The GELU function is not too far from the more familiar ReLU, [...]The post A simpler GELU activation function approximation first appeared on John D. Cook.

A common technique for memorizing numbers is to associate numbers with words. The Major mnemonic system does this by associating consonant sounds with each digit. You form words by inserting vowels as you please. There are many possible encodings of numbers, but sometimes you want to pick a canonical word for each number, what's commonly [...]The post Spreading out words in space first appeared on John D. Cook.

Databasetechnology is a mature field, and techniques for optimizing databases are well understood. However, surprises can still happen. Certain performance optimizations you might expect to be automatic are not really. I'm working with a legacy code developed some time ago, before modern notions of separation of concerns between code business logic and data storage. The [...]The post On Making Databases Run Faster first appeared on John D. Cook.

Like the previous post, this post also seeks to approximately reproduce a hand-drawn plot. This time the goal is reproduce figure 7.3 from A&S page 298. This plot is a visualizing of the function of a complex variable w(z) = exp(-z^2) erfc(- iz) where erfc is the complementary error function. A&S calls the graph above [...]The post Duplicating a hand-drawn contour plot first appeared on John D. Cook.

The plots in old (i.e. pre-computer) math books are impressive. These plots took a lot of effort to produce, and so they weren't created lightly. Consequently they tend to be aesthetically and mathematically interesting. A few weeks ago I recreated a plot from A&S using Mathematica, and today I'd like to do the same for [...]The post Reproducing a hand-drawn plot first appeared on John D. Cook.

Gaussian elimination is pretty simple if all goes well, and in introductory courses all does go well. You have an n * n matrix A, an n * 1 column vector b, and you want to find an n * 1 column vector x such that Ax = b. You subtract multiples of the first [...]The post What is partial pivoting? first appeared on John D. Cook.

An earlier post needed to calculate how much the speed of a planet varies in orbit. The planet moves fastest as perihelion, the point in its orbit closes to the sun, and it moves slowest at aphelion, when it is furthest from the sun. The ratio of the maximum to minimum speed turns out to [...]The post Max and min orbital speed first appeared on John D. Cook.

The previous post looked at one challenge with designing a calendar for Mars, namely how to vary the number of days per month so that each month corresponds to a change of 30 degrees with respect to the sun. This is a bigger problem on Mars than here on Earth. That post assumed that a [...]The post Martian Leap Years first appeared on John D. Cook.

I recently started reading The Case for Mars by Robert Zubrin. This post will unpack one line from that book regarding creating a calendar for Mars: Equipartitioned months don't work for Mars, because the planet's orbit is elliptical, which causes the seasons to be of unequal length. This sentence doesn't sit well at first for [...]The post A calendar for Mars first appeared on John D. Cook.

Making your code to run faster starts with understanding where in the code the runtime is actually spent. But suppose, for whatever reason, the code profiling tools won't work? I recently used MS Visual Studio on a legacy C++ code. The code crashed shortly after startup when attempting to profile, though otherwise the code ran [...]The post Code Profiling Without a Profiler first appeared on John D. Cook.

Is the shape of a triangle determined by its perimeter and area? In other words, if two triangles have the same area and the same perimeter, are the triangles similar? [1] It's plausible. A triangle has three degrees of freedom: the lengths of the three sides. Specifying the area and perimeter removes two degrees of [...]The post Do perimeter and area determine a triangle? first appeared on John D. Cook.

When my children were little, I read the Little House on the Prairie books aloud to them and I naturally saw the books through the eyes of a child. Last night I started reading the books by myself for the first time and saw them very differently. Laura Ingalls Wilder wrote the Little House books [...]The post Settlers versus Hipsters first appeared on John D. Cook.

The previous post looked at how iterations of converge to the golden ratio . That post said we could start at any positive x. We could even start at any x > -3/4 because that would be enough for the derivative of (1 + x) to be less than 1, which means the iteration will [...]The post Complex golden convergence first appeared on John D. Cook.

The golden ratio satisfies the following equation. The proof most commonly given is to let x equal the right-hand side of the equation, then observe that x^2 = 1 + x, the quadratic equation for the golden ratio. The quadratic has two roots: and -1/. Since x > 1, x = . This [...]The post Golden convergence first appeared on John D. Cook.

The Major mnemonic system makes numbers easier to memorize by encoding them as words. Each digit corresponds to one or more consonant sounds, and you can fill in vowels as you wish. In August 2022 I tried creating a few images using DALL-E 2. The results were disappointing and sometimes disturbing. To illustrate the use [...]The post Mnemonic images with Grok 3 first appeared on John D. Cook.

Is it possible to come up with truly innovative ideas when you're not part of the institutions where the expertise resides? According to one study, the answer would seem to be No." The book, The Sociology of Philosophies by Randall Collins, makes a case for how great ideas through history have always developed, almost [...]The post Standing with Intellectual Giants first appeared on John D. Cook.

This morning I read Alberto Romeo's article Grok 3: Another Win For The Bitter Lesson. I wasn't clear on what exactly the bitter lesson was, so I followed the link to Romeo's previous article GPT-4: The Bitterer Lesson, which lead to Rich Sutton's original article The Bitter Lesson. Sutton opens his article by saying The [...]The post Rich Sutton's bitter lesson of AI first appeared on John D. Cook.

The multiplication table of a finite group forms a Latin square. You form the multiplication table of a finite group just as you would the multiplication tables from your childhood: list the elements along the top and side of a grid and fill in each square with the products. In the context of group theory [...]The post Multiplication tables and Latin squares first appeared on John D. Cook.

The Buenos Aires constant is 2.92005097731613... What's so special about this number? Let's see when we use it to initialize the following Python script. s = 2.920050977316134 for _ in range(10): i = int(s) print(i) s = i*(1 + s - i) What does this print? 2, 3, 5, 7, 11, 13, 17, 19, 23, [...]The post The Buenos Aires constant first appeared on John D. Cook.

The other day MathMatize posted roses are red books go on a shelf 1+2+3+4+ ... with a photo of Ramanujan on X. This was an allusion to the bizarre equation 1 + 2 + 3 + ... = - 1/12. This comes up often enough that I wanted to write a post that I could [...]The post 1 + 2 + 3 + ... = -1/12 first appeared on John D. Cook.

The cosine of a multiple of can be written as a polynomial in cos . For example, cos 3 = 4 cos3 - 3 cos and cos 4 = 8 cos4 - 8 cos2 + 1. But it may or may not be possible to write the sine of a [...]The post Multiple angle asymmetry first appeared on John D. Cook.

I read something once about an American telegraph operator who had to switch over to using Russian Morse code during WWII. I wondered how hard that would be, but let it go. The idea came back to me and I decided to settle it. It would be hard to switch from being able to recognize [...]The post Russian Morse Code first appeared on John D. Cook.

Two families of orthogonal polynomials are named after Chebyshev because he explored their properties. These are prosaically named Chebyshev polynomials of the first and second kind. I recently learned there are Chebyshev polynomials of the third and fourth kind as well. You might call these posthumous Chebyshev polynomials. They were not developed by Mr. Chebyshev, [...]The post Posthumous Chebyshev Polynomials first appeared on John D. Cook.

I recently ran across an interesting family of Pythagorean triples [1]. You can verify that a^2 + b^2 = c^2 for all n. Sparseness When written in binary, a has only two bits set, and c has only four bits set. It's not as immediately obvious, but b has only two bits that are not [...]The post Sparse binary Pythagorean triples first appeared on John D. Cook.

The reactions to the new DeepSeek-R1 AI model in recent days seem limitless. Some say it runs so much faster than existing models that we will no longer need the billions of dollars in compute hardware that big tech is preparing to buy. Is that plausible? To get an answer, we need only look [...]The post DeepSeek-R1: Do we need less compute now? first appeared on John D. Cook.

The previous post discussed complex numbers, dual numbers, and double numbers. All three systems are constructed by adding some element to the real numbers that has some special algebraic property. The complex numbers are constructed by adding an element i such that i^2 = -1. The dual numbers add an element 0 with [...]The post Matrix representations of number systems first appeared on John D. Cook.

The complex numbers are formed by adding an element i to the real numbers such that i^2 = - 1. We can create other number systems by adding other elements to the reals. One example is dual numbers. Here we add a number 0 with the property ^2 = 0. Dual numbers have [...]The post Euler's formula for dual numbers and double numbers first appeared on John D. Cook.

The simplest trick for converting from one base to another is grouping. To convert between base b and base bk, group numbers in sets of k and convert one group at a time. To convert from binary to octal, for instance, group bits in sets of three, starting from the right end, and convert each [...]The post Tricks for radix conversion by hand first appeared on John D. Cook.

The previous post started by saying that rounding has a surprising amount of detail. An example of this is double rounding: if you round a number twice, you might not get the same result as if you rounded directly to the final precision. For example, let's say we'll round numbers ending in 0, 1, 2, [...]The post Double rounding first appeared on John D. Cook.

Rounding numbers has a surprising amount of detail. It may seem trivial but, as with most things, there is a lot more to consider than is immediately obvious. I expect there have been hundreds if not thousands of pages devoted to rounding in IEEE journals. An example of the complexity of rounding is what William [...]The post A magical land where rounding equals truncation first appeared on John D. Cook.

Abramowitz and Stegun has quite a few intriguing plots. The post will focus on the follow plot, Figure 9.4, available here. We will explain what the plot is and approximately reproduce it. The plot comes from the chapter on Bessel functions, but the caption says it is a plot of the Hankel function H0(1). Why [...]The post Duplicating Hankel plot from A&S first appeared on John D. Cook.

Last week Heron's formula came up in the post An Unexpected Triangle. Given the lengths of the sides of a triangle, there is a simple expression for the area of the triangle. where the sides are a, b, and c and s is the semiperimeter, half the perimeter. Is there an analogous formula for the [...]The post Area of a quadrilateral from the lengths of its sides first appeared on John D. Cook.

The purpose of this post is to expand on the following sentence [1]: Russian letters are created by entering [Ctrl-k followed by] a corresponding Latin letter followed by an equals sign -, or, in a few places, a percent sign %. The Russian alphabet has 33 letters, so there couldn't be a Latin letter for [...]The post Entering Russian characters in Vim with digraphs first appeared on John D. Cook.

It's not simple to transliterate Russian names to English. Sometimes there is a unique mapping, or at least a standard mapping, of a particular name, but often there is not. An example that comes up frequently in mathematics is Pafnuty Lvovich Chebyshev (1821-1894). This Russian mathematician's name has been transliterated at Tchebichef, [...]The post Chebyshev and Russian transliteration first appeared on John D. Cook.

Let J(x) be the function plotted below. This is the Bessel function J1, but we drop the subscript because it's the only Bessel function we're interested in for this post. You can think of J as a sort of damped sine. We can create versions of J with different frequencies by multiplying the argument x [...]The post An unexpected triangle first appeared on John D. Cook.

Many are voicing concern that the world is running out of data and that this will be a blocker to progress toward smarter AI models. One paper in fact projects timelines for when we will run out. AI researchers are looking for ways to adapt.Nvidia has trained a specific model to generate synthetic data for [...]The post Can AI Models Reason: Is Data All You Need? first appeared on John D. Cook.

Sometimes it's useful to apply dimensional analysis where it doesn't belong, to imagine things having physical dimension when they don't. This post will look at artificially injecting dimensions into equations involving factorials and related functions. Factorials The factorial of n is defined as the product of n terms. If each of these terms had units [...]The post Dimensional analysis for gamma function values first appeared on John D. Cook.

Yesterday I wrote about how the right notation could make Newton's interpolation theorem much easier to remember, revealing it as an analog of Taylor series. This post will do something similar for the binomial theorem. Let's start with the following identity. It's not clear that this is true, or how one might generalize it. But [...]The post Falling power analog of binomial theorem first appeared on John D. Cook.

Suppose one company would like to buy another company's client list, but only if the lists don't overlap too much. Neither company wants to hand over their list to the other before a sale takes place. What can they do? A low-tech solution would be for both parties to provide their client lists to a [...]The post Why eliminate trusted third parties? first appeared on John D. Cook.

Newton's interpolation formula looks awfully complicated until you introduce the right notation. With the right notation, it looks like a Taylor series. Not only is this notation simpler and more memorable, it also suggests extensions. The notation we need comes in two parts. First, we need the forward difference operator defined by and its [...]The post Discrete Taylor series first appeared on John D. Cook.

The previous post was an AI-generated podcast that I friend made by crawling my web site. I decided to create an actual podcast for posting occasional audio files. I expect to post very sporadically. I've posted two audio files, and I have one more in mind to post some day. Maybe that'll be the end [...]The post Podcast feed first appeared on John D. Cook.

A friend just sent me an audio file of a podcast about my consulting that he created by asking an AI to crawl my web site. The podcast is a remarkably natural-sounding conversation between two synthetic hosts. The only clues that the audio is automatically generated are a couple of mispronounced acronyms. DownloadThe post Consulting Podcast first appeared on John D. Cook.

A recent article reported on the successful factoring of a 512-bit RSA key. The process took $8 worth of rented computing power. What does that say about the security of RSA encryption? The following Python function estimates the computation steps required to factor a number b bits long using the best known algorithm. We're going [...]The post RSA security in light of news first appeared on John D. Cook.

We're awaiting the release of OpenAI's o3 model later this month. Its performance is impressive on very hard benchmarks like SWE-bench Verified, Frontier Math and the ARC AGI benchmark (discussed previously in this blog). And yet at the same time some behaviors of the frontier AI models are very concerning. Their performance on assorted math [...]The post Can AI models reason like a human? first appeared on John D. Cook.

One difficulty when working at a command line is navigating between directories, particularly between locations with long paths. There are several ways to mitigate this. One of the simplest is using cd - to return to the previous directory. Another is to use pushd and popd. Still another is to set the CDPATH variable. qcd [...]The post Quick change directory first appeared on John D. Cook.

The security of RSA encryption depends on the difficulty of factoring the product of two large primes. If you can factor large numbers efficiently, you can break RSA. But if can break RSA, can you factor large numbers? Sorta. It's conceivable that there is a way to break RSA encryption without having to recover the [...]The post Converse of RSA first appeared on John D. Cook.

Steganography attempts to prevent messages from being read by unintended recipients by hiding the messages rather than (or in addition to) encrypting them. Steganography is used when you not only want to keep your communication private, you want to hide the fact that you've communicated at all. Fun fact: The words steganography and stegosaurus are [...]The post Unicode Steganography first appeared on John D. Cook.