CHANCE News 12.02
|
word |
rank |
freq |
| one |
32 |
3297 |
two |
69 |
1412 |
three |
151 |
610 |
four |
260 |
359 |
five |
318 |
286 |
six |
420 |
220 |
seven |
903 |
114 |
eight |
1002 |
104 |
nine |
1302 |
81 |
This is similar to the fact that Benford's law which states that in
many different sets of numbers, the frequences of the leading digits
of numbers decreases logrithmcally. Since it would seem that most
of these words would correspond to the leading digit of a number it
is natural to compare this with the Benford distribution.
To do this we looked at two other data sets of words.
The British National Corpus (BNC) is a 100 million word collection of samples of written and spoken language from a wide range of sources, designed to represent a wide cross-section of current British English, both spoken and written.
The data providing the frequencies of the words is available from Adam Kilgarriff's website . For our second check we searched on google for the words one, two, three etc and recorded the number of web pages that turned up. Of course these were huge numbers and related to the number of websites rather than distince texts. Since the word "one" has two different meanings we considered only the words for numbers from 2 to 9. We normalized the frequences so that their sum adds to 1. Here are the results.
| Word | Brown | BNC | Benford | |
| two | .44 | .39 | .38 | .25 |
| three | .19 | .20 | .20 | .18 |
| four | .11 | .11 | .13 | .14 |
| five | .09 | .10 | .10 | .11 |
| six | .07 | .08 | .08 | .10 |
| seven | .04 | .04 | .05 | .08 |
| eight | .03 | .04 | .04 | .07 |
| nine | .03 | .03 | .03 | .07 |
We see that the distriptions for the different word data sets are quite similar but they are not consistent with the frequences that would be predicted by the Benford distribution.
Science and Nature: The gift of the gab.
The Independent (London), 24 March, 2003
Kate Ravilious
Least effort and the origins of scaling in human language.
Proceedings of the National Academy of Sciences, 4 February, 2003
Ramon i Ferrer Cancho and Ricard V. Solé.
This article is a good example of how a science writer writing about a recent technical paper can attempt to attract the attention of the reader. In the first paragraph we read:
Japanese, English, Swahili and Hungarian might sound completely unrelated, but it turns out that they have more in common than most of us realise. Researchers have identified a pattern in all human languages, and it now appears that this pattern may explain the very origins of how we began to talk.
In 1916 the French stenographer J. B. Estoup (2) noted that if you listed words in a text according to the frequence of their occurence and then for each word multiply its frequency (F) by its rank (R) you would always get about the same number. Pereto (3) observed the same phenomena relating to corporate wealth. In his book "Human behavior and the Principle of Least Effort" (1) George Zipf showed that this constant product law applied to ranking populations of cities and many more applications and the law became known as Zipf's law.
At Tapinternet.com we were able to obtain a frequency account by just providing the URL. Applying this to the last two issues of Chance News we find that the top 20 words according to frequency are:
Chance News 12.01 |
Chance News 11.05 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Like most texts "the" and "and" have the highest frequencies and so have rank 1 and 2 respectively. However, the rankings of other words differs as would be expected since, for example, we had no reason to refer to Medicare in the earlier Chance News. Since these are small samples we also do not expect the products to be exactly equal but they are not two different. Also, since some words have the same frequencies and different ranks it is not possible that the products all equal a constant. However, let's now look at a bigger text database.
If F(w)R(w) = c then logF(w) = log(c) - log(R(w) so a logarithmic plot of rank versus frequency should lie on a straight line with slope -1. To check this for a larger source we used the British National Corpus mentioned in the previous commentary. Here is the result of this plot.
The line with slope -1 fits the higher part of the graph corresponding to high frequence word but to fit the lower part of the graph corresponding to words uses infrequently we had to choose a line with larger negative slope (1.64). This would mean that log(f(w) = c - 1.63Log(r(w) or f(w)r(w)^1.63 = log(c). Thus the relation between the frequences and ranks in both parts of the graph can be represented by laws of the form
f = cr-a
A law of this form is called a "power law" . The
log log plot of a power law will give a streat line with slope -a.
Cancho and Solé (1) noted that two such laws were needed and commented on the change in the exponent of the power law saying: We suggest that here the change in exponents is related to the average amount of words that the human brain is able to store and use.
As shown in Cancho and Solé (1) if the relation between word frequency and rank satisfys a power law with exponent a then the probability p(n) that a word is present n times in the text power law with exponent b = 1 + 1/a. Thus if Zipf's law were to hold with exponent 1, then p(n) should obey a power low with exponent 2. To check with with our data we do a log log plot of p(n) against n.
A single line seemed to fit this data pretty well. The line has slope -.64 which is bigger than the -.5 predicted for Zipfs law. In their discussion using similar data Cancho and Solé again use two lines and for the bottom of the graph the slope is close to -.5 for the top half it is -.63. Thus Zipf's law works best for the top halve when we compare frequency with rank and for the bottom half when we look at the distribution of frequencies.
We cannot resist the following interesting application of Zipf's laws for word frequencies carried out by Gabriel Landini. The Voynich manuscript is a famous manuscript shrowded in mystery. This mystory is described by Victory Segal in her brief account of a BBC4 documentary in the Sunday Times (London) 8 Dec. 2002. She writes:
While it is easy to mock conspiracy theorists as loners in X Files T-shirts, this documentary explores a puzzle whose appeal stretches way beyond the glow of the internet chatroom. The Voynich manuscript was discovered in an Italian villa by an antiquarian book dealer, Wilfred Voynich, early last century. Decorated with pictures of plants, stars and "naked nymphs", its 238 pages are written in an unreadable language, a peculiar grid of symbols that have resisted all attempts at translation.
Some think it is a code, others a language in its own right. It might come from the 13th century or it might be a modern forgery, perpetrated by Voynich himself. This is a perfect introduction to the manuscript and its lore, veering between theories at a dizzying rate. Roger Bacon, John Dee and Emperor Rudolph II all play their part, as do National Security Council cryptographers, Yale maths professors and an array of eccentric amateurs, from computer games programmers to herbalists. And, of course, there is a man in South Carolina who thinks extraterrestrials hold the key.
You can find more details of the history and and current research on the web. We enjoyed the discussion here which has several nice pictures from the manuscript including this one:
Several electronic versions of the Voynich manuscript which use arbitray alphabets to transcribe the symbols are availible. These have been used to study statistical properties of the manuscript. In particular Landini used two of these to see if Zipf's law appied to the frequencies of the Voynich text. His results are described here. Here are plots that he gives for the Rank-Frenquency relation and the words relations using a translation called vornich.now:
For comparison Landini gives similar plots for several well known text sources: Alice in Wonderland,
Of course it would be nice if one could conclude that the Voynich manuscript is not a hoax from the fact that the frequency of words in the Voynich manuscript satisfies Zipf's law. This is complicated by the fact that is has been verified by a number of studies that words generated randomly also appear to satisfy Zipf's law. However
Zipf did not give a mathematical formulation but as the title of his book suggests he had the idea that it was related to a principle of least effort. For the case of development of a language Zipf wrote:
Whenever a person uses words to convey meanings he will automatically try to get his ideas across most efficiently by seeking a balance between the economy of a small wieldy vocabuary of more general reference on the one hand, and the economy of a larger one of more precice reference on the other hand, with the result that the vocabulary of n different words in his resulting flow of speech will represent a vocabulary balance between our theoretical Forces of Unification and Diversificaition.
To check this we look at larger text sources. We have alread mention
the
Of course we cannot expect a constant product in a small sample of
text. Note that the rank of the words obviously depends on the context.
We would not expect chance to be ranked this high in a comparable
Zipf's laws
G. K. Zipf described a number of common properties of natural languages
including two "laws" of word frequencies. He proposed that
these laws were the result of his controversial principle of least
effort (see reference 1). The two laws of word frequencies are:
The "rank-frequency" law.
This is the most famous one; unfortunately many people call it "Zipf's
law" as if it was the only one. Interestingly, Zipf mentions
in his book that this way of looking at word frequencies "was
suggested by a friend" (!). The procedure to estimate this relation
is very simple: the words in a text are sorted by decreasing frequency
and a rank number is assigned to each word. For words with the same
frequency, the subsorting and ranking is arbitrary. The plot of log(frequency)
(y axis) versus log(rank) (x axis) approximates a straight line of
slope -1.
The "number-frequency" law.
Being n the frequency of a word, the plot of log(n) (y axis) versus
log(number of words with frequency n) (x axis) approximates a straight
line of slope -0.5. While the rank-frequency law tends to occur for
the high frequency words (although not necessarily for the first few
ranking positions), the number-frequency law is observed for the low
frequency words.
The three word normalized frequences are pretty similar but Benford has too few ones which we suggested could be due to the two uses of the word "one" both of which are counted in the other examples.
We checked Tom's observation in two other data sets.
We recieved the following comments from reader JeffSimonoff related to an item in the February 10, 2003 issue of Sports Illus trated:
In a small item named "Lining Up for LeBron" (on page 37), author Chris Ballard gives his odds that each of seven NBA teams will finish last in the NBA during the season, thereby giving them the best shot at Ohio schoolboy star LeBron James in the NBA Draft Lottery. The odds given are as follows:
| Team | Odds |
| Miami Heat | 10-1 |
| Memphis Grisslies | 7-1 |
| Atalanta Hawks | 5-1 |
| Los Angeles Clippers | 7-2 |
| Denver Nuggets | 3-1 |
| Toronto Raptors | Even |
| Cleveland Cavaliers |
Miami Heat: 10 to 1
Memphis Grizzlies: 8 to 1
Atlanta Hawks: 7 to 1
Los Angeles Clippers: 5 to 1
Denver Nuggets: 7 to 2
Toronto Raptors: 3 to 1
Cleveland Cavaliers: Even
What Ballard didn't understand, of course, is that odds are equivalent to probabilities (this is the same point John Allen Paulos made in his June 28, 2001, column on William Safire's mistaken odds that different Democrats would be nominated for President in 2004. The odds above are equivalent to the following probabilities:
Miami Heat: .091
Memphis Grizzlies: .111
Atlanta Hawks:
.125
Los Angeles Clippers: .167
Denver Nuggets: .222
Toronto
Raptors: .25
Cleveland Cavaliers: .5
This gives a total probability that one of these teams finishes last
of
1.466, which is of course impossible. I sent a letter to SI about
this,
but it wasn't printed.
Best,
Jeff
==============================================================================
Jeffrey S. Simonoff Phone: (212) 998-0452
Professor of
Statistics
Leonard N. Stern School of Business
http://www.amstat.org/pressroom/roxborough.pdf
Oberserved Feb 28 2003
March 3-9: 2 to 1
March 10-16: 3 to 2
March 17-23: 7 to 4
March 24-30: 5 to 1
March 31 or after (but before Dec. 31, 2003): 3 to 2
Time frame in which the US will begin its attack of Iraq (an attack requires minimum 1000 military units)
observed March 15, 2003
03/10/03 - 03/16/03 +2000
03/17/03 - 03/23/03 EV
03/24/03 - 03/30/03 +120
03/31/03 - 04/06/03 +200
04/07/03 or later (but before 12/31/03) +150
Copyright (c) 2003 Laurie Snell
This work is freely redistributable under the terms of the \GNU General Public License published by the Free Software Foundation. This work comes with ABSOLUTELY NO WARRANTY.