CHANCE News 12.02
March 1, 2003 to May 1, 2003

Prepared by J. Laurie Snell, Bill Peterson, Jeanne Albert, and Charles Grinstead, with help from Fuxing Hou and Joan Snell.

We are now using a listserv to send out notices that a new Chance News has been posted on the Chance website. You can sign on or off or change your address at this Chance listserv. This listserv is used only for this posting and not for comments on Chance News. We do appreciate comments and suggestions for new articles. Please send these to jlsnell@dartmouth.edu.

The current and previous issues of Chance News and other materials for teaching a Chance course are available from the Chance website.

Chance News is distributed under the GNU General Public License (so-called 'copyleft'). See the end of the newsletter for details.

War is...a trinity of violence, chance, and reason.

Carl Von Clausewitz
On War

Contents of Chance News 12.02

1. Forsooth.

2.When should probabilities determined from odds add to 1?

3. What's different about this Chance News?

4. Counting the number of protesters.

5. Betting on the war.

6. War makes stock buyers ignore the fundamentals.

7. Comes war, bet on stocks of companies with military contracts.

8. Graphics in the New York Times .

9. Graphics in the Atlantic Monthly.

10. A linguistic curiosity.

Here are Forsooth items from the February and March 2003 R.S.S News.

In response to KM's questioning of whether a loo seat should be left up or down, I'd say it's all to do with probability. The next person to use a lavatory could be a woman (probability 1:2) or a man (same probability). If a woman she will have the seat down (probability 1:1) but a man is equally likely to have the seat up (1:2) or down (likewise). So as the probability that the next user will require the seat up is only 1:4, each user should leave the seat down.

Letter to Times Magazine
21 Dec 2002

Sir, You report (November 8) that 98 per cent of convictions for dangerous driving were of men drivers. This means that men are 49 times more likely than women to be convicted of dangerous driving.

From a letter to The Times
15 November 2002

1 in 50 people have an IQ in the top 2% that makes them eligible to join Mensa.

MENSA website (2002)

Jeff Norman suggested the following forsooth item:

About 20 percent of Turkish earthquakes
take place between 3 and 4 a.m.

CNN Turk

Here's a histogram based on data from NOAA's National Geophysical Data Center showing the number of eathquakes in each hour (Greenwich Mean Time) for significant earthquakes in Turkey since 1900.

A Chi Square goodness of fit tests shows that you cannot reject the hypothesis of a uniform distributions.

Low blood lead concentration may damage IQ.

Headline for an article in the Asian News International
Washington, April 17, 2003

Jeff Simonoff sent us the following comments relating to an item in the February 10, 2003 issue of Sports Illustrated:

In a small item named "Lining Up for LeBron" (on page 37), author Chris Ballard gives his odds that each of seven NBA teams will finish last in the NBA during the season, thereby giving them the best shot at Ohio schoolboy star LeBron James in the NBA Draft Lottery. The odds given are as follows:

 

Team	Odds	Probability
Miami Heat	10 to 1	.091
Memphis Grizzlies	8 to 1	.111
Atlanta Hawks	7 to 1	.125
Los Angeles Clippers	5 to 1	.167
Denver Nuggets	7 to 2	.222
Toronto Raptors	3 to 1	.25
Cleveland Cavaliers	Even	.5
Total		1.466

What Ballard didn't understand, of course, is that odds are equivalent to probabilities (this is the same point John Allen Paulos made in his June 28, 2001, column on William Safire's mistaken odds that different Democrats would be nominated for President in 2004. When we determine these probabilities from the odds we find a total probability that one of these teams finishes last of 1.466, which is of course impossible. I sent a letter to SI about this, but it wasn't printed.

Of course, Jeff is correct in this context, but if these were odds given by Roxy for Las Vegas betting, the corresponding probabilities would also add up to more than one since that is how bookies make money. For example, here are the odds (as of 1 May 2003) from Roxy's Americasline.com for betting on who will be elected president in 2004:

ODDS TO BE ELECTED PRESIDENT OF THE UNITED STATES IN 2004

Converting these odds to probabilities we found that they sum to 1.2. Those who set odds for betting usually assume that the proportion of money bet on an outcome is a good estimate for the probability that the outcome will occur. Assuming this is the case, then the probabilities would add up to 1. If these probabilities were used to create the odds, then the expected profit would be $0.

On the other hand, if we multiply the probabilities by 1.2, making their sum 1.2, and use these to determine the odds, the expected profit will be about 17% of the money bet which is about what bookies want.

We found it interesting that, during the Iraq war, even the chance news items often involved issues related to the war. We decided to limit this issue of Chance News primarily to such items. However, we remind readers that the monthly electronic newsletter STATS also discusses current chance items. At their website you can also request to receive an e-mail message with links to important recent articles when the next issue comes out.

We also remind readers of John Paulos' monthly column for ABCNews.com. His current column "Beauty is in the Wallet
of the Beholder" relates the stock market, beauty contests, and a simple game. Previous columns are archived on the ABCnews website. The March and April columns, Assessing Military Edge, and Polls, a Proverb and the Price were motivated by the Iraq war.

Our first war related story was suggested by Mark Rizzardi.

http://www.sfgate.com/rallycount/.
San Francisco Chronicle, 21 Feb. 2003, A1
Wyatt Buchanan

The crowd count conumdrum.
San Francisco Bay Area Independent Media Center, 1 March, 2003.
Peter Maiden

40,000? 250,000?
Making crowd estimates a mix of guesswork, science, politics.
San Jose Mercury News, 28 Jan. 2003, Science & Health 1F
Isa M. Krieger

The problem of estimating the number of people in a crowed has often led to controversies. For example, as reported in Chance News 4.17, for the 1995 Louis Farrakhan's Million Man March on Washington, the Park Service estimated the crowd at 400,000 while an analysis by the Boston University's Center for Remote Sensing, based on computer enhancements of photographs, estimated the crowd at 870,000 with a margin of error of 25%. This led to the threat of a law suit by Farrakhan, and to the Park Service giving up estimating crowd sizes.

The Chronicle article discusses the problems encountered estimating crowd size at a rally protesting a war with Iraq held in San Francisco on Sunday Feb 16. There had been two previous rallys, October 26, 2002 and January 18, 2003 in the same area. In Peter Maiden's article we find the following history of the estimates for these previous marches.

In the October rally. the organizers estimated there were 100,000 protesters. The police estimated the number to be 25-40,000 and this is what was presented in the media. This seemed like too big a discrepancy and so the organization ANSWER hired a helicopter to take pictures at the next rally. This occurred on January 18th and Peter Maiden took the photos which were then posted on the Indymedia website. For this rally, the police first estimated 55,000 protesters. A chronicle reporter showed the police the photographs from the Indymedia website. As a result they increased their estimate to 150,000 with the excuse that they had estimated the count for those at the City Plaza but not for those on Market street. In her article, Krieger described two more "mathematical" methods of estimation that were carried out during this rally as follows:

Method 1: Assign someone to a spot on the sidewalk with the job of counting the number of marchers who pass in one minute. Then multiply this sum by the number of minutes it takes for all marchers to pass that spot. This was the approach taken by ANSWER, the anti-war march organizer, said spokesman Bill Hackman. He says they counted 1,000 people a minute for 180 minutes-- for a total of 180,000 marchers.

Method 2: Measure the square meters of the area, then divide by crowd density. This was the method used by Chandler, who has degrees in physics and math. He went home after the march and used aerial images to calculate that Market Street is 34 meters wide and the parade route was 2.7 kilometers long. The total area of the parade route, not counting Civic Center Plaza, was 92,000 square meters, he concluded. He estimated that each square meter of space is occupied by 1.5 people. So he concluded that the total number of marchers on Market Street, when filled, was 138,000. He said that marchers filled the parade route twice so he doubled this number concluding that there were more than 250,000 marchers.

In February, ANSWERS and others planned another San Francisco rally which took place February 16 along the same route as the previous marches: down Market Street ending at the Civic Center Plaza as indicated in the following graph provided on the Chronicle's website.

One can get a good idea how difficult the estimation problem is from the following picture taken by Peter Maiden of the crowd just in the Civic Center Plaza

and this picture taken by Darryl Bush, taken for the Chronicle, of the march on Market street.

Remarkably, after this march the organizers, the police and the mainstream media agreed on a crowd count of about 200,000 people.

Anticipating that the estimates would be made by observational methods they felt to be unreliable, the Chronicle had arranged to have a more scientific estimate for this march. They hired the Air Flight Service, a company with a great deal of experience in taking photographs for topographical maps, to take aerial photographs directly above Market Street and the Civic Center. The photographs made possible a discrete count of individuals and provided a view of the spaces between them. On its website, the Chronicle describes the methodology used to estimate the crowd size from the photographs as follows:

The aerial photographs of Sunday's anti-war march were taken at a time believed by organizers to be close to the peak of the march. The plane made a pass over the length of Market Street and the Civic Center area, taking contiguous photographs. The lens in the camera looked directly down from 2,000 feet.

Air Flight Service produced 9-by-9-inch black and white negatives, which were enlarged into prints with enough clarity to define individuals on the street. They then applied a grid pattern to the prints to provide units in which the marchers could be counted. The entire length of the 1.75-mile route is visible in the images. Because the density of the crowd varied over the route, the grids were sorted into five categories - 100 percent, 75 percent, 50 percent, 25 percent and 10 percent filled with people. This was done by visual estimates. Each of the nearly 300 grids was given a colored dot representing these percentages.

At this point, the hand counting began. A representative sample of grids for each density was selected. For instance, eight 50 percent grids were counted, eight 25 percent grids were counted and so forth. The totals for each group were examined to ensure there were not extreme deviations. This tested the original visual estimate that had assigned grids to density groups. If necessary, a sample grid's category was changed to reflect the count.

Here is a grid for which that was estimated to be 75% full.

An average number of people for each density grid was established using the sample groups. Air Flight Service then tallied the grids to come up with a total of nearly 65,000 people visible from the air at the moment the photographs were shot. The company said the margin of error is 10 percent.

The Chronicle article comments:

Calculating a precise number of protesters for the entire rally is not possible from this survey, but the result is much more accurate than the visual scan method most commonly used by police and organizers.

The Chronicle emphasized that their count was only of the number in the protest at the peak time but did not count those who entered before or left after this time. However, they said that their count of 65,000, significantly lower than the 200,000 estimate, was supported by information about public transportation. For example the Bay Area Rapid Transit System determined that 66,250 exited at the four downtown stations closest to the rally on Sunday between 8 a.m. and midnight, around 43,000 more than the number of downtown passengers last Sunday.

Of course the number of protestors is an important political statement, so one might hope that the media would try to get it right. However, before the Chronicle analysis was reported, the crowd size was routinely reported as 200,000 with no reference to any justification for this number. Then, despite the care the Chronicle took to state that they counted only the number of protestors present at a particular time, when a UPI story was reported in the Washington Times on Feb. 19, we read:

The San Francisco Chronicle said high-resolution aerial photographs commissioned by the newspaper showed that around 65,000 people took part in last Sunday's march through downtown rather than the 200,000 announced originally.

They conveniently left out a statement in the UPI story that this was an estimate for the number of protesters at a specific time. Ironically Debra Sauders, who writes a column for the Chronicle, began her February 26th column "Does size matter?" with:

ANTI-WAR protesters bristled when The Chronicle reported last week that a mere 65,000 protesters showed up at San Francisco's Feb. 16 anti-war protest.

requiring the Chronicle to make a correction stating that they had only counted protestors who were there at a specific time.

DISCUSSION QUESTIONS:

(1) What's missing from the Chronicle estimate is the percentage of protesters who are present at the peak time of the march. Do you have any suggestions on how this percentage might be estimated?

(2) What do you think of the method used by the Air Flight Service? Can you suggest any improvements?

(3) Alex S. Jones, the director of Harvard University's Joan Shorenstein Center on the Press, Politics and Public Policy, and former reporter for the New York Times, said:

There's an old saying in journalism: People only see what they believe.
This is an emotional issue, not a factual issue as far as most people are concerned.

Why would the police like to see small numbers? Do you think the news media might also have a bias for low or high estimates?

Decisions, Decisions.
The New Yorker, 24 March, 2003, p 33
James Surovie

Betting on Hussein's fate is heavy-cyberspace wagering seen as good predictor.
Boston Globe, 11 March 2003, B1
Farah Stockman

You can bet on it: playing the odds on war.
New York Times, 2 March 2003, Section 4, p 2
Sam Lubell

Here the probabilities add up to 1.664 so the bookies hope to make about 60% of the money bet. Note that the March 2 odds suggested that the most likely time for the start of the war would be during the week of March 17-23: the war started March 20. If the the odds were determined by the bettors' wagers, their intuition was pretty good.

These articles report that predictions based on gambling are beginning to be considered seriously and to compete favorably with more traditional methods for making predictions. A well-known example of this is the Iowa Election Markets. Here, for example, you buy and sell "futures contracts " associated with various political events. There is a market for each event and for each market you can buy contracts corresponding to different possible outcomes of the event. This is an educational project and bettors use real money but can have at most $500 in their opening account.

In the Iowa Election Markets, at this time, one of the markets, related to the outcome of the Democratic convention, allows you can buy and sell contracts on Hillary Clinton, Dick Gephardt, John Kerry, Joe Lieberman and "the rest of the field" (ROF): The market will end when the Democratic convention makes its decision and contracts for the person chosen will then be paid off at $1 per contract and the other contracts will be worthless. While Hillary Clinton is not a candidate at this time, she was added as a result of recent polls. Other candidates may be added depending on the results of future polls. Here is the state of this market on May 6th.

(1) Note that by the Iowa market Hillary Clinton is more likely than any of the other three candidates to be nominated, but by Roxy's odds is less likely than any of these three to be the next president. Can you explain this?

(2) TradeSports also has a market relating to Bush being the next president. This market suggests that Bush has a 66% chance of being president. A similar market on the Iowa electronic market estimates this probability to be about 61%. The latest Gallup/CNN (February 24-26) asked:

If George W. Bush runs for re-election in 2004, in general are you more likely to vote for Bush or for the Democratic party's candidate for president?

47% of the registered voters said they would vote for Bush and 39% said they would vote for the democratic candidate. Can you explain these differences? Which do you think provides the best estimate?

(3) Pollsters say that these markets are not to be trusted because they are not a representative sample and do not provide a margin of error. Do you agree?

References:

(1) David M. Pennock, Finn Årup Nielsen, C. Lee Giles." Extracting Collective Probabilistic Forecasts from Web Games".
Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2001), pp. 174-183, ACM, New York, 2001

(2) JE Berg, R Forsythe, FD Nelson and TA Rietz, 2001, “ Results from a Dozen Years of Election Futures Markets Research,” forthcoming in CA Plott and V Smith (Eds) Handbook of Experimental Economic Resuts.

(3) Robert Matthews, "Ask the oracle, " New Scientist, March 23, 2002.

Lone shares mirror mood swings of market.
Wall Street Journal, 31 March, 2003. C1
Ken Brown

This article is based on the a graph with the following description:

When global political or market events frighten investors, they tend to see the stock market as one big entity where individual stocks move up or down together. Below, the average six-month correlation between the performance of the average individual S&P-500 stocks and the index itself over the past 13 years is plotted. The higher the percentage of correlation, the more individual stocks move in unison with the overall market.

We wondered how many readers would understand how the graph was made. Then we realized that we ourselves did not understand it. We consulted Credit Suisse First Boston and learned that the data for the graph was prepared by Mika Toikka. Toikka told us that the article got it wrong. In particular, the index itself was not involved in making the graph. He explained that the graph was constructed as follows:

For a given day, for each pair of the 500 stocks in the S&P-500, the correlation between the pair's last six months daily returns (percentage increases or decreases) was computed. The average of these correlations over all possible pairs for the given day was calculated. These averages then were plotted for the thirteen-year period.

When the averages are high the returns of the individual stocks are highly correlated, so they would tend to follow the S&P index. The point of the graph is to show that this appears to occur at times of wars and other crises, suggesting that, at these times, buyers and sellers are not looking at the "fundamentals" of the stocks but rather are just following the market.

Referring to the period during the Iraq war, Toikka is quoted as saying:

It's been a difficult environment to pick up outperformance by picking stocks. Everything has been moving together so i'ts difficult to pick out home runs.

DISCUSSION QUESTIONS:

(1) Does this say that the market is not an "efficient" market during a war, and so we should be able to make money trading during a war?

(2) How do you think stock experts "pick out the home runs"?

Speaking of stocks, we neglected to report that the famous Wall Street Journal contest between the darts and the experts ended with the six-month period from March 14, 2002 to August 30, 2002. The contest had 147 six-month periods. The score of the experts vs darts was 90 to 57 in favor of the experts. For the experts vs the Dow Industrials the score was 80 to 67 in their favor.

If the winners were the result of tossing a coin 147 times, the expected number of wins for the experts would would be 73.5 with standard deviation 6.06. Thus the experts' 90 wins against the darts, was 16.5 greater than expected by chance. This is a deviation of more than two standard deviations but less than 3, making the results significant at the 95% level. But the experts' 80 wins against the Dow Industrials was only 6.5 more than expected by chance, which is about one standard deviation and hence not significant.

The dart contest still goes on but without the experts. Now it is the darts against the readers. The Journal compares the six month performance of six randomly chosen readers' stocks with six stocks chosen by throwing darts. In addition to this contest, the reader whose stock does the best gets a Wall Street Journal tote bag.

The winner of the tote bag for the six-month period ending March 31 was Paul Nelson who chose ViaSat Inc. to win. This is a digital satellite communication company whose clients include the U.S. military. Its stock rose 80% over this six-month period. Here is a comparison of the progress of this stock over this six-month period compared to the performance of the Dow.

The August 2002 meeting of the American Statistical Association had an interesting pair of talks on how graphics are done at the New York Times. The speakers were Charles Blow, the Graphics Director, and Arohie Tse, the Graphics Editor at the Times. The Times graphics are prepared by a group of about 30 people, mostly quite young and from a wide variety of backgrounds (none are statisticians) who work together trying out ideas on each other. They also consult statisticians and other researchers. They typically have at most three days to get the data and to make the graphic. Blow wrote in his abstract:

Information graphics designers in print journalism are presented with a wide variety of data types and formats, which often can't be captured by any standard visualization technique. They need to respond with eye-catching graphics that communicate information clearly and without distortion to a diverse readership--and they need to be ready before the story becomes stale.

Tse gave examples of graphics projects that he was responsible for, including the many graphics that accompanied the analysis of the Times's account of the Florida presidential ballots and the graphics associated with the release of the 2000 Census data. He compared graphs before and after color was introduced in the Times to show that much more information could now be included in graphics by using color. He said this was particularly true when reporting results of surveys, political polls and weather data.

While examples of Times graphics have won high praise from the experts including Tufte, they do not always avoid the well- known graphics pitfalls. Howard Wainer illustrated this by several examples of recent graphics from the Times when he introduced the speakers. In his wonderful Chance Lecture "How to display data badly" Wainer remarked that he only picked on the big guys, so most of his examples of bad graphics were taken from major newspapers.

Here is another interesting source of graphics in the media.

The Atlantic Monthly has a regular feature called "The Nation in Numbers" or when appropriate, "The World in Numbers". It consists of a two-page, color graphics display, along with data that is often organized around a map of the United States (or the world). There is also a brief companion text and some related tables or graphs. Here is a brief description of the topics and displays from the past few months.

April - Abortion Decisions: How the advent of prenatal testing has complicated the moral calculus of abortion worldwide

A world map indicates by country both abortion legal status and its annual rates per 1000 women aged 15--44 years, and there is a brief discussion of whether and how these variables might be connected. One table displays the percentages for select countries, of people and geneticists who would abort in the first trimester for certain conditions; another gives the percentage for select countries of geneticists who would test prenatally or provide a referral for a couple with four girls who would abort another girl.

March - Mortal Justice: the demography of the death penalty,

A map of the U.S. displays state-by-state information about the death penalty, such as execution rate, states with a moratorium on the death penalty, and the number of death-row inmates exonerated; nationwide totals are also provided.

One graph examines some counties in Texas where the execution rate is especially high; another compares execution and murder rates by the race of the offender and victim. The companion text notes that the race of the victim appears to effect the execution rate more than the race of the offender.

January/February - Does money buy happiness?

A scatter plot of smiling, neutral, or sad faces, labeled by country, is shown where the horizontal axis represents income per person and the vertical represents average life satisfaction. A companion graph displays materialism and happiness indices for Americans; another charts money versus love as predictors of happiness.

Tom Allaway sent us a note about a linguistic curiosity he discovered from the Brown Corpus. In linguistics"Corpus" means a collection of words. The Brown Corpus consists of about 1,000,000 words obtained by samples of size about 2000 taken from a collection of 500 texts. These 500 texts were chosen from 15 different categories of English texts published in the US in 1961 including newspaper articles, government reports, different categories of books etc.

Tom found at the virtual language centre a list of the 2000 most frequent words in the Brown Corpus and noted that the ranking of the words for the integers from 1 to 9 increased, i.e. the frequency of occurrence decreased. Here is what he found:

word	rank	freq	rank x freq
one	32	3297	105,504
two	69	1412	97,428
three	151	610	92,110
four	260	359	93,340
five	318	286	90,948
six	420	220	92,400
seven	903	114	102,942
eight	1002	104	104,208
nine	1302	81	105,462

The third column, which we included, illustrates Zipf's law which states that the product of the rank and the frequencies of words in a text should be approximately the same for all words. The existence of these large data sets of words has led to new research related to Zipf's law, and we will discuss some of this research in the next Chance news. However, Zipf's law is not relevant to Tom's observation that the frequencies for words representing single digit numbers are monotonically decreasing.

Tom's observation is similar to Benford's law which states that in many different sets of numbers, the frequencies of the leading digits of numbers decreases logarithmically. Since it would seem that most of these words would correspond to the leading digit of a number, it is natural to compare this with the Benford distribution.

To do this we looked at two other data sets of words. The first was the British National Corpus described as follows:

The British National Corpus (BNC) is a 100 million word collection of samples of written and spoken language from a wide range of sources, designed to represent a wide cross-section of current British English, both spoken and written.

The data providing the frequencies of the words is available from Adam Kilgarriff's website . For our second we searched on google for the words one, two, three etc and recorded the number of web pages that turned up. Of course these were huge numbers and related to the number of websites rather than distinct texts.

Since the word "one" has two different meanings we considered only the words for numbers from 2 to 9. We normalized the frequencies so that their sum adds to 1. Here are the results:

We see that the desriptions for the different word data sets are quite simila,r but they are not consistent with the frequencies that would be predicted by the Benford distribution.

(1) Can you suggest an explanation for Tom's observation?

Copyright (c) 2003 Laurie Snell

This work is freely redistributable under the terms of the GNU General Public License published by the Free Software Foundation. This work comes with ABSOLUTELY NO WARRANTY.

CHANCE News 12.02
March 1, 2003 to May 1, 2002