T/Z and Mutual Information

Significance tests calculator.

In phraseological studies it is often important to know whether the collocations of a keyword are statistically significant - i.e.whether the apparent "attraction" between a keyword and another word/a collocate is a matter of chance - or whether it is a "rule". Few pieces of language analysis software can to this, but this wee calculator can.

	Use it this way:
	The upper fields captioned "Keyword", "Collocate" and "Corpus" are only for practical information. Enter the number of tokens (words) in the corpus, and the total number of occurrences of the presumed collocation (here 60152 and 23 respectively) Enter the "span" (i.e. the number of words on each side of the keyword - here 8, namely 4 words to the left of the keyword and 4 words to the right of the keyword). Enter the number of concordance lines (here) 99. Enter the number of occurrences of the presumed collacation in the concordance lines (here 5). Click the "Calc" button, and the results will appear in the Results' fields.

	Once the calculation has been done, the results can be copied to the clipboard, and pasted into any application that accepts text. (See an example below). Ain't got a clue as to the use of this - and still interested? Well, have a look at Jun Da's article which explains it very well indeed.
	A different Mutual Information
	A more theoretical approach is "Using Statistics in Lexical Analysis" by Kenneth Church et al., Bell Laboratories and Oxford University Press. http://www.patrickhanks.com/papers/usingStats.pdf (Viewed March 2004). Hanks et al.'s article does not include information about concordance lines, and concequently it uses a slightly different formula. This calculator can easily be used to calculate Mutual Information this way too if you set span to 1, and Concordance lines to the number of occurrences of the keyword in the corpus. Using Hanks' first example (strong + northerly) the screen should look thus:

	And the textual output from this calculation will read:

STATISTICS FOR :
Keyword: strong
Collocate :northerly
From Corpus :AP
---------------------------------------
FREQUENCIES:
Tokens: 44300000
Corpus occurrences of 'northerly': 28
Span: 1
Concordance lines: 7809
Observed occurrences/Joint frequency: 7
-----------------------------------------
RESULTS:
Expected frequency: 0,004936
Standard deviation: 0,070
SCORES:
T-score: 2,643886
Z-Score: 99,567358
Mutual Information: 10,470
-------------------------------------------

Produced by T/Z and MI-score
No warranty accepted whatsoever.

Copyright (c)Hans J. Klarskov Mortensen, 2003
E-mail : h@hjkm.dk

	Yes, I want to download T/Z-score and Mutual Information. Once downloaded unzip it, read the ReadMe-file and doubleclick the -exe-file.
	If you find the program useful or if you have comments on it, a couple of words would be very much appreciated indeed, and you might even receive an update if it ever surfaces. If you state a good reason, the source code is also available.