Tokeniser

This little utility takes files of freeform ASCII/ANSI text as input and converts them to tokenised files.

An example:

Input:
At 7 a.m this morning, a cold but bright day, I heard the birds sing, but they didn't notice me.

Output:
At
7
a.m.
this
morning
,
a
cold
but
bright
day
,
I
heard
...
..
did
n't
notice
me
.

It works with multiple files, accepts abbreviations and trailing parts-of-words. See the help-file shipped with it for more details..

------------------------------------------------------------

Disclaimer and licencing

This software is provided AS IS without warranty of any kind, either express or implied. There is no warranty as to merchantable quality or fitness for a particular purpose. Use at your own risk. The software (the exeutable and this helpfile) may be distributed and used freely, but you may not sell it or charge any fee for access to it. Both the executable and this helpfile must be distributed as one whole. All matrial is copyright (c) Hans J. Klarskov Mortensen 2004. Any comment on it will be appreciated. Please contact me at h@hjkm.dk

I accept the terms stated above and want to download the tokeniser.

Once downloaded unzip the package to a directory of your own choice. Double-click the exe-file to run it.