A robust Hebrew tokenizer

Splits a chunk of Hebrew text into space tokens. 
   python hebtokenizer.py < input > output
default encoding is utf8. can specify inpunt and output encodings with the --ie and --oe switches, respectively: 
   python hebtokenizer.py --ie cp1255 --oe utf16 < input > output
hebtokenizer.py

requires python 2.5 or above (python3 not supported)