ArgumentTokenizer

This class is responsible for correctly loading argument lists from stdin or files. It handles splitting the input into individual arg tokens, accounting for quotes and whitespace escaping.

Tokenisation mechanism should be as similar to shell as possible - content in single quotes is passed through as-is, content in double quotes allows escaping of " and \ characters via \ character, and \ outside quotes can be used to escape any character whatsoever.