Token Counting – Programming Languages

Token Counting – Programming Languages


So given that we’ve just seen how to code up HTML comments, let’s test out that knowledge. Suppose we have our definition of HTML comments plus a rule for word tokens that are anything that’s not a space, a left angle, a right angle, one or more of the above. We return that token, and now I give you the following HTML input fragment, “ob fuscation tuse tangle.” For this blue string, for our lexer, we’ve got word tokens. We’ve got HTML comments. How many word tokens are we going to find?

Leave a Reply

Your email address will not be published. Required fields are marked *