"

 COPYRIGHT (c) 1989 by Claus Gittinger

              All Rights Reserved

 This software is furnished under a license and may be used

 only in accordance with the terms of that license and with the

 inclusion of the above copyright notice.   This software may not

 be provided or otherwise made available to, or used by, any

 other person.  No title to or ownership of the software is

 hereby transferred.

"

"{ Package: 'stx:libcomp' }"

"{ NameSpace: Smalltalk }"

Object subclass:#Scanner

	instanceVariableNames:'typeArray actionArray source lineNr token tokenType tokenPosition

		tokenValue tokenName tokenLineNr tokenLastEndPosition hereChar

		peekChar peekChar2 requestor exitBlock errorFlag ignoreErrors

		ignoreWarnings saveComments currentComments collectedSource

		scanColonAsKeyword outStream outCol inArrayLiteral lastDirective

		parserFlags didWarnAboutSTXSpecialComment

		didWarnAboutUnderscoreInIdentifier didWarnAboutOldStyleAssignment

		didWarnAboutDollarInIdentifier didWarnAboutPeriodInSymbol

		unicodeActions'

	classVariableNames:'DefaultActionArray DefaultTypeArray DefaultUnicodeActions

		EmptySourceNotificationSignal'

	poolDictionaries:''

	category:'System-Compiler'

!

Scanner class instanceVariableNames:'TypeArray ActionArray UnicodeActions'

"

 No other class instance variables are inherited by this class.

"

!

Object subclass:#Comment

	instanceVariableNames:'commentType commentString startPosition endPosition'

	classVariableNames:''

	poolDictionaries:''

	privateIn:Scanner

!

Object subclass:#Directive

	instanceVariableNames:''

	classVariableNames:''

	poolDictionaries:''

	privateIn:Scanner

!

Scanner::Directive subclass:#ClassDirective

	instanceVariableNames:'className'

	classVariableNames:''

	poolDictionaries:''

	privateIn:Scanner::Directive

!

Scanner::Directive::ClassDirective subclass:#ClassHintDirective

	instanceVariableNames:''

	classVariableNames:''

	poolDictionaries:''

	privateIn:Scanner::Directive

!

!Scanner class methodsFor:'documentation'!

bugs

"

   array constant containing keywords as in:

        #(

                foo:bar:

                fee:baz:

         )

   is scanned as 4-element array containing ( #foo: #bar: #fee: #baz: )

   this MUST be fixed.

   workaround:

        #(

                #'foo:bar:'

                #'fee:baz:'

         )

"

!

copyright

"

 COPYRIGHT (c) 1989 by Claus Gittinger

              All Rights Reserved

 This software is furnished under a license and may be used

 only in accordance with the terms of that license and with the

 inclusion of the above copyright notice.   This software may not

 be provided or otherwise made available to, or used by, any

 other person.  No title to or ownership of the software is

 hereby transferred.

"

!

documentation

"

    Scanner reads from a stream and returns individual smalltalk tokens

    Its main method is #nextToken, which reads and returns the next token

    Possibly placing additional information (such as tokenValue) into

    instance variables as a side effect.

    TODO:

        some testers noticed that ST-80's scanner methods are called

        xLetter, xDigit etc. For code using these (internals), the nextNumber,

        nextIdentifier etc. methods should be renamed.

        (to me, these seem to be internal private methods;

         their public use is not a good idea ..)

        Scanner is typically subclassed for parsing and #nextToken

        invoked via #self-sends.

        This should be changed and scanner ought to be an instance variable

        of Parser - this allows more flexible use of the scanner/parser

        framework (i.e. changing the scanner without affecting the parser).

    Extensions:

        this scanner allows for 3-character binary selectors.

        also, # is a valid selector. (however, ## is currently scanned as a symbol literal).

    [author:]

        Claus Gittinger

    [see also:]

        Parser

"

! !

!Scanner class methodsFor:'initialization'!

binarySelectorCharacters

    "return a collection of characters which are allowed in binary selectors"

    ^ Method binarySelectorCharacters.

!

extendedBinarySelectorCharacters

    "return a collection of characters which are optionally allowed in binary selectors"

    ^ String

        with:(Character value:16rB1)  "/ plus-minus

        with:(Character value:16rD7)  "/ times

        with:(Character value:16rB7)  "/ centered dot

        with:(Character value:16rF7)  "/ divide

        with:(Character value:16rAB)  "/ <<

        with:(Character value:16rBB). "/ >>

    "Modified (comment): / 17-11-2016 / 09:22:42 / cg"

!

setupActions

    "initialize the scanner's actionTables - these are used to dispatch

     into scanner methods as characters are read.

     Compatibility note: in previous versions, these tables used to be kept

     in classVariables, which made reuse hard as subclasses had no easy way of

     defining their own tables. These are now class instance variables."

    |block actionArray typeArray unicodeActions unicodeTypes|

    actionArray := Array new:256.

    typeArray := Array new:256.

    unicodeActions := Dictionary new.

    unicodeTypes := Dictionary new.

    "/ TODO: later versions should be configurable w.r.t separators.

    "/ #(9 10 12 13 26 32) do: [:i | TypeArray at:(i+1) put: #separator].

    block := [:s :char | s nextNumber].

    actionArray from:($0 codePoint) to:($9 codePoint) put:block.

    block := [:s :char | s nextSpecial].

    self binarySelectorCharacters do:[:binop |

        |codePoint|

        codePoint := binop codePoint.

        codePoint <= typeArray size ifTrue:[

            typeArray at:codePoint put:#special.

            actionArray at:codePoint put:block

        ] ifFalse:[

            unicodeTypes at:codePoint put:#extendedSpecial.

            unicodeActions at:codePoint put:block

        ].

    ].

    block := [:s :char | s nextExtendedSpecial:char].

    self extendedBinarySelectorCharacters do:[:binop |

        |codePoint|

        codePoint := binop codePoint.

        codePoint <= typeArray size ifTrue:[

            typeArray at:codePoint put:#extendedSpecial.

            actionArray at:codePoint put:block

        ] ifFalse:[

            unicodeTypes at:codePoint put:#extendedSpecial.

            unicodeActions at:codePoint put:block

        ].

    ].

    "/ that one is a special case (both binarySelector AND syntax).

    typeArray at:($| codePoint) put:nil.

    block := [:s :char | s nextToken:char].

    ';.^|()[]{}' do:[:ch |

        actionArray at:(ch codePoint) put:block

    ].

    block := [:s :char | s nextIdentifier].

    actionArray from:($a codePoint) to:($z codePoint) put:block.

    actionArray from:($A codePoint) to:($Z codePoint) put:block.

    "kludge: action is characterToken, but type is special"

    typeArray at:($| codePoint) put:#special.

    "kludge: action is nextColonOrAssign, but type is special"

    typeArray at:($: codePoint) put:#special.

    actionArray at:($' codePoint) put:[:s :char | s nextString:char escapeStyle:nil].

    actionArray at:($$ codePoint) put:[:s :char | s nextCharacter].

    actionArray at:($# codePoint) put:[:s :char | s nextHash].

    actionArray at:($!! codePoint) put:[:s :char | s nextExcla].

    actionArray at:($% codePoint) put:[:s :char | s nextPrimitive].

    actionArray at:($: codePoint) put:[:s :char | s nextColonOrAssign].

    actionArray at:($_ codePoint) put:[:s :char | s nextUnderline].

    "/ an experiment to allow 'special' identifiers (in backticks, allowing arbitrary characters inside)

    "/ actionArray at:($` codePoint) put:[:s :char | s nextBacktickIdentifier].

    unicodeActions at:2190 "<- left arrow" put:[:s :char | s nextAssignmentArrow].

    ActionArray := DefaultActionArray := actionArray.

    TypeArray := DefaultTypeArray := typeArray.

    UnicodeActions := DefaultUnicodeActions := unicodeActions.

    "

     Scanner setupActions

     Scanner withAllSubclassesDo:[:cls | cls setupActions ]

    "

    "Modified: / 02-07-2017 / 01:11:27 / cg"

    "Modified: / 08-02-2019 / 19:13:32 / Claus Gittinger"

! !

!Scanner class methodsFor:'instance creation'!

for:aStringOrStream

    "create & return a new scanner reading from aStringOrStream"

    ^ self basicNew initializeFor:aStringOrStream

    "Modified: 23.5.1997 / 12:08:42 / cg"

!

new

    "create & return a new scanner"

    ^ self basicNew initialize.

    "Modified: / 23.5.1997 / 12:08:42 / cg"

    "Created: / 26.5.1999 / 12:02:16 / stefan"

! !

!Scanner class methodsFor:'Signal constants'!

doNotShowCompilerWarningAgainActionQuery

    ^ DoNotShowCompilerWarningAgainActionQuery

!

emptySourceNotificationSignal

    ^ EmptySourceNotificationSignal

    "Created: / 16.5.1998 / 15:55:14 / cg"

! !

!Scanner class methodsFor:'accessing'!

actionArray

    ActionArray isNil ifTrue:[

        self setupActions

    ].

    ^ ActionArray ? DefaultActionArray

!

flushActionArray

    ActionArray := DefaultActionArray := nil.

    TypeArray := DefaultTypeArray := nil.

    UnicodeActions := DefaultUnicodeActions := nil.

    "

     Scanner flushActionArray

     Parser flushActionArray

     ByteCodeCompiler flushActionArray

     Explainer flushActionArray

    "

    "Modified: / 11-12-2018 / 11:25:53 / Claus Gittinger"

!

typeArray

    TypeArray isNil ifTrue:[

        self setupActions

    ].

    ^ TypeArray ? DefaultTypeArray

!

unicodeActions

    UnicodeActions isNil ifTrue:[

        self setupActions

    ].

    ^ UnicodeActions ? DefaultUnicodeActions

    "Created: / 25-03-2011 / 13:57:34 / cg"

! !

!Scanner class methodsFor:'class initialization'!

initialize

    "initialize the classes defaults. Typically, these are changed

     later in the 'private.rc' file."

    EmptySourceNotificationSignal isNil ifTrue:[

        EmptySourceNotificationSignal := QuerySignal new mayProceed:true.

        EmptySourceNotificationSignal notifierString:'empty source given to evaluate'.

        EmptySourceNotificationSignal nameClass:self message:#emptySourceNotificationSignal.

    ].

    "

     self initialize

    "

    "Modified: / 16.5.1998 / 15:55:41 / cg"

! !

!Scanner class methodsFor:'defaults'!

allowDollarInIdentifier

    <resource: #obsolete>

    "return true, if $-characters are allowed in identifiers.

     Notice, that dollars are NEVER allowed as the first character in an identifier."

    ^ ParserFlags allowDollarInIdentifier

    "Created: 7.9.1997 / 01:32:18 / cg"

    "Modified: 7.9.1997 / 01:39:44 / cg"

!

allowDollarInIdentifier:aBoolean

    "this allows turning on/off $-characters in identifiers.

     Notice, that dollars are NEVER allowed as the first character in an identifier.

     If turned off (the default), dollars are not allowed in identifiers,

     but instead are scanned as character-constant prefix.

     If turned on, dollars are in identifiers are allowed, while extra

     dollars are still scanned as constant character prefix.

     If you have to fileIn old VW-Vsn2.x classes, turn this off

     before filing them in; i.e.:

        Compiler allowDollarInIdentifiers:false"

    ParserFlags allowDollarInIdentifier:aBoolean.

    "Created: 7.9.1997 / 01:34:49 / cg"

    "Modified: 7.9.1997 / 01:39:30 / cg"

!

allowDolphinExtensions

    "return true, if ##(..) computed literal Arrays are allowed"

    ^ ParserFlags allowDolphinExtensions

!

allowDolphinExtensions:aBoolean

    "this allows turning on/off support for computed literal Arrays ##(..) as in dolphin.

     If you want to fileIn Dolphin classes, enable this with:

        Compiler allowDolphinComputedArrays:true"

    ParserFlags allowDolphinExtensions:aBoolean.

    "

     self allowDolphinExtensions:true

     self allowDolphinExtensions:false

    "

!

allowLiteralNameSpaceSymbols

    <resource: #obsolete>

    "return true, if literal nameSpace symbols are allowed (#foo::bar) are allowed"

    ^ ParserFlags allowLiteralNameSpaceSymbols

!

allowOldStyleAssignment

    "return true, if underscore-assignment (pre ST-80v4 syntax) are to be allowed"

    ^ ParserFlags allowOldStyleAssignment

!

allowOldStyleAssignment:aBoolean

    "this allows turning on/off recognition of underscore-assignment (pre ST-80v4 syntax).

     You must turn this off, if code with variables named '_' is to be filedIn"

    ParserFlags allowOldStyleAssignment:aBoolean

!

allowQualifiedNames

    <resource: #obsolete>

    "return true, if '#{..}' and 'namespace.varName' qualified names are allowed"

    ^ ParserFlags allowQualifiedNames

    "Modified (comment): / 23-09-2018 / 01:08:26 / Claus Gittinger"

!

allowQualifiedNames:aBoolean

    "this allows turning on/off support for qualifiedNames #{ .., } as in vw3.

     If you want to fileIn vw3 or later classes, enable this with:

        Compiler allowQualifiedNames:true

     Notice, that qualified names are not really supported semantically

     (they are parsed, but treated like regular globals)

    "

    ParserFlags allowQualifiedNames:aBoolean.

    "

     self allowQualifiedNames:true

     self allowQualifiedNames:false

    "

!

allowSqueakExtensions

    <resource: #obsolete>

    "return true, if support for squeak extensions

        computed arrays { .., }

        c/java style arguments in message sends rec foo(arg1, ... argN)

     is enabled."

    ^ ParserFlags allowSqueakExtensions

!

allowSqueakExtensions:aBoolean

    "this allows turning on/off support for squeak extensions:

        computed arrays { .., }

        c/java style arguments in message sends rec foo(arg1, ... argN)

     If you want to fileIn Squeak classes, enable this with:

        Compiler allowSqueakComputedArrays:true"

    ParserFlags allowSqueakExtensions:aBoolean.

    "

     self allowSqueakExtensions:true

     self allowSqueakExtensions:false

    "

!

allowUnderscoreInIdentifier

    <resource: #obsolete>

    "return true, if underscores are allowed in identifiers"

    ^ ParserFlags allowUnderscoreInIdentifier

!

allowUnderscoreInIdentifier:aBoolean

    "this allows turning on/off underscores in identifiers.

     If turned off (the default), underscores are not allowed in identifiers,

     but instead scanned as assignment character (old ST/80 syntax).

     If turned on, underscores are in identifiers are allowed, while extra

     underscores are still scanned as assignment.

     If you have to fileIn old VW-Vsn2.x classes,

     turn them off with:

        Compiler allowUnderscoreInIdentifiers:false"

    ParserFlags allowUnderscoreInIdentifier:aBoolean.

    "Modified: 7.9.1997 / 01:35:19 / cg"

!

maxBinarySelectorSize

    ^ Method maxBinarySelectorSize

!