#REFACTORING by exept
class: Smalltalk class
changed: #recursiveInstallAutoloadedClassesFrom:rememberIn:maxLevels:noAutoload:packageTop:showSplashInLevels:
Transcript showCR:(... bindWith:...) -> Transcript showCR:... with:...
"{ Encoding: utf8 }"
"
COPYRIGHT (c) 2004 by eXept Software AG
All Rights Reserved
This software is furnished under a license and may be used
only in accordance with the terms of that license and with the
inclusion of the above copyright notice. This software may not
be provided or otherwise made available to, or used by, any
other person. No title to or ownership of the software is
hereby transferred.
"
"{ Package: 'stx:libbasic' }"
"{ NameSpace: Smalltalk }"
Object subclass:#CharacterEncoder
instanceVariableNames:''
classVariableNames:'AccessLock CachedEncoders EncoderClassesByName EncodersByName
EncodingDetectors Jis7KanjiEscapeSequence
Jis7KanjiOldEscapeSequence Jis7RomanEscapeSequence
JisISO2022EscapeSequence NullEncoderInstance'
poolDictionaries:''
category:'Collections-Text-Encodings'
!
CharacterEncoder subclass:#CompoundEncoder
instanceVariableNames:'decoder encoder'
classVariableNames:''
poolDictionaries:''
privateIn:CharacterEncoder
!
CharacterEncoder subclass:#NullEncoder
instanceVariableNames:''
classVariableNames:''
poolDictionaries:''
privateIn:CharacterEncoder
!
CharacterEncoder subclass:#InverseEncoder
instanceVariableNames:'decoder readAhead'
classVariableNames:''
poolDictionaries:''
privateIn:CharacterEncoder
!
CharacterEncoder::NullEncoder subclass:#DefaultEncoder
instanceVariableNames:''
classVariableNames:''
poolDictionaries:''
privateIn:CharacterEncoder
!
CharacterEncoder subclass:#OtherEncoding
instanceVariableNames:''
classVariableNames:''
poolDictionaries:''
privateIn:CharacterEncoder
!
CharacterEncoder subclass:#TwoStepEncoder
instanceVariableNames:'encoder1 encoder2'
classVariableNames:''
poolDictionaries:''
privateIn:CharacterEncoder
!
!CharacterEncoder class methodsFor:'documentation'!
copyright
"
COPYRIGHT (c) 2004 by eXept Software AG
All Rights Reserved
This software is furnished under a license and may be used
only in accordance with the terms of that license and with the
inclusion of the above copyright notice. This software may not
be provided or otherwise made available to, or used by, any
other person. No title to or ownership of the software is
hereby transferred.
"
!
documentation
"
please read howToAddMoreCoders.
Character mappings are based on information in character maps found at either:
http://std.dkuug.dk/i18n/charmaps
or:
http://www.unicode.org/Public/MAPPINGS
No Warranty.
All the ISO 8859 codesets include ASCII as a proper codeset within them:
ISO-8859-1: Latin 1 - Western European Languages.
ISO-8859-2: Latin 2 - Eastern European Languages.
ISO-8859-3: Latin 3 - Afrikaans, Catalan, Dutch, English, Esperanto, German,
Italian, Maltese, Spanish and Turkish.
ISO-8859-4: Latin 4 - Danish, English, Estonian, Finnish, German, Greenlandic, Lappish and Latvian.
ISO-8859-5: Latin/Cyrillic - Bulgarian, Byelorussian, English, Macedonian, Russian, Serbo-Croat and Ukranian.
ISO-8859-6: Latin/Arabic - Arabic.
ISO-8859-7: Latin/Greek - Greek.
ISO-8859-8: Latin/Hebrew - Hebrew.
ISO-8859-9: Latin 5 - Danish, Dutch, English, Finnish, French, German, Irish, Italian,
Norwegian, Portuguese, Spanish, Swedish and Turkish.
ISO-8859-10: Latin 6 - Danish, English, Estonian, Finnish, German, Greenlandic, Icelandic,
Sami (Lappish), Latvian, Lithuanian, Norwegian, Faroese and Swedish.
[author:]
Claus Gittinger
[see also:]
EncodedStream
Base64Coder
"
!
examples
"
[exBegin]
|s1 s2|
s1 := 'hello'.
s2 := CharacterEncoder encodeString:s1 from:#'iso8859-1' into:#'unicode'.
s2
[exEnd]
[exBegin]
|s1 s2|
s1 := 'hello'.
s2 := CharacterEncoder encodeString:s1 from:#'iso8859-1' into:#'iso8859-7'.
s2
[exEnd]
"
!
howToAddMoreCoders
"
Coders can be hand-written or automagically generated via a mapping table.
Examples for hand-written coders are UTF8_to_ISO10464 or JIS0208_to_JIS7.
The table driven encode/decode methods can be generated from a character mapping document
as found on the unicode consortium host
(for example: 'http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT')
or from the i18n character maps:
(for example: 'http://std.dkuug.dk/i18n/charmaps/ISO-8859-1
In order to add another coder (for example: for EBCDIC or ms-codePage 278),
perform the following steps:
- create a public subclass of CharacterEncoderImplementations::CharacterEncoderImplementation named (for example) CharacterEncoderImplementations::CP267.
- define the mappingURL1_relativeName (if the table is found on 'www.unicode.org')
or the mappingURL2_relativeName (if it is found on 'std.dkuug.dk') method, which
should return the name of the tables file, relative to the top directory there
(which is '.../Public/MAPPINGS' on www.unicode.org and '.../i18n/charmaops' on 'std.dkuug.dk'.
In this example, the table from 'std.dkuug.dk' is used, and named 'EBCDIC-CP-FI' there.
- generate code by evaluating (make sure that CharacterEncoderGenerator is loaded from stx:goodies):
CharacterEncoder::CP267 generateCode
That's all!!
The existing code was generated by:
CharacterEncoder::SingleByteEncoder subclassesDo:[:cls | Transcript showCR:cls name. cls flushCode; generateCode ]
CharacterEncoder::SingleByteEncoder subclassesDo:[:cls | cls allSubclassesDo:[:sub | Transcript showCR:sub name. sub flushCode; generateSubclassCode]]
or individually:
CharacterEncoder::ASCII flushCode; generateCode.
CharacterEncoder::ISO8859_1 flushCode; generateCode.
CharacterEncoder::ISO8859_2 flushCode; generateCode.
CharacterEncoder::ISO8859_3 flushCode; generateCode.
CharacterEncoder::ISO8859_4 flushCode; generateCode.
CharacterEncoder::ISO8859_5 flushCode; generateCode.
CharacterEncoder::ISO8859_6 flushCode; generateCode.
CharacterEncoder::ISO8859_7 flushCode; generateCode.
CharacterEncoder::ISO8859_8 flushCode; generateCode.
CharacterEncoder::ISO8859_9 flushCode; generateCode.
CharacterEncoder::ISO8859_10 flushCode; generateCode.
CharacterEncoder::ISO8859_11 flushCode; generateCode.
CharacterEncoder::ISO8859_13 flushCode; generateCode.
CharacterEncoder::ISO8859_14 flushCode; generateCode.
CharacterEncoder::ISO8859_15 flushCode; generateCode.
CharacterEncoder::ISO8859_16 flushCode; generateCode.
CharacterEncoder::KOI8_R flushCode; generateCode.
CharacterEncoder::GSM0338 flushCode; generateCode.
CharacterEncoder::KOI8_U flushCode; generateSubclassCode.
CharacterEncoder::JIS0208 flushCode; generateCode.
Please check if your encoder tables are complete; for example, with:
0 to:255 do:[:ebc |
|asc ebc2|
asc := CharacterEncoderImplementations::EBCDIC new decode:ebc.
asc notNil ifTrue:[
ebc2 := CharacterEncoderImplementations::EBCDIC new encode:asc.
self assert:(ebc2 = ebc)
].
].
0 to:255 do:[:asc |
|ebc asc2|
ebc := CharacterEncoderImplementations::EBCDIC new encode:asc.
ebc notNil ifTrue:[
asc2 := CharacterEncoderImplementations::EBCDIC new decode:ebc.
self assert:(asc2 = asc)
].
].
"
! !
!CharacterEncoder class methodsFor:'instance creation'!
decoderForUTF8
"return an encoder-instance which can map utf8 to/from unicode"
^ InverseEncoder new decoder:self encoderForUTF8
"
self encoderForUTF8
self decoderForUTF8
"
!
encoderFor:encodingNameSymbol
"given the name of an encoding, return an encoder-instance which can map these from/into unicode."
^ self
encoderFor:encodingNameSymbol
ifAbsent:[
"/ proceed to ignore this error in the future.
"/ (EncodersByName at:#unicode) at:encodingNameSymbol put:NullEncoderInstance.
"/ (EncoderClassesByName at:#unicode) at:encodingNameSymbol put:NullEncoder.
"/ self error:'no encoder for ' , encodingNameSymbol mayProceed:true.
('CharacterEncoder [warning]: no encoder for "' , encodingNameSymbol,'"') infoPrintCR.
NullEncoderInstance
]
"
CharacterEncoder encoderFor:#'blabla2'
CharacterEncoder encoderFor:#'latin1'
self encoderFor:#'arabic'
self encoderFor:#'ms-arabic'
self encoderFor:#'cp1250'
self encoderFor:#'cp1251'
self encoderFor:#'cp1252'
self encoderFor:#'cp1253'
self encoderFor:#'iso8859-5'
self encoderFor:#'koi8-r'
self encoderFor:#'koi8-u'
self encoderFor:#'jis0208'
self encoderFor:#'jis7'
self encoderFor:#'utf8'
(self encoderFor:#'utf16le') encodeString:'hello'
(self encoderFor:#'utf16le') encode:5
(self encoderFor:#'utf16be') encodeString:'hello'
(self encoderFor:#'utf16be') encode:5
(self encoderFor:#'utf32le') encodeString:'hello'
(self encoderFor:#'utf32be') encodeString:'hello'
self encoderFor:#'sgml'
self encoderFor:#'java'
"
"Modified: / 12-07-2012 / 19:35:43 / cg"
!
encoderFor:encodingNameSymbolArg ifAbsent:exceptionValue
"given the name of an encoding, return an encoder-instance which can map these from/into unicode."
|encodingNameSymbol enc clsName cls unicodeEncoders unicodeEncoderClasses|
encodingNameSymbolArg isNil ifTrue:[
^ NullEncoderInstance
].
encodingNameSymbol := encodingNameSymbolArg asLowercase asSymbolIfInternedOrSelf.
(encodingNameSymbol == #'iso10646-1' or:[encodingNameSymbol == #unicode]) ifTrue:[
"encode unicode from/into unicode"
^ NullEncoderInstance
].
encodingNameSymbol includesMatchCharacters ifTrue:[
AccessLock critical:[
unicodeEncoders := EncodersByName at:#unicode ifAbsent:nil.
].
unicodeEncoders notNil ifTrue:[
unicodeEncoders keysAndValuesDo:[:eachEncodingAlias :eachEncoderInstance |
(encodingNameSymbol matches:eachEncodingAlias) ifTrue:[
^ eachEncoderInstance.
].
].
].
AccessLock critical:[
unicodeEncoderClasses := self encoderClassesByName at:#unicode.
].
unicodeEncoderClasses notNil ifTrue:[
unicodeEncoderClasses keysAndValuesDo:[:eachEncodingAlias :eachEncoderClassOrName |
(encodingNameSymbol matches:eachEncodingAlias) ifTrue:[
eachEncoderClassOrName isBehavior ifTrue:[
cls := eachEncoderClassOrName
] ifFalse:[
cls := CharacterEncoderImplementations at:eachEncoderClassOrName.
].
cls notNil ifTrue:[
^ cls new.
]
].
].
].
^ exceptionValue value
].
AccessLock critical:[
unicodeEncoders := EncodersByName at:#unicode ifAbsentPut:[Dictionary new].
enc := unicodeEncoders at:encodingNameSymbol ifAbsent:nil.
].
enc isNil ifTrue:[
AccessLock critical:[
unicodeEncoderClasses := self encoderClassesByName at:#unicode ifAbsentPut:[Dictionary new].
clsName := unicodeEncoderClasses at:encodingNameSymbol ifAbsent:nil.
].
clsName notNil ifTrue:[
clsName isBehavior ifTrue:[
cls := clsName
] ifFalse:[
cls := CharacterEncoderImplementations at:clsName.
].
cls notNil ifTrue:[
enc := cls new.
AccessLock critical:[
unicodeEncoders at:encodingNameSymbol put:enc.
]
].
].
].
enc notNil ifTrue:[
^ enc
].
"/ no direct encoder from unicode->encodingNameSymbol
"/ search for unicode->any and: any->encodingNameSymbol
AccessLock critical:[
unicodeEncoderClasses := self encoderClassesByName at:#unicode ifAbsent:nil.
].
unicodeEncoderClasses keysAndValuesDo:[:eachEncodingAlias :eachEncoderClass |
|dict2 enc1 enc2|
AccessLock critical:[
dict2 := self encoderClassesByName at:eachEncodingAlias ifAbsent:nil.
].
dict2 notNil ifTrue:[
clsName := dict2 at:encodingNameSymbol ifAbsent:nil.
clsName notNil ifTrue:[
clsName isBehavior ifTrue:[
cls := clsName
] ifFalse:[
cls := CharacterEncoderImplementations at:clsName.
].
cls notNil ifTrue:[
enc2 := cls new.
enc1 := self encoderFor:eachEncodingAlias.
(enc1 notNil and:[enc2 notNil]) ifTrue:[
enc := TwoStepEncoder new encoder1:enc1 encoder2:enc2.
AccessLock critical:[
unicodeEncoders at:encodingNameSymbol put:enc.
].
^ enc.
]
]
]
].
].
self encoderClassesByName keysAndValuesDo:[:encoding1 :dict1 |
dict1 keysAndValuesDo:[:encoding2 :clsName1|
|clsName2 cls1 cls2 dict2 enc1 enc2|
encoding2 = encodingNameSymbol ifTrue:[
AccessLock critical:[
dict2 := self encoderClassesByName at:#unicode.
].
clsName2 := dict2 at:encoding1 ifAbsent:nil.
clsName2 notNil ifTrue:[
clsName1 isBehavior ifTrue:[
cls1 := clsName1
] ifFalse:[
cls1 := CharacterEncoderImplementations at:clsName1.
].
clsName2 isBehavior ifTrue:[
cls2 := clsName2
] ifFalse:[
cls2 := CharacterEncoderImplementations at:clsName2.
].
(cls1 notNil and:[cls2 notNil]) ifTrue:[
enc1 := cls1 new.
enc2 := cls2 new.
enc := TwoStepEncoder new encoder1:enc1 encoder2:enc2.
^ enc.
].
]
]
]
].
^ exceptionValue value
"
CharacterEncoder encoderFor:#'latin1'
self encoderFor:#'iso10646-1'
self encoderFor:#'arabic'
self encoderFor:#'ms-arabic'
self encoderFor:#'iso8859-5'
self encoderFor:#'koi8-r'
self encoderFor:#'koi8-u'
self encoderFor:#'jis0208'
self encoderFor:#'jis7'
self encoderFor:#'unicode'
self encoderFor:#'UTF-8'
self encoderFor:'UTF-8'
"
"Modified: / 12-07-2012 / 19:45:58 / cg"
"Modified (comment): / 05-03-2018 / 16:04:52 / stefan"
!
encoderForUTF8
"return an encoder-instance which can map unicode into/from utf8"
^ self encoderFor:#utf8
"
self encoderForUTF8
"
"Modified (comment): / 17-01-2018 / 13:07:31 / stefan"
!
encoderToEncodeFrom:oldEncodingArg into:newEncodingArg
|oldEncoding newEncoding encoders encoderClasses encoder decoder clsName cls|
oldEncoding := oldEncodingArg ? #unicode.
oldEncoding == #'iso10646-1' ifTrue:[ oldEncoding := #unicode].
newEncoding := newEncodingArg ? #unicode.
newEncoding == #'iso10646-1' ifTrue:[ newEncoding := #unicode].
oldEncoding = newEncoding ifTrue:[^ NullEncoderInstance].
(oldEncoding match:newEncoding) ifTrue:[^ NullEncoderInstance].
(oldEncoding = #unicode) ifTrue:[
"/ unicode -> something
^ self encoderFor:newEncoding.
].
oldEncoding := oldEncoding asSymbol.
newEncoding := newEncoding asSymbol.
AccessLock critical:[
encoders := EncodersByName at:oldEncoding ifAbsentPut:[Dictionary new].
encoder := encoders at:newEncodingArg ifAbsent:nil.
encoder isNil ifTrue:[
encoderClasses := self encoderClassesByName at:oldEncoding ifAbsentPut:[Dictionary new].
clsName := encoderClasses at:newEncoding ifAbsent:nil.
clsName notNil ifTrue:[
clsName isBehavior ifTrue:[
cls := clsName
] ifFalse:[
cls := CharacterEncoderImplementations at:clsName.
]
].
].
].
cls notNil ifTrue:[
encoder := cls new.
].
encoder isNil ifTrue:[
"/ something -> unicode
decoder := self encoderFor:oldEncoding.
(newEncoding == #unicode) ifTrue:[
encoder := InverseEncoder new decoder:decoder.
] ifFalse:[
"/ do it as: oldEncoding -> unicode -> newEncoding
"/ unicode -> something
encoder := self encoderFor:newEncoding.
encoder := CompoundEncoder new encoder:encoder decoder:decoder.
].
].
AccessLock critical:[
(EncodersByName at:oldEncoding) at:newEncoding put:encoder
].
^ encoder
"
CharacterEncoder initialize
CharacterEncoder encoderToEncodeFrom:#'latin1' into:#'jis7'
CharacterEncoder encoderToEncodeFrom:#'koi8-r' into:#'mac-cyrillic'
CharacterEncoder encoderToEncodeFrom:#'ms-arabic' into:#'mac-arabic'
CharacterEncoder encoderToEncodeFrom:#'iso8859-5' into:#'koi8-r'
CharacterEncoder encoderToEncodeFrom:#'iso8859-5' into:#'unicode'
CharacterEncoder encoderToEncodeFrom:#'koi8-r' into:#'koi8-u'
CharacterEncoder encoderToEncodeFrom:#'utf-8' into:#unicode
"
"Modified: / 12-07-2012 / 19:45:15 / cg"
"Modified: / 16-01-2018 / 17:11:17 / stefan"
"Modified (comment): / 17-01-2018 / 12:58:32 / stefan"
! !
!CharacterEncoder class methodsFor:'Compatibility-ST80'!
encoderNamed: encoderName
"/ q & d hack
encoderName == #default ifTrue:[
^ DefaultEncoder new
].
self halt:'should not be reached'.
^ self new
!
platformName
^ OperatingSystem platformName
"Created: 20.6.1997 / 17:34:03 / cg"
"Modified: 20.6.1997 / 17:38:40 / cg"
! !
!CharacterEncoder class methodsFor:'accessing'!
nullEncoderInstance
^ NullEncoderInstance
! !
!CharacterEncoder class methodsFor:'class initialization'!
encoderClassesByName
EncoderClassesByName isNil ifTrue:[
self initializeEncoderClassesByName
].
^ EncoderClassesByName
!
initialize
AccessLock notNil ifTrue:[^ self]. "/ already initialized
AccessLock := RecursionLock name:'CharacterEncoder'.
NullEncoderInstance := NullEncoder new.
EncodersByName := Dictionary new.
CachedEncoders := Dictionary new.
self initializeEncoderClassesByName.
"
self initialize
"
"Modified: / 01-04-2011 / 14:30:06 / cg"
"Modified (format): / 23-01-2013 / 09:56:53 / Jan Vrany <jan.vrany@fit.cvut.cz>"
"Modified: / 27-02-2017 / 15:43:56 / stefan"
!
initializeEncoderClassesByName
"initialize the dictionary which maps commonly used names
to encoder classes.
This is done, because some encodings come along with different names"
|ud|
EncoderClassesByName := Dictionary new.
EncoderClassesByName at:#'unicode' put:(ud := Dictionary new:237).
ud at:#'fontspecific' put:NullEncoder.
ud at:#'adobe-fontspecific' put:NullEncoder.
ud at:#'ms-oem' put:NullEncoder.
ud at:#'ms-default' put:NullEncoder.
"/ className decoded-name array-of-encodingNames
#(
(ASCII unicode ( ascii 'us-ascii' 'iso-ir-6' 'ibm-367' 'ms-cp367' 'cp367' 'iso646-us' 'ibm-cp367' 'ansi_x3.4-1968' ))
(#'ASCII::ASCII7' unicode ( ascii7))
(BIG5 unicode ( big5 ))
(CNS11643 unicode ( 'cns11643' ))
(CP437 unicode ( 'cp437' 'cp-437' 'ibm-437' 'ms-cp437' 'microsoft-cp437' 'ibm-cp437' ))
(EBCDIC unicode ( 'ebcdic' ))
(#'EBCDIC::EBCDIC_037' unicode ( 'ebcdic-037' 'cp-037' 'cp-37' ))
"/ (GB2313_1980 unicode ( 'gb2313' 'gb2313-1980' ))
(GB2312_1980_0 unicode ( 'gb2312' 'gb2312.1980' 'gb2312.1980-0'))
(HANGUL unicode ( 'hangul' ))
(ISO10646_1 unicode ( unicode 'iso10646_1' 'iso10646-1' 'iso-10646-1' ))
(ISO10646_to_UTF8 unicode ( utf8 'utf-8' 'utf_8' ))
(ISO10646_to_UTF16BE unicode ( utf16b utf16be 'utf-16b' 'utf-16be' ))
(ISO10646_to_UTF16LE unicode ( utf16l utf16le 'utf-16e' 'utf-16le' ))
(ISO10646_to_UTF8_MAC unicode ( 'utf8-mac' 'utf-8-mac' ))
(ISO10646_to_XMLUTF8 unicode ( 'utf8-XML' ))
(ISO8859_1 unicode ( 'iso8859_1' 'iso8859-1' 'iso-8859-1' 'latin-1' 'latin1' 'iso-ir-100' 'ibm-819' 'ms-cp819' 'ibm-cp819' 'iso8859'))
(ISO8859_2 unicode ( 'iso8859_2' 'iso8859-2' 'iso-8859-2' 'latin2' 'latin-2' 'iso-ir-101'))
(ISO8859_3 unicode ( 'iso8859_3' 'iso8859-3' 'iso-8859-3' 'latin3' 'latin-3' 'iso-ir-109'))
(ISO8859_4 unicode ( 'iso8859_4' 'iso8859-4' 'iso-8859-4' 'latin4' 'latin-4' 'iso-ir-110'))
(ISO8859_5 unicode ( 'iso8859_5' 'iso8859-5' 'iso-8859-5' 'cyrillic' 'iso-ir-144' ))
(ISO8859_6 unicode ( 'iso8859_6' 'iso8859-6' 'iso-8859-6' 'arabic' 'asmo-708' 'ecma-114' 'iso-ir-127' ))
(ISO8859_7 unicode ( 'iso8859_7' 'iso8859-7' 'iso-8859-7' 'greek' 'iso-ir-126' 'ecma-118'))
(ISO8859_8 unicode ( 'iso8859_8' 'iso8859-8' 'iso-8859-8' 'hebrew' 'iso-ir-138' ))
(ISO8859_9 unicode ( 'iso8859_9' 'iso8859-9' 'iso-8859-9' 'latin5' 'latin-5' 'iso-ir-148'))
(ISO8859_10 unicode ( 'iso8859_10' 'iso8859-10' 'iso-8859-10' 'latin6' 'latin-6' 'iso-ir-157'))
(ISO8859_11 unicode ( 'iso8859_11' 'iso8859-11' 'iso-8859-11' 'thai' ))
(ISO8859_13 unicode ( 'iso8859_13' 'iso8859-13' 'iso-8859-13' 'latin7' 'latin-7' ))
(ISO8859_14 unicode ( 'iso8859_14' 'iso8859-14' 'iso-8859-14' 'latin8' 'latin-8' 'latin-celtic' ))
(ISO8859_15 unicode ( 'iso8859_15' 'iso8859-15' 'iso-8859-15' 'latin9' 'latin-9' 'iso-ir-203'))
(ISO8859_16 unicode ( 'iso8859_16' 'iso8859-16' 'iso-8859-16' 'latin10' 'latin-10' ))
(JIS0201 unicode ( 'jis0201' #'jisx0201.1976-0'))
(JIS0208 unicode ( jis0208 'jisx0208' 'jisx0208.1983-0' 'jisx0208.1990-0'))
(JIS0208_to_JIS7 jis0208 ( jis7 'jis-7' 'x-jis7' 'x-iso2022-jp' 'iso2022-jp'))
(JIS0208_to_EUC jis0208 ( euc #'x-euc-jp' ))
(JIS0208_to_SJIS jis0208 ( 'sjis' 'shiftjis' 'x-sjis' #'x-shift-jis' #'shift-jis'))
(JIS0212 unicode ( 'jis0212' ))
(JOHAB unicode ( 'johab' ))
(KOI7 unicode ( 'koi7' ))
(KOI8_R unicode ( #'koi8-r' 'cp878' ))
(KOI8_U unicode ( #'koi8-u' ))
(KSC5601 unicode ( #'ksc5601' ))
(MAC_Arabic unicode ( #'mac-arabic' 'macarabic' ))
(MAC_CentralEuropean unicode ( #'mac-centraleuropean' #'mac-centraleurope' 'maccentraleurope' 'maccentraleuropean' ))
(MAC_Croatian unicode ( #'mac-croatian' 'maccroatian'))
(MAC_Cyrillic unicode ( #'mac-cyrillic' 'maccyrillic' ))
(MAC_Dingbats unicode ( #'mac-dingbats' 'macdingbats' 'macdingbat'))
(MAC_Farsi unicode ( #'mac-farsi' 'macfarsi' ))
(MAC_Greek unicode ( #'mac-greek' #'macgreek' ))
(MAC_Hebrew unicode ( #'mac-hebrew' #'machebrew' ))
(MAC_Iceland unicode ( #'mac-iceland' #'maciceland' ))
(MAC_Japanese unicode ( #'mac-japanese' #'macjapanese' ))
(MAC_Korean unicode ( #'mac-korean' #'mackorean' ))
(MAC_Roman unicode ( #'mac-roman' #'macroman' 'macintosh' 'cp10000' ))
(MAC_Romanian unicode ( #'mac-romanian' #'macromanian' ))
(MAC_Symbol unicode ( #'mac-symbol' #'macsymbol' ))
(MAC_Thai unicode ( #'mac-thai' #'macthai' ))
(MAC_Turkish unicode ( #'mac-turkish' #'macturkish' ))
(MS_Ansi unicode ( #'ms-ansi' 'microsoft-ansi'))
(MS_CP1252 unicode ( 'cp1252' 'cp-1252' 'ms-cp1252' 'microsoft-cp1252' 'windows-1252' 'windows-latin1'))
(MS_Arabic unicode ( 'cp1256' 'cp-1256' 'ms-arabic' 'ms-cp1256' 'microsoft-cp1256' 'microsoft-arabic' 'windows-1256' ))
(MS_Baltic unicode ( 'cp1257' 'cp-1257' 'ms-baltic' 'ms-cp1257' 'microsoft-cp1257' 'microsoft-baltic' 'windows-1257' ))
(MS_Cyrillic unicode ( 'cp1251' 'cp-1251' 'ms-cyrillic' 'ms-cp1251' 'microsoft-cp1251' 'microsoft-cyrillic' 'windows-1251' ))
(MS_EastEuropean unicode ( 'cp1250' 'cp-1250' 'ms-easteuropean' 'ms-ee' 'ms-cp1250' 'microsoft-cp1250' 'microsoft-easteuropean' 'windows-1250' ))
(MS_Greek unicode ( 'cp1253' 'cp-1253' 'ms-greek' 'ms-cp1253' 'microsoft-cp1253' 'microsoft-greek' 'windows-1253' ))
(MS_Hebrew unicode ( 'cp1255' 'cp-1255' 'ms-hebrew' 'ms-cp1255' 'microsoft-cp1255' 'microsoft-hebrew' 'windows-1255' ))
"/ (MS_Symbol unicode ( 'ms-symbol' 'microsoft-symbol' ))
(MS_Turkish unicode ( 'cp1254' 'cp-1254' 'ms-turkish' 'ms-cp1254' 'microsoft-cp1254' 'microsoft-turkish' 'windows-1254' ))
(NEXT unicode ( 'next' 'nextstep' ))
(ISO10646_to_SGML unicode ( 'sgml' ))
(ISO10646_to_JavaText unicode ( 'java' 'javaText' ))
"/ (AdobeStandard unicode ( 'Adobe Standard' 'AdobeStandard' 'Adobe' 'adobe-standard' ))
"/ (AdobeSymbol unicode ( 'Adobe Symbol' 'AdobeSymbol' 'Symbol' 'adobe-symbol' ))
) triplesDo:[:className :decodesTo :encodesTo |
|decodesToDict|
"/ notice that the encoders are not yet installed as autoloaded.
"/ Therefore, we remember their names here.
decodesToDict := EncoderClassesByName at:decodesTo ifAbsentPut:[Dictionary new].
encodesTo do:[:eachEncodingAlias |
decodesToDict at:eachEncodingAlias put:className ifPresent:[:classAlready | self halt:'conflicting alias'].
].
].
"/ flush
"/ EncodersByName := Dictionary new.
"
self initializeEncoderClassesByName
"
"Modified (format): / 23-01-2013 / 09:56:53 / Jan Vrany <jan.vrany@fit.cvut.cz>"
"Modified: / 27-02-2017 / 16:17:43 / stefan"
"Modified: / 12-11-2017 / 13:05:38 / cg"
"Modified: / 08-10-2018 / 08:59:01 / Claus Gittinger"
! !
!CharacterEncoder class methodsFor:'constants'!
jis7KanjiEscapeSequence
"return the escape sequence used to switch to kanji in jis7 encoded strings.
This happens to be the same as ISO2022-JP's escape sequence."
Jis7KanjiEscapeSequence isNil ifTrue:[
Jis7KanjiEscapeSequence := Character esc asString , '$B'.
].
^ Jis7KanjiEscapeSequence.
"Created: 26.2.1996 / 17:38:08 / cg"
"Modified: 30.6.1997 / 16:03:16 / cg"
!
jis7KanjiOldEscapeSequence
"return the escape sequence used to switch to kanji in some old jis7 encoded strings."
Jis7KanjiOldEscapeSequence isNil ifTrue:[
Jis7KanjiOldEscapeSequence := Character esc asString , '$@'.
].
^ Jis7KanjiOldEscapeSequence.
!
jis7RomanEscapeSequence
"return the escape sequence used to switch to roman in jis7 encoded strings"
Jis7RomanEscapeSequence isNil ifTrue:[
Jis7RomanEscapeSequence := Character esc asString , '(J'.
].
^ Jis7RomanEscapeSequence.
"Created: 26.2.1996 / 17:38:08 / cg"
"Modified: 30.6.1997 / 16:03:16 / cg"
!
jisISO2022EscapeSequence
"return the escape sequence used to switch to kanji in iso2022 encoded strings"
JisISO2022EscapeSequence isNil ifTrue:[
JisISO2022EscapeSequence := Character esc asString , '&@' , Character esc asString , '$B'.
].
^ JisISO2022EscapeSequence.
! !
!CharacterEncoder class methodsFor:'encoding & decoding'!
decodeString:anEncodedStringOrByteCollection
^ self new decodeString:anEncodedStringOrByteCollection
"
CharacterEncoderImplementations::ISO8859_1 decodeString:'hello'
CharacterEncoderImplementations::ISO8859_1 decodeString:'hello' asByteArray
"
"Modified (comment): / 17-01-2018 / 13:44:41 / stefan"
!
decodeString:aString from:oldEncoding
^ self encodeString:aString from:oldEncoding into:#unicode
"
self encodeString:'hello' into:#ebcdic
self decodeString:(self encodeString:'hello' into:#ebcdic) from:#ebcdic
"
"Modified (format): / 17-01-2018 / 15:47:00 / stefan"
!
encode:codePoint from:oldEncodingArg into:newEncodingArg
|oldEncoding newEncoding encoder|
oldEncodingArg == newEncodingArg ifTrue:[
^ codePoint
].
oldEncoding := oldEncodingArg.
newEncoding := newEncodingArg.
(oldEncoding isNil or:[oldEncoding == #'iso10646-1' or:[oldEncoding == #'ms-default']]) ifTrue:[
oldEncoding := #unicode
].
(newEncoding isNil or:[newEncoding == #'iso10646-1' or:[newEncoding == #'ms-default']]) ifTrue:[
newEncoding := #unicode.
].
oldEncoding == newEncoding ifTrue:[
^ codePoint
].
(oldEncoding == #unicode and:[newEncoding == #'iso8859-1' and:[codePoint <= 16rFF]]) ifTrue:[
^ codePoint
].
(newEncoding == #unicode and:[oldEncoding == #'iso8859-1' and:[codePoint <= 16rFF]]) ifTrue:[
^ codePoint
].
encoder := self encoderToEncodeFrom:oldEncoding into:newEncoding.
^ encoder encode:codePoint.
"Modified: / 17-01-2018 / 14:33:08 / stefan"
!
encodeString:aUnicodeString
"given a string in unicode, return a string in my encoding for it"
^ self new encodeString:aUnicodeString
"
CharacterEncoderImplementations::ISO8859_1 encodeString:'hello'
"
"Modified (comment): / 16-01-2018 / 21:57:35 / stefan"
!
encodeString:aString from:oldEncodingArg into:newEncodingArg
|oldEncoding newEncoding encoder|
oldEncodingArg == newEncodingArg ifTrue:[
^ aString
].
oldEncoding := oldEncodingArg.
newEncoding := newEncodingArg.
"/ some hard coded aliases
(oldEncoding isNil or:[oldEncoding == #'iso10646-1' or:[oldEncoding == #'ms-default']]) ifTrue:[
oldEncoding := #'unicode'
].
(newEncoding isNil or:[newEncoding == #'iso10646-1' or:[newEncoding == #'ms-default']]) ifTrue:[
newEncoding := #'unicode'
].
oldEncoding == newEncoding ifTrue:[
^ aString
].
"/ for single-byte strings, iso8859-1 and unicode (up to FF) have the same encoding
(oldEncoding == #unicode and:[newEncoding == #'iso8859-1' and:[aString isWideString not]]) ifTrue:[
^ aString
].
(newEncoding == #unicode and:[oldEncoding == #'iso8859-1' and:[aString isWideString not]]) ifTrue:[
^ aString
].
encoder := self encoderToEncodeFrom:oldEncoding into:newEncoding.
^ encoder encodeString:aString.
"
self encodeString:(self encodeString:'hello' into:#ebcdic) from:#ebcdic into:#ascii
self encodeString:(self encodeString:'hello' into:#ebcdic) from:#ebcdic into:#unicode
self encodeString:(self encodeString:'Äh ... hello' into:#ebcdic) from:#ebcdic into:#utf8
"
"Modified (comment): / 17-01-2018 / 15:49:40 / stefan"
!
encodeString:aString into:newEncoding
^ self encodeString:aString from:#unicode into:newEncoding
"
self encodeString:'hello' into:#ebcdic
self encodeString:(self encodeString:'hello' into:#ebcdic) from:#ebcdic into:#ascii
self encodeString:(self encodeString:'hello' into:#ebcdic) from:#ebcdic into:#unicode
self encodeString:(self encodeString:'hello' into:#ebcdic) from:#ebcdic into:#utf8
"
"Modified (comment): / 17-01-2018 / 15:48:07 / stefan"
! !
!CharacterEncoder class methodsFor:'private'!
flushCode
self initialize.
self isAbstract ifFalse:[
(self mapFileURL1_relativePathName notNil
or:[ self mapFileURL2_relativePathName notNil]) ifTrue:[
self class removeSelector:#mapping.
].
].
"
self flushCode
"
! !
!CharacterEncoder class methodsFor:'private-mapping setup'!
generateCode
(CharacterEncoderCodeGenerator new targetClass:self) generateCode.
!
generateSubclassCode
(CharacterEncoderCodeGenerator new targetClass:self) generateSubclassCode.
!
mapFileURL1_codeColumn
^ 1
!
mapFileURL1_relativePathName
"must be redefined in concrete subclass(es)"
^ nil
!
mapFileURL2_relativePathName
"must be redefined in concrete subclass(es)"
^ nil
!
mappingURL1
|rel|
rel := self mapFileURL1_relativePathName.
rel isNil ifTrue:[
^ nil
].
^ 'http://www.unicode.org/Public/MAPPINGS/' , rel
!
mappingURL2
|rel|
rel := self mapFileURL2_relativePathName.
rel isNil ifTrue:[
^ nil
].
^ 'http://std.dkuug.dk/i18n/charmaps/' , rel
! !
!CharacterEncoder class methodsFor:'queries'!
isAbstract
"Return if this class is an abstract class.
True is returned for CharacterEncoder here; false for subclasses.
Abstract subclasses must redefine this again."
^ self == CharacterEncoder
!
isEncoding:subSetEncodingArg subSetOf:superSetEncodingArg
"return true, if superSetEncoding encoding includes all characters of subSetEncoding.
(this means: characters are included - not that they have the same encoding)"
|subSetEncoding superSetEncoding|
subSetEncodingArg = superSetEncodingArg ifTrue:[^ true].
subSetEncoding := subSetEncodingArg asLowercase.
superSetEncoding := superSetEncodingArg asLowercase.
(subSetEncoding match:superSetEncoding) ifTrue:[^ true].
(('iso10646*' match:superSetEncoding)
or:[superSetEncoding = 'unicode'
or:[superSetEncoding = 'ms-ansi'
or:[superSetEncoding = 'ms-default']]]) ifTrue:[
"/ assume that any character is in unicode
^ true.
].
"/ if the subSet is iso8859-*, that means ascii (i.e. the lower 7 bits of iso8859 only).
((subSetEncoding = 'iso8859*') or:[subSetEncoding = 'iso8859-*']) ifTrue:[
('ascii*' match:superSetEncoding) ifTrue:[^ true].
('ms-ansi*' match:superSetEncoding) ifTrue:[^ true].
('ms-default*' match:superSetEncoding) ifTrue:[^ true].
].
(subSetEncoding = 'ascii') ifTrue:[
('iso8859*' match:superSetEncoding) ifTrue:[^ true].
('ms-ansi*' match:superSetEncoding) ifTrue:[^ true].
('ms-default*' match:superSetEncoding) ifTrue:[^ true].
].
"/ TODO: check the charSets mappingTables...
"/ self halt.
^ false.
!
nameOfDecodedCode
"Most coders decode from their code into unicode / encode from unicode into their code.
There are a few exceptions to this, though - these must redefine this."
^ #'unicode'
!
nameOfEncoding
^ (self nameWithoutPrefix asLowercase copyReplaceAll:$_ with:$-) asSymbol
!
supportedExternalEncodings
"return an array of arrays containing the names of supported
encodings which are supported for external resources (i.e. files).
The first element contains the internally used symbolic name,
the second contains a user-readable string (description).
More than one external name may be mapped onto the same symbolic."
^ #(
('utf8' 'Unicode as 8Bit characters' )
('utf16BE' 'Unicode as 16Bit big-endian' )
('utf16LE' 'Unicode as 16Bit little-endian' )
"/ ('utf7' 'Unicode as 7Bit characters' )
"/ nil
('ascii' 'Common 7bit subset of iso8859' )
('iso8859-1' 'Western' )
('iso8859-2' 'Central European' )
('iso8859-3' 'South European' )
('iso8859-4' 'Baltic' )
('iso8859-5' 'Cyrillic' )
('iso8859-6' 'Arabic' )
('iso8859-7' 'Greek' )
('iso8859-8' 'Hebrew' )
('iso8859-15' 'Western with Euro' )
('iso8859-16' 'South European with Euro' )
"/ nil
('macintosh' 'MAC Western' )
"/ nil
('koi7' 'Cyrillic (Old)' )
('koi8-r' 'Cyrillic' )
('koi8-u' 'Cyrillic (Ukraine)' )
"/ nil
('cp437' 'Windows US / codepage 437' )
('cp850' 'Windows Latin1 / codepage 850' )
('cp1250' 'Windows Latin2 / codepage 1250' )
('cp1251' 'Windows Cyrillic / codepage 1251')
('cp1252' 'Windows ANSI / codepage 1252' )
"/ ('mac' 'macintosh 8 bit' )
('next' 'NeXT 8 bit' )
"/ ('hp' 'hpux 8 bit' )
"/ nil
('euc' 'EUC - extended unix code (japanese)' )
('jis7' 'JIS7 - jis 7bit escape codes (japanese)' )
('iso-2022-jp' 'Same as jis 7bit' )
('sjis' 'SJIS - shift jis 8bit codes (japanese)' )
"/ nil
('gb' 'GB - mainland china' )
('big5' 'BIG5 - taiwan' )
"/ ('ksc' 'korean' )
('sgml' 'SGML (XML/HTML) character escapes' )
('java' 'JavaText (\uXXXX) character escapes' )
)
"Modified: / 23-10-2006 / 13:27:48 / cg"
!
userFriendlyNameOfEncoding
^ self nameOfEncoding asUppercaseFirst
! !
!CharacterEncoder class methodsFor:'utilities'!
detectAndSkipBOMInStream:stream
"skips over the BOM and returns one of
#utf8
#utf32be
#utf32le
#utf16le
#utf16be
if no BOM is detected, the stream is repositions to where it was before."
|pos byte1|
pos := stream position.
stream atEnd ifTrue:[^ nil].
byte1 := stream peek asInteger.
"/ EF-BB-BF -> utf8
byte1 == 16rEF ifTrue:[
stream next.
stream peek asInteger == 16rBB ifTrue:[
stream next.
stream next asInteger == 16rBF ifTrue:[
^ #utf8
]
].
stream position:pos. ^nil
].
"00-00-FE-FF big endian utf32"
byte1 == 16r00 ifTrue:[
stream next.
stream peek asInteger == 16r00 ifTrue:[
stream next.
stream peek asInteger == 16rFE ifTrue:[
stream next.
stream next asInteger == 16rFF ifTrue:[
^ #utf32be
]
]
].
stream position:pos. ^nil
].
"FF-FE little endian utf16 or utf32"
byte1 == 16rFF ifTrue:[
stream next.
stream peek asInteger == 16rFE ifTrue:[
stream next.
stream peek asInteger == 0 ifTrue:[
stream next.
stream next asInteger == 0 ifTrue:[
"FF-FE-00-00 little endian utf32"
^ #utf32le.
].
stream skip:-2
].
^ #utf16le
].
stream position:pos. ^nil
].
"FE-FF big endian utf16"
byte1 == 16rFE ifTrue:[
stream next.
stream next asInteger == 16rFF ifTrue:[
^ #utf16be
].
].
stream position:pos.
^ nil
"
|s enc|
s := #[1 2 3 4] readStream.
enc := self detectAndSkipBOMInStream:s.
self assert:(enc == nil).
self assert:(s position == 0).
s := #[16rFF 2 3 4] readStream.
enc := self detectAndSkipBOMInStream:s.
self assert:(enc == nil).
self assert:(s position == 0).
s := #[16rFF 16rFE 3 4] readStream.
enc := self detectAndSkipBOMInStream:s.
self assert:(enc == #utf16le).
self assert:(s position == 2).
s := #[16rFE 16rFF 3 4] readStream.
enc := self detectAndSkipBOMInStream:s.
self assert:(enc == #utf16be).
self assert:(s position == 2).
s := #[16rFF 16rFE 0 0 3 4] readStream.
enc := self detectAndSkipBOMInStream:s.
self assert:(enc == #utf32le).
self assert:(s position == 4).
s := #[0 0 16rFE 16rFF 0 0 3 4] readStream.
enc := self detectAndSkipBOMInStream:s.
self assert:(enc == #utf32be).
self assert:(s position == 4).
"
!
detectBOMInBuffer:buffer
"returns one of
#utf8
#utf32be
#utf32le
#utf16le
#utf16be
nil"
^ self detectAndSkipBOMInStream:(buffer readStream)
!
guessEncodingOfBuffer:buffer
"try to guess a string-buffer's encoding.
Basically looks for BOM (byte order marks)
pr a special string of the form
encoding #name
or:
encoding: name
within the given buffer
(which is usually found within the first few bytes of a textFile).
Many editors and tools write such comments (eg. emacs, st/x, etc.)"
buffer size < 4 ifTrue:[
"not enough bytes to determine the contents"
^ nil.
].
EncodingDetectors isNil ifTrue:[
self initializeEncodingDetectors.
].
EncodingDetectors do:[:each |
|guess|
(guess := each value:buffer) notNil ifTrue:[
^ guess
].
].
^ nil
!
guessEncodingOfFile:aFilename
"look for a BOM (byte order mark) or a special string of the form:
encoding #name
or:
encoding: name
within the given buffer
(which is usually found in the first few bytes of a textFile).
If that's not found, use heuristics (in CharacterArray) to guess.
Return a symbol like #utf8."
|s buffer|
s := aFilename asFilename readStreamOrNil.
s isNil ifTrue:[^ nil].
buffer := String new:512.
s nextBytes:buffer size into:buffer.
s close.
^ self guessEncodingOfBuffer:buffer.
"
self guessEncodingOfFile:'../../libview/resources/de.rs' asFilename
self guessEncodingOfFile:'../../libview/resources/ru.rs' asFilename
self guessEncodingOfFile:'../../libview/resources/th.rs' asFilename
"
"Modified: / 31-05-2011 / 15:45:19 / cg"
"Modified: / 16-01-2018 / 17:12:41 / stefan"
!
guessEncodingOfStream:aStream
"look for a BOM (byte order mark) or a special string of the form:
encoding #name
or:
encoding: name
in the first few bytes of aStream.
Return a symbol like #utf8."
|oldPosition buffer|
"/ must be able to position back
aStream isPositionable ifFalse:[
^ nil
].
buffer := String new:512.
oldPosition := aStream position.
aStream nextBytes:buffer size into:buffer.
aStream position:oldPosition.
^ self guessEncodingOfBuffer:buffer
"Modified: / 31-05-2011 / 15:45:23 / cg"
"Modified: / 16-01-2018 / 17:12:57 / stefan"
"Modified (format): / 17-01-2018 / 15:51:09 / stefan"
!
initializeEncodingDetectors
"setup the list of encoding detectors.
This is a list of blocks, which get a buffer as argument,
and return an encoding symbol or nil.
Can be customized for more detectors
(used to be hard-coded in guessEncodingOfBuffer:)"
EncodingDetectors := OrderedCollection new.
"check for Unicode Byte Order Marks (BOM)"
EncodingDetectors add:[:buffer | self detectBOMInBuffer:buffer].
"check for an inline encoding markup (charset= / encoding=) substring"
EncodingDetectors
add:[:buffer |
|guess lcBuffer quote|
lcBuffer := buffer asLowercase.
guess :=
#(charset encoding) doWithExit:[:keyWord :exit |
|encoderOrNil idx s w enc|
guess isNil ifTrue:[
(idx := lcBuffer findString:keyWord) ~~ 0 ifTrue:[
s := ReadStream on:buffer.
s position:idx-1 + keyWord size.
s skipSeparators.
"do not include '=' here, otherwise
files containing xml code (<?xml charset='utf8'> will be parsed as UTF-8"
[':#=' includes:s peek] whileTrue:[
s next.
s skipSeparators.
].
s skipSeparators.
('"''' includes:s peek) ifTrue:[
quote := s next.
w := s upTo:quote.
] ifFalse:[
w := s upToElementForWhich:[:ch | ch isSeparator or:[ch == $" or:[ch == $' or:[ch == $> ]]]].
].
w notNil ifTrue:[
enc := w withoutQuotes.
(enc startsWith:'x-') ifTrue:[
enc := enc copyFrom:3.
].
encoderOrNil := self encoderFor:enc ifAbsent:nil.
encoderOrNil notNil ifTrue:[
exit value:(encoderOrNil nameOfEncoding)
].
].
].
].
nil
].
guess
].
"/ check for a string like /*@!!Encoding:1252*/
EncodingDetectors
add:[:buffer |
|guess idx s keyWord codePageNr enc encoderOrNil|
keyWord := '@!!Encoding:'.
(idx := buffer findString:keyWord) ~~ 0 ifTrue:[
s := ReadStream on:buffer.
s position:idx-1 + keyWord size.
s skipSeparators.
s peek isDigit ifTrue:[
codePageNr := Integer readFrom:s.
enc := 'cp%1' bindWith:codePageNr.
encoderOrNil := self encoderFor:enc ifAbsent:nil.
encoderOrNil notNil ifTrue:[
guess := (encoderOrNil nameOfEncoding)
].
].
].
guess
].
"/ check for JIS7 encoding
EncodingDetectors
add:[:buffer |
(buffer includesString:self jisISO2022EscapeSequence) ifTrue:[
#'iso2020-jp'
] ifFalse:[
(buffer includesString:self jis7KanjiEscapeSequence) ifTrue:[
#jis7
] ifFalse:[
(buffer includesString:self jis7KanjiOldEscapeSequence) ifTrue:[
#jis7
] ifFalse:[
nil
]
]
]
].
"/ TODO: look for EUC, SJIS etc.
"/ Disabled, due to too many false positives.
"/ if required, think about it, fix it and uncomment it
"/ EncodingDetectors
"/ add:[:buffer |
"/ |guess idx|
"/
"/ idx := buffer
"/ findFirst:[:char |
"/ |code|
"/ code := char codePoint.
"/ code between:16rA1 and: 16rFE
"/ ].
"/ ((idx ~~ 0)
"/ and:[ (buffer at:(idx + 1)) codePoint between:16rA1 and: 16rFE ])
"/ ifTrue:[
"/ guess := #euc
"/ ] ifFalse:[
"/ "/ look for SJIS ...
"/ ]
"/ ].
"Modified: / 17-01-2018 / 15:55:36 / stefan"
"Modified: / 05-02-2019 / 09:23:37 / Claus Gittinger"
!
showCharacterSet
|font|
font := View defaultFont.
"/ font := (Font family:'courier' face:'medium' style:'roman' size:12 encoding:'iso10646-1').
CharacterSetView
openOn:font
label:'Characters of ',self nameWithoutPrefix
clickLabel:nil
asInputFor:nil
encoder:self
"
CharacterEncoderImplementations::MS_Ansi showCharacterSet
CharacterEncoderImplementations::ISO8859_1 showCharacterSet
CharacterEncoderImplementations::ISO8859_2 showCharacterSet
CharacterEncoderImplementations::ISO8859_3 showCharacterSet
CharacterEncoderImplementations::ISO8859_4 showCharacterSet
CharacterEncoderImplementations::ISO8859_5 showCharacterSet
CharacterEncoderImplementations::ISO8859_6 showCharacterSet
CharacterEncoderImplementations::ISO8859_7 showCharacterSet
CharacterEncoderImplementations::ISO8859_8 showCharacterSet
CharacterEncoderImplementations::ISO8859_9 showCharacterSet
"
! !
!CharacterEncoder methodsFor:'encoding & decoding'!
decodeString:anEncodedStringOrByteCollection
"given a string in my encoding, return a unicode-string for it"
^ self subclassResponsibility
"Modified: / 16-01-2018 / 19:54:51 / stefan"
"Modified (format): / 17-01-2018 / 13:45:06 / stefan"
!
encodeCharacter:aUnicodeCharacterOrCodePoint
"encode aUnicodeCharacterOrCodePoint to a (8-bit) String or ByteArray"
^ self encodeString:aUnicodeCharacterOrCodePoint asString.
"Created: / 17-01-2018 / 13:59:44 / stefan"
!
encodeString:aUnicodeString
"given a string in unicode, return a string or ByteArray in my encoding for it"
^ self subclassResponsibility
"Modified: / 16-01-2018 / 19:54:44 / stefan"
"Modified (comment): / 17-01-2018 / 13:54:44 / stefan"
! !
!CharacterEncoder methodsFor:'error handling'!
decodingError
"report an error that there is no unicode-codePoint for a given codePoint in this encoding.
(which is unlikely) or that the encoding is undefined for that value
(for example, holes in the ISO-8859-3 encoding)"
|badCodePoint sender|
sender := thisContext sender.
((sender selector == #encode:) or:[sender selector == #decode:]) ifFalse:[
badCodePoint := sender methodHome argAt:1
].
^ (DecodingError new)
defaultValue:(self defaultDecoderValue);
parameter:badCodePoint;
messageText:'invalid code';
suspendedContext:sender;
raiseRequest.
!
defaultDecoderValue
"placed into a decoded string, in case there is no unicode codePoint
for a given encoded codePoint.
(typically 16rFFFF)."
^ 16rFFFF
!
defaultEncoderValue
"placed into an encoded string, in case there is no codePoint
for a given unicode codePoint.
(typically $?)."
^ $? codePoint
!
encodingError
"report an error that some unicode-codePoint cannot be represented by this encoder"
|badCodePoint sender|
sender := thisContext sender.
((sender selector == #encode:) or:[sender selector == #decode:]) ifFalse:[
badCodePoint := sender methodHome argAt:1
].
^ (EncodingError new)
defaultValue:(self defaultEncoderValue);
parameter:badCodePoint;
messageText:'unrepresentable code (some character cannot be represented)';
suspendedContext:sender;
raiseRequest
"Modified: / 12-07-2012 / 20:36:37 / cg"
! !
!CharacterEncoder methodsFor:'printing'!
printOn:aStream
aStream
nextPutAll:(self nameOfDecodedCode);
nextPutAll:'->';
nextPutAll:(self nameOfEncoding)
! !
!CharacterEncoder methodsFor:'queries'!
characterSize:charOrCodePoint
"return the number of bytes required to encode codePoint"
^ self subclassResponsibility
"Created: / 15-06-2005 / 15:11:04 / janfrog"
!
isEncoderFor:encoding
"does this encode to encoding?"
|encodingNameSymbol|
encodingNameSymbol := encoding asLowercase.
encodingNameSymbol = #'iso10646-1' ifTrue:[ encodingNameSymbol := #unicode].
^ encodingNameSymbol = self nameOfEncoding
!
isNullEncoder
^ false
!
nameOfDecodedCode
"Most coders decode from their code into unicode / encode from unicode into their code.
There are a few exceptions to this, though - these must redefine this."
^ self class nameOfDecodedCode
!
nameOfEncoding
^ self class nameOfEncoding
!
userFriendlyNameOfEncoding
^ self class userFriendlyNameOfEncoding
! !
!CharacterEncoder methodsFor:'stream support'!
encodeCharacter:aUnicodeCharacter on:aStream
"given a character in unicode, encode it onto aStream.
Subclasses can redefine this to avoid allocating many new string instances."
aStream nextPutAll:(self encodeCharacter:aUnicodeCharacter).
"Created: / 16-02-2017 / 16:18:33 / stefan"
"Modified: / 17-01-2018 / 14:00:28 / stefan"
!
encodeString:aUnicodeString on:aStream
"given a string in unicode, encode it onto aStream.
Subclasses can redefine this to avoid allocating many new string instances.
(but must then also redefine encodeString:aUnicodeString to collect the characters)"
aStream nextPutAll:(self encodeString:aUnicodeString).
!
readNext:countArg charactersFrom:aStream
|writeStream count "{ Class:SmallInteger }"|
count := countArg.
writeStream := CharacterWriteStream on:(String new:count).
count timesRepeat:[
writeStream nextPut:(self readNextCharacterFrom:aStream).
].
^ writeStream contents.
"Created: / 16-01-2018 / 20:08:10 / stefan"
"Modified: / 17-01-2018 / 16:44:29 / stefan"
!
readNextCharacterFrom:aStream
^ self subclassResponsibility
"Created: / 14-06-2005 / 17:03:21 / janfrog"
"Modified: / 15-06-2005 / 15:27:49 / janfrog"
"Modified: / 20-06-2005 / 13:13:52 / masca"
"Modified: / 16-01-2018 / 20:12:07 / stefan"
! !
!CharacterEncoder::CompoundEncoder class methodsFor:'documentation'!
documentation
"
A compoundEncoder uses two real encoders;
to encode:
string -> decoder(encode) -> encoder -> result
to decode:
string -> encoder -> decoder -> result
|e|
e := CompoundEncoder new.
e encoder:ISO8859_5 decoder:KOI8_R.
e decode:16rB0. 'CYRILLIC CAPITAL LETTER A; 16rB0 in 8859-5; 16rE1 in KOI8-R'.
e encode:16rE1.
"
! !
!CharacterEncoder::CompoundEncoder methodsFor:'accessing'!
encoder:encoderArg decoder:decoderArg
"set instance variables (automatically generated)"
decoder := decoderArg.
encoder := encoderArg.
! !
!CharacterEncoder::CompoundEncoder methodsFor:'encoding & decoding'!
decodeString:anEncodedStringOrByteCollection
^ decoder encodeString:(encoder decodeString:anEncodedStringOrByteCollection)
"Modified (format): / 17-01-2018 / 13:44:08 / stefan"
!
encodeString:anEncodedStringOrByteCollection
^ encoder encodeString:(decoder decodeString:anEncodedStringOrByteCollection)
"Modified (format): / 17-01-2018 / 13:46:26 / stefan"
! !
!CharacterEncoder::CompoundEncoder methodsFor:'printing'!
printOn:aStream
aStream
nextPutAll:(decoder nameOfEncoding);
nextPutAll:'->'.
"/ nextPutAll:(decoder nameOfDecodedCode);
"/ nextPutAll:'->';
"/ nextPutAll:(encoder nameOfEncoding)
encoder printOn:aStream
! !
!CharacterEncoder::CompoundEncoder methodsFor:'queries'!
characterSize:aCharacterOrCodepoint
"return the number of bytes required to encode aCharacterOrCodepoint"
^ encoder characterSize:(decoder decode:aCharacterOrCodepoint)
"Created: / 16-01-2018 / 17:58:51 / stefan"
! !
!CharacterEncoder::CompoundEncoder methodsFor:'stream support'!
readNext:count charactersFrom:aStream
^ decoder encodeString:(encoder readNext:count charactersFrom:aStream) asString
"Created: / 16-01-2018 / 20:50:56 / stefan"
!
readNextCharacterFrom:aStream
^ (decoder encodeString:(encoder readNextCharacterFrom:aStream) asString) first
"Created: / 16-01-2018 / 21:10:28 / stefan"
! !
!CharacterEncoder::NullEncoder class methodsFor:'documentation'!
documentation
"
A NullEncoder does nothing.
"
! !
!CharacterEncoder::NullEncoder methodsFor:'encoding & decoding'!
decodeString:anEncodedStringOrByteCollection
^ anEncodedStringOrByteCollection asString
"Modified: / 17-01-2018 / 13:43:42 / stefan"
!
encodeString:aString
^ aString
! !
!CharacterEncoder::NullEncoder methodsFor:'queries'!
characterSize:charOrCodePoint
"return the number of bytes required to encode aCharacterOrCodepoint"
^ charOrCodePoint asCharacter bytesPerCharacter
"
NullEncoder basicNew characterSize:$a codePoint
NullEncoder basicNew characterSize:16r3fe
NullEncoder basicNew characterSize:16r3ffe
"
"Modified (comment): / 16-01-2018 / 21:15:01 / stefan"
!
isNullEncoder
^ true
! !
!CharacterEncoder::NullEncoder methodsFor:'stream support'!
readNext:count charactersFrom:aStream
^ (aStream next:count) asString
"Created: / 16-01-2018 / 20:19:38 / stefan"
!
readNextCharacterFrom:aStream
|chOrNil|
chOrNil := aStream next.
chOrNil notNil ifTrue:[
^ chOrNil asCharacter
].
^ nil.
"Created: / 16-01-2018 / 20:04:01 / stefan"
! !
!CharacterEncoder::InverseEncoder class methodsFor:'documentation'!
documentation
"
An InverseEncoder does the inverse - i.e. encode is really a decode
and decode is really an encode.
InverseEncoder is always used to encode to unicode and decode from unicode
(see CharacterEncoder class >> #encoderToEncodeFrom:into:).
"
! !
!CharacterEncoder::InverseEncoder methodsFor:'accessing'!
decoder:anEncoder
decoder := anEncoder.
! !
!CharacterEncoder::InverseEncoder methodsFor:'encoding & decoding'!
decodeString:anEncodedStringOrByteCollection
^ decoder encodeString:anEncodedStringOrByteCollection
"Modified (format): / 17-01-2018 / 13:43:57 / stefan"
!
encodeString:anEncodedStringOrByteCollection
^ decoder decodeString:anEncodedStringOrByteCollection
"Modified (format): / 17-01-2018 / 13:46:47 / stefan"
! !
!CharacterEncoder::InverseEncoder methodsFor:'printing'!
printOn:aStream
aStream
nextPutAll:(decoder nameOfEncoding);
nextPutAll:'->';
nextPutAll:(decoder nameOfDecodedCode)
! !
!CharacterEncoder::InverseEncoder methodsFor:'queries'!
characterSize:charOrCodePoint
"return the number of bytes required to encode codePoint"
^ decoder characterSize:charOrCodePoint
! !
!CharacterEncoder::InverseEncoder methodsFor:'stream support'!
readNext:count charactersFrom:aStream
"decode the next count bytes or characters on aStream from unicode to something else"
^ decoder encodeString:(aStream next:count).
"Created: / 16-01-2018 / 20:53:42 / stefan"
"Modified (comment): / 17-01-2018 / 13:28:41 / stefan"
!
readNextCharacterFrom:aStream
"decode the next byte or character on aStream from unicode to something else"
^ decoder encodeString:(String with:aStream next).
"Created: / 16-01-2018 / 21:08:11 / stefan"
"Modified: / 17-01-2018 / 13:29:59 / stefan"
! !
!CharacterEncoder::DefaultEncoder class methodsFor:'documentation'!
documentation
"
That is only a dummy for ST80 compatibility
"
! !
!CharacterEncoder::OtherEncoding class methodsFor:'private'!
flushCode
"do nothing here"
"Modified (comment): / 16-01-2018 / 17:08:17 / stefan"
! !
!CharacterEncoder::OtherEncoding class methodsFor:'testing'!
isAbstract
^ self == CharacterEncoder::OtherEncoding
"Created: / 17-01-2018 / 16:06:13 / stefan"
"Modified: / 17-01-2018 / 17:50:37 / stefan"
! !
!CharacterEncoder::TwoStepEncoder class methodsFor:'documentation'!
documentation
"
A twoStepEncoder uses two real encoders;
to encode:
string -> encoder1(encode) -> encoder2(encode) -> result
to decode:
string -> encoder2(decode) -> encoder1(decode) -> result
"
! !
!CharacterEncoder::TwoStepEncoder methodsFor:'accessing'!
encoder1:encoder1Arg encoder2:encoder2Arg
"set instance variables (automatically generated)"
encoder1 := encoder1Arg.
encoder2 := encoder2Arg.
! !
!CharacterEncoder::TwoStepEncoder methodsFor:'encoding & decoding'!
decodeString:anEncodedStringOrByteCollection
^ encoder1 decodeString:(encoder2 decodeString:anEncodedStringOrByteCollection)
"Modified (format): / 17-01-2018 / 13:45:20 / stefan"
!
encodeString:aString
^ encoder2 encodeString:(encoder1 encodeString:aString)
! !
!CharacterEncoder::TwoStepEncoder methodsFor:'printing'!
printOn:aStream
aStream
nextPutAll:(encoder1 nameOfDecodedCode);
nextPutAll:'->';
nextPutAll:(encoder1 nameOfEncoding);
nextPutAll:'->';
nextPutAll:(encoder2 nameOfEncoding)
! !
!CharacterEncoder::TwoStepEncoder methodsFor:'queries'!
characterSize:charOrCodePoint
"return the number of bytes required to encode codePoint"
"/ naive; actually, we have to do a real encoding to get this info proper
^ (encoder2 characterSize:charOrCodePoint)
"Created: / 22-11-2012 / 13:07:47 / cg"
!
nameOfEncoding
^ "encoder1 nameOfEncoding , '-' ," encoder2 nameOfEncoding
! !
!CharacterEncoder::TwoStepEncoder methodsFor:'stream support'!
readNext:count charactersFrom:aStream
^ encoder1 decodeString:(encoder2 readNext:count charactersFrom:aStream)
"Created: / 16-01-2018 / 20:47:52 / stefan"
!
readNextCharacterFrom:aStream
^ (encoder1 decodeString:(encoder2 readNextCharacterFrom:aStream) asString) first
"Created: / 16-01-2018 / 21:06:48 / stefan"
! !
!CharacterEncoder class methodsFor:'documentation'!
version
^ '$Header$'
!
version_CVS
^ '$Header$'
! !
CharacterEncoder initialize!