BNFC – The T World

[haskell][BNFC]n3337 Raw-string

it 2017年1月21日 2017年1月22日BNFC, haskell, プログラム, 字句解析 0

n3337のRaw-stringをBNFCで定義する。

{-
Unicode:
	Unicode scalar value
		Basic Multilingual Plane(BMP) = U+0000 - U+FFFF(*)
		Supplementary Multilingual Plane = U+010000 - U+10FFFF
		*) Reserved U+D800 - U+DFFF for Surrogate Pair encode rule.

	=> char

r-char:
	Any member of the source character set, except a right parenthesis ) followed
	by the initial d-char-sequence (which may be empty) followd by a double quote.

	=> (char - [")"])

d-char:
	Any member of the basic source character set except:
	space, the left parenthesis (, the right parenthesis ), the backslash \,
	and the control characters representing horizontal tab, vertical tab,
	form feed, and newline.

	=> (basic source char - ["() \\\t\v\f\n"])

	2.3 Character sets
	The basic source character set onsists of 96 characters:
	the space character, the control characters representing horizontal tab,
	vertical tab, form feed, and new-line, plus the following 91 graphical characters:
	a-z, A-Z, 0-9 and  _{} []#() <>%:; .?*+- /^&|~ !=,\" '

	=> ["\t\v\f\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"]
-}
LTest. Test::= Raw_string;
token Raw_string ({"u8"} | ["uUL"])? {"R\""} (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '(' (char - [")"])* ')' (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '"';

Unicode:

Unicode scalar value

Basic Multilingual Plane(BMP) = U+0000 - U+FFFF(*)

Supplementary Multilingual Plane = U+010000 - U+10FFFF

*) Reserved U+D800 - U+DFFF for Surrogate Pair encode rule.

=> char

r-char:

Any member of the source character set, except a right parenthesis ) followed

by the initial d-char-sequence (which may be empty) followd by a double quote.

=> (char - [")"])

d-char:

Any member of the basic source character set except:

space, the left parenthesis (, the right parenthesis ), the backslash \,

and the control characters representing horizontal tab, vertical tab,

form feed, and newline.

=> (basic source char - ["() \\\t\v\f\n"])

2.3 Character sets

The basic source character set onsists of 96 characters:

the space character, the control characters representing horizontal tab,

vertical tab, form feed, and new-line, plus the following 91 graphical characters:

a-z, A-Z, 0-9 and _{} []#() <>%:; .?*+- /^&|~ !=,\" '

=> ["\t\v\f\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"]

LTest. Test::= Raw_string;

token Raw_string ({"u8"} | ["uUL"])? {"R\""} (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '(' (char - [")"])* ')' (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '"';

~$ ./TestRawStr 
u8R"2sS(あ)6wY"
Parse Successful!

[Abstract Syntax]

LTest (Raw_string "u8R\"2sS(\12354)6wY\"")

[Linearized tree]

u8R"2sS(あ)6wY"
~$ 
~$ ./TestRawStr 
R"^^(あああ)^^"
Parse Successful!

[Abstract Syntax]

LTest (Raw_string "R\"^^(\12354\12354\12354)^^\"")

[Linearized tree]

R"^^(あああ)^^"
~$

~$ ./TestRawStr

u8R"2sS(あ)6wY"

Parse Successful!

[Abstract Syntax]

LTest (Raw_string "u8R\"2sS(\12354)6wY\"")

[Linearized tree]

u8R"2sS(あ)6wY"

~$ ./TestRawStr

R"^^(あああ)^^"

Parse Successful!

[Abstract Syntax]

LTest (Raw_string "R\"^^(\12354\12354\12354)^^\"")

[Linearized tree]

R"^^(あああ)^^"

うまくパースしてくれているようで …

[haskell][BNFC]Unicodeスカラ値のサロゲートペア範囲考慮

it 2017年1月21日 2017年1月21日BNFC, haskell, プログラム, 字句解析 0

ユニコードスカラ値にはサロゲートペアのための領域0xD800-0xDFFFがあり、この範囲は文字を割り当てることがきない。このためcharの範囲からこの部分を外す。 [crayon-67ed8203dd5bb736401 …

[haskell]BNFCでUnicodeを扱える様にする

it 2017年1月18日 2017年1月18日BNFC, haskell, プログラム 0

BNFCのcharは0〜255の文字コードしか扱えない。これをUnicodeの範囲に広げる。

$ git diff
diff --git a/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs b/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs
index 054d576..e2b744f 100644
--- a/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs
+++ b/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs
@@ -63,7 +63,7 @@ cMacros = [
   "$s = [a-z\\222-\\255] # [\\247]    -- small isolatin1 letter FIXME",
   "$d = [0-9]                -- digit",
   "$i = [$l $d _ ']          -- identifier character",
-  "$u = [\\0-\\255]          -- universal: any character"
+  "$u = [\\x0000-\\x10FFFF]          -- universal: any character"
   ]
 
 rMacros :: CF -> [String]
$

$ git diff

diff --git a/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs b/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs

index 054d576..e2b744f 100644

--- a/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs

+++ b/source/src/BNFC/Backend/Haskell/CFtoAlex3.hs

@@ -63,7 +63,7 @@ cMacros = [

"$s = [a-z\\222-\\255] # [\\247] -- small isolatin1 letter FIXME",

"$d = [0-9] -- digit",

"$i = [$l $d _ '] -- identifier character",

- "$u = [\\0-\\255] -- universal: any character"

+ "$u = [\\x0000-\\x10FFFF] -- universal: any character"

]

rMacros :: CF -> [String]

[crayon-67ed8203dd6dd818522378 …

月	火	水	木	金	土	日
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30