字句解析 – The T World

DeclVisitorが呼んでくれる関数のリスト

it 2017年12月23日 2017年12月23日C++, clang, プログラム, 字句解析 0

clangのDeclVisitorが呼んでくれる関数のリストがお手軽にほしかったので調べてみました。以下のコマンドで出力することができます。　# DeclVisitor.hのマクロを参考にいたしました。 [crayo …

[haskell][BNFC]n3337 Raw-string

it 2017年1月21日 2017年1月22日BNFC, haskell, プログラム, 字句解析 0

n3337のRaw-stringをBNFCで定義する。

{-
Unicode:
	Unicode scalar value
		Basic Multilingual Plane(BMP) = U+0000 - U+FFFF(*)
		Supplementary Multilingual Plane = U+010000 - U+10FFFF
		*) Reserved U+D800 - U+DFFF for Surrogate Pair encode rule.

	=> char

r-char:
	Any member of the source character set, except a right parenthesis ) followed
	by the initial d-char-sequence (which may be empty) followd by a double quote.

	=> (char - [")"])

d-char:
	Any member of the basic source character set except:
	space, the left parenthesis (, the right parenthesis ), the backslash \,
	and the control characters representing horizontal tab, vertical tab,
	form feed, and newline.

	=> (basic source char - ["() \\\t\v\f\n"])

	2.3 Character sets
	The basic source character set onsists of 96 characters:
	the space character, the control characters representing horizontal tab,
	vertical tab, form feed, and new-line, plus the following 91 graphical characters:
	a-z, A-Z, 0-9 and  _{} []#() <>%:; .?*+- /^&|~ !=,\" '

	=> ["\t\v\f\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"]
-}
LTest. Test::= Raw_string;
token Raw_string ({"u8"} | ["uUL"])? {"R\""} (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '(' (char - [")"])* ')' (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '"';

Unicode:

Unicode scalar value

Basic Multilingual Plane(BMP) = U+0000 - U+FFFF(*)

Supplementary Multilingual Plane = U+010000 - U+10FFFF

*) Reserved U+D800 - U+DFFF for Surrogate Pair encode rule.

=> char

r-char:

Any member of the source character set, except a right parenthesis ) followed

by the initial d-char-sequence (which may be empty) followd by a double quote.

=> (char - [")"])

d-char:

Any member of the basic source character set except:

space, the left parenthesis (, the right parenthesis ), the backslash \,

and the control characters representing horizontal tab, vertical tab,

form feed, and newline.

=> (basic source char - ["() \\\t\v\f\n"])

2.3 Character sets

The basic source character set onsists of 96 characters:

the space character, the control characters representing horizontal tab,

vertical tab, form feed, and new-line, plus the following 91 graphical characters:

a-z, A-Z, 0-9 and _{} []#() <>%:; .?*+- /^&|~ !=,\" '

=> ["\t\v\f\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"]

LTest. Test::= Raw_string;

token Raw_string ({"u8"} | ["uUL"])? {"R\""} (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '(' (char - [")"])* ')' (["\t\n0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_{}[]#()<>%:;.?*+-/^&|~!=,\\\"\'"] - ["() \\\t\n"])* '"';

~$ ./TestRawStr 
u8R"2sS(あ)6wY"
Parse Successful!

[Abstract Syntax]

LTest (Raw_string "u8R\"2sS(\12354)6wY\"")

[Linearized tree]

u8R"2sS(あ)6wY"
~$ 
~$ ./TestRawStr 
R"^^(あああ)^^"
Parse Successful!

[Abstract Syntax]

LTest (Raw_string "R\"^^(\12354\12354\12354)^^\"")

[Linearized tree]

R"^^(あああ)^^"
~$

~$ ./TestRawStr

u8R"2sS(あ)6wY"

Parse Successful!

[Abstract Syntax]

LTest (Raw_string "u8R\"2sS(\12354)6wY\"")

[Linearized tree]

u8R"2sS(あ)6wY"

~$ ./TestRawStr

R"^^(あああ)^^"

Parse Successful!

[Abstract Syntax]

LTest (Raw_string "R\"^^(\12354\12354\12354)^^\"")

[Linearized tree]

R"^^(あああ)^^"

うまくパースしてくれているようで …

[haskell][BNFC]Unicodeスカラ値のサロゲートペア範囲考慮

it 2017年1月21日 2017年1月21日BNFC, haskell, プログラム, 字句解析 0

ユニコードスカラ値にはサロゲートペアのための領域0xD800-0xDFFFがあり、この範囲は文字を割り当てることがきない。このためcharの範囲からこの部分を外す。 [crayon-662cb7e314d8f829011 …

エスケープシーケンス２

it 2014年4月25日 2014年4月25日プログラム, 字句解析 0

先日のエスケープシーケンスBNFを修正。これでうまいこと字句解析できそうだ。

terminal-character = a | b | c;
first-quote-symbol = ';
escape-symbol = \;
terminal-string = first-quote-symbol (terminal-character* (escape-symbol first-quote-symbol)* terminal-character*)? first-quote-symbol;

terminal-character = a | b | c;

first-quote-symbol = ';

escape-symbol = \;

terminal-string = first-quote-symbol (terminal-character* (escape-symbol first-quote-symbol)* terminal-character*)? first-quote-symbol;

月	火	水	木	金	土	日
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30