erl_tokenize
所属分类：特征抽取
开发工具：Rust
文件大小：0KB
下载次数：0
上传日期：2023-08-05 10:04:34
上 传 者：
sh-1993
说明： 用Rust.编写的Erlang源代码标记器。，
(An Erlang source code tokenizer written in Rust.,)
文件列表:
Cargo.toml (669, 2023-08-05)
LICENSE-APACHE (11363, 2023-08-05)
LICENSE-MIT (1093, 2023-08-05)
examples/ (0, 2023-08-05)
examples/tokenize.rs (1045, 2023-08-05)
src/ (0, 2023-08-05)
src/error.rs (4511, 2023-08-05)
src/hidden_token.rs (2423, 2023-08-05)
src/lexer.rs (2293, 2023-08-05)
src/lexical_token.rs (7294, 2023-08-05)
src/lib.rs (1210, 2023-08-05)
src/position.rs (3306, 2023-08-05)
src/token.rs (13320, 2023-08-05)
src/tokenizer.rs (5030, 2023-08-05)
src/tokens.rs (41367, 2023-08-05)
src/util.rs (3544, 2023-08-05)
src/values.rs (6293, 2023-08-05)
tests/ (0, 2023-08-05)
tests/lib.rs (1691, 2023-08-05)
erl_tokenize
============
[](https://crates.io/crates/erl_tokenize)
[](https://docs.rs/erl_tokenize)
[](https://github.com/sile/erl_tokenize/actions)
[](https://coveralls.io/github/sile/erl_tokenize?branch=master)
Erlang source code tokenizer written in Rust.
[Documentation](https://docs.rs/erl_tokenize)
Examples
--------
Tokenizes the Erlang code `io:format("Hello").`:
```rust
use erl_tokenize::Tokenizer;
let src = r#"io:format("Hello")."#;
let tokenizer = Tokenizer::new(src);
let tokens = tokenizer.collect::, _>>().unwrap();
assert_eq!(tokens.iter().map(|t| t.text()).collect::>(),
["io", ":", "format", "(", r#""Hello""#, ")", "."]);
```
Executes the example `tokenize` command:
```bash
$ cargo run --example tokenize -- /dev/stdin < qux.
EOS
[Position { filepath: None, offset: 0, line: 1, column: 1 }] Symbol(Hyphen)
[Position { filepath: None, offset: 1, line: 1, column: 2 }] Atom("module")
[Position { filepath: None, offset: 7, line: 1, column: 8 }] Symbol(OpenParen)
[Position { filepath: None, offset: 8, line: 1, column: 9 }] Atom("foo")
[Position { filepath: None, offset: 11, line: 1, column: 12 }] Symbol(CloseParen)
[Position { filepath: None, offset: 12, line: 1, column: 13 }] Symbol(Dot)
[Position { filepath: None, offset: 13, line: 1, column: 14 }] Whitespace(Newline)
[Position { filepath: None, offset: 14, line: 2, column: 1 }] Whitespace(Newline)
[Position { filepath: None, offset: 15, line: 3, column: 1 }] Symbol(Hyphen)
[Position { filepath: None, offset: 16, line: 3, column: 2 }] Atom("export")
[Position { filepath: None, offset: 22, line: 3, column: 8 }] Symbol(OpenParen)
[Position { filepath: None, offset: 23, line: 3, column: 9 }] Symbol(OpenSquare)
[Position { filepath: None, offset: 24, line: 3, column: 10 }] Atom("bar")
[Position { filepath: None, offset: 27, line: 3, column: 13 }] Symbol(Slash)
[Position { filepath: None, offset: 28, line: 3, column: 14 }] Integer(BigUint { data: [] })
[Position { filepath: None, offset: 29, line: 3, column: 15 }] Symbol(CloseSquare)
[Position { filepath: None, offset: 30, line: 3, column: 16 }] Symbol(CloseParen)
[Position { filepath: None, offset: 31, line: 3, column: 17 }] Symbol(Dot)
[Position { filepath: None, offset: 32, line: 3, column: 18 }] Whitespace(Newline)
[Position { filepath: None, offset: 33, line: 4, column: 1 }] Whitespace(Newline)
[Position { filepath: None, offset: 34, line: 5, column: 1 }] Atom("bar")
[Position { filepath: None, offset: 37, line: 5, column: 4 }] Symbol(OpenParen)
[Position { filepath: None, offset: 38, line: 5, column: 5 }] Symbol(CloseParen)
[Position { filepath: None, offset: 39, line: 5, column: 6 }] Whitespace(Space)
[Position { filepath: None, offset: 40, line: 5, column: 7 }] Symbol(RightArrow)
[Position { filepath: None, offset: 42, line: 5, column: 9 }] Whitespace(Space)
[Position { filepath: None, offset: 43, line: 5, column: 10 }] Atom("qux")
[Position { filepath: None, offset: 46, line: 5, column: 13 }] Symbol(Dot)
[Position { filepath: None, offset: 47, line: 5, column: 14 }] Whitespace(Newline)
TOKEN COUNT: 29
ELAPSED: 0.007222 seconds
```
References
----------
- [erl_scan](http://erlang.org/doc/man/erl_scan.html) module
- [Erlang Data Types](http://erlang.org/doc/reference_manual/data_types.html)
近期下载者：
相关文件：
收藏者：