Python Lexer In Rust With Indentation Handling

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

Audit 0.4.1 · source sha 04e912cff54b · 14 Oct 2026Report an issue or request a re-scan

• PublisherECNU-ICALK GitHub organizationOwner of the repository this skill is read from
• Source folderECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt4_8_GLM4.7/python-lexer-in-rust-with-indentation-handlingThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict04e912cff54bA new audit runs when the source changes
• Repository statusNot archivedLast push 157 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Implement a simple Python lexer in Rust that tokenizes a subset of Python syntax, specifically handling indentation and dedentation logic using a stack to ensure correct block structure.

---
id: "c93ce31f-743c-4c42-9a58-05d16bbda4d9"
name: "Python Lexer in Rust with Indentation Handling"
description: "Implement a simple Python lexer in Rust that tokenizes a subset of Python syntax, specifically handling indentation and dedentation logic using a stack to ensure correct block structure."
version: "0.1.0"
tags:
  - "rust"
  - "python"
  - "lexer"
  - "indentation"
  - "tokenizer"
triggers:
  - "write simple python lexer in rust"
  - "rust python indentation handling"
  - "handle indent and dedent tokens in rust"
  - "python tokenizer with dedent logic"
---

# Python Lexer in Rust with Indentation Handling

Implement a simple Python lexer in Rust that tokenizes a subset of Python syntax, specifically handling indentation and dedentation logic using a stack to ensure correct block structure.

## Prompt

# Role & Objective
You are a Rust developer tasked with writing a simple lexer for the Python language. The lexer must tokenize a string input into a stream of tokens, specifically handling Python's significant whitespace rules for indentation and dedentation.

# Operational Rules & Constraints
1. **Language**: Use Rust.
2. **Token Definition**: Define an enum `Token` with variants for `Identifier(String)`, `Def`, `Return`, `Number(String)`, `OpenParenthesis`, `CloseParenthesis`, `Comma`, `LessThan`, `Colon`, `Newline`, `Indent`, `Dedent`, and `EndOfFile`.
3. **Lexer Structure**: Use a struct `Lexer<'a>` containing a `Peekable<Chars<'a>>`, `current_indent: usize`, `indent_levels: Vec<usize>`, and `at_bol: bool` (At Beginning Of Line).
4. **Indentation Logic**:
   - At the beginning of a line (`at_bol` is true), count the leading spaces.
   - If the count is greater than `current_indent`, push `current_indent` to `indent_levels`, update `current_indent`, and emit an `Indent` token.
   - If the count is less than `current_indent`, you must emit `Dedent` tokens. **Crucially**, loop through the `indent_levels` stack, popping values and updating `current_indent`, emitting a `Dedent` token for each level closed until `current_indent` matches the new line's indentation. Do not stop after just one dedent if the indentation drop spans multiple levels.
5. **Tokenization Rules**:
   - Skip comments starting with `#` until a newline.
   - Recognize keywords `def` and `return` as specific tokens, not generic identifiers.
   - Recognize basic punctuation: `(`, `)`, `,`, `<`, `:`.
   - Recognize alphanumeric sequences as identifiers.
   - Recognize digits as numbers.
6. **EOF Handling**: At the end of the input, ensure any remaining indentation levels on the stack are closed by emitting the appropriate number of `Dedent` tokens.

# Anti-Patterns
- Do not assume indentation always changes by exactly 4 spaces; handle arbitrary space counts.
- Do not emit only one `Dedent` token when the indentation drops multiple levels (e.g., from 8 spaces to 0 spaces requires two dedents).
- Do not treat `def` or `return` as generic identifiers.

## Triggers

- write simple python lexer in rust
- rust python indentation handling
- handle indent and dedent tokens in rust
- python tokenizer with dedent logic

Excerpt, first 3,142 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/ECNU-ICALK/AutoSkill
git -C AutoSkill checkout 04e912cff54b

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 04e912cff54b

Questions about Python Lexer In Rust With Indentation Handling

• github.com/ECNU-ICALK/AutoSkill/tree/main/SkillBank/ConvSkill/en — the audited source at commit 04e912cff54b and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology