crates/stdlib/src/_tokenize.rs (2)

396-419: Extract magic numbers 60 and 63 to named constants.

The function uses magic numbers 60 (FSTRING_MIDDLE) and 63 (TSTRING_MIDDLE) directly. For consistency with the other token constants defined at the top of the module (lines 25-29), consider adding named constants.

♻️ Suggested refactor

Add to the constants section (after line 29):

const TOKEN_FSTRING_MIDDLE: u8 = 60;
const TOKEN_TSTRING_MIDDLE: u8 = 63;

Then update the function:

                 TokenKind::FStringStart => {
                     if depth == 0 {
-                        return 60; // FSTRING_MIDDLE
+                        return TOKEN_FSTRING_MIDDLE;
                     }
                     depth -= 1;
                 }
                 TokenKind::TStringStart => {
                     if depth == 0 {
-                        return 63; // TSTRING_MIDDLE
+                        return TOKEN_TSTRING_MIDDLE;
                     }
                     depth -= 1;
                 }
                 _ => {}
             }
         }
-        60 // default to FSTRING_MIDDLE
+        TOKEN_FSTRING_MIDDLE // default

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@crates/stdlib/src/_tokenize.rs` around lines 396 - 419, The function
find_fstring_middle_type currently returns magic numbers 60 and 63; define named
constants (e.g., TOKEN_FSTRING_MIDDLE and TOKEN_TSTRING_MIDDLE) alongside the
other token constants at the top of the module and replace the literal returns
(and the final default 60) with those constants so the function returns
TOKEN_FSTRING_MIDDLE or TOKEN_TSTRING_MIDDLE instead of raw numbers.

---

107-149: Consider the memory overhead of cloning TokenizerState.

The clone-modify-write pattern (line 109 reads/clones, line 140 writes back) avoids holding the lock during potentially slow readline() calls. However, in the Yielding phase, this clones the entire source string and token vectors on each next() call, which could be expensive for large files.

A more efficient approach would be to hold the write lock directly in the Yielding phase (since no external calls are made):

♻️ Suggested optimization (optional)

     impl IterNext for PyTokenizerIter {
         fn next(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyIterReturn> {
-            let mut state = zelf.state.read().clone();
-
             loop {
-                match &mut state.phase {
-                    TokenizerPhase::Reading { source } => {
+                // Check phase without holding lock
+                let is_reading = matches!(&zelf.state.read().phase, TokenizerPhase::Reading { .. });
+                
+                if is_reading {
+                    let mut state = zelf.state.write();
+                    if let TokenizerPhase::Reading { source } = &mut state.phase {
                         let line = zelf.readline(vm)?;
                         // ... rest of reading logic
-                    }
-                    TokenizerPhase::Yielding { .. } => {
-                        let result =
-                            emit_next_token(&mut state, zelf.extra_tokens, vm)?;
-                        *zelf.state.write() = state;
-                        return Ok(result);
                     }
+                } else {
+                    let mut state = zelf.state.write();
+                    return emit_next_token(&mut state, zelf.extra_tokens, vm);
                 }
             }
         }
     }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@crates/stdlib/src/_tokenize.rs` around lines 107 - 149, The current
PyTokenizerIter::next clones the entire TokenizerState at the start
(zelf.state.read().clone()), which causes repeated heavy clones when in
TokenizerPhase::Yielding; instead, only clone/read for the Reading branch but
for the Yielding branch acquire a write lock on zelf.state (i.e., get a write
guard) and call emit_next_token with a mutable reference into that guarded state
(TokenizerPhase::Yielding) so you avoid cloning source/tokens on each next();
keep the readline path unchanged (read/clone/modify/write) but switch to holding
the write lock inside the TokenizerPhase::Yielding arm before calling
emit_next_token and returning its PyIterReturn.