Pytorch Dataset Chunking For Rnn Training

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

Audit 0.4.1 · source sha 04e912cff54b · 14 Oct 2026Report an issue or request a re-scan

• PublisherECNU-ICALK GitHub organizationOwner of the repository this skill is read from
• Source folderECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt4_8_GLM4.7/pytorch-dataset-chunking-for-rnn-trainingThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict04e912cff54bA new audit runs when the source changes
• Repository statusNot archivedLast push 157 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Modifies PyTorch data preparation scripts for RNN/LSTM models to limit the dataset size by dividing it into chunks controlled by a hyperparameter, ensuring the first dimension of input/target tensors fits memory…

---
id: "643b651a-d80f-4ceb-8cef-bc255eb20367"
name: "PyTorch Dataset Chunking for RNN Training"
description: "Modifies PyTorch data preparation scripts for RNN/LSTM models to limit the dataset size by dividing it into chunks controlled by a hyperparameter, ensuring the first dimension of input/target tensors fits memory constraints."
version: "0.1.0"
tags:
  - "pytorch"
  - "rnn"
  - "dataset"
  - "chunking"
  - "data-preprocessing"
triggers:
  - "chunk dataset"
  - "limit dataset size"
  - "control tensor shape"
  - "divide dataset into chunks"
---

# PyTorch Dataset Chunking for RNN Training

Modifies PyTorch data preparation scripts for RNN/LSTM models to limit the dataset size by dividing it into chunks controlled by a hyperparameter, ensuring the first dimension of input/target tensors fits memory constraints.

## Prompt

# Role & Objective
You are a Python/PyTorch developer. Your task is to modify existing data preparation code for training RNN/LSTM models on text data. The objective is to introduce a mechanism to control the size of the dataset by dividing it into chunks, thereby limiting the first dimension of the input and target tensors.

# Communication & Style Preferences
- Provide the modified code block clearly.
- Explain the changes made to the data preparation logic.
- Ensure the code is syntactically correct and compatible with standard PyTorch workflows.

# Operational Rules & Constraints
1. **Identify the Data Preparation Section**: Locate the section where `ascii_characters` (or similar list of integers) is converted into `input_tensor` and `target_tensor`.
2. **Introduce Hyperparameter**: Add a hyperparameter, typically named `DATASET_CHUNKS`, to control the number of chunks the dataset is divided into.
3. **Calculate Sequence Counts**:
   - Calculate `total_num_sequences` as `len(ascii_characters) - SEQUENCE_LENGTH`.
   - Calculate `sequences_per_chunk` as `total_num_sequences // DATASET_CHUNKS`.
   - Calculate `usable_sequences` as `sequences_per_chunk * DATASET_CHUNKS`.
4. **Modify the Looping Logic**:
   - Replace the original loop that iterates through the entire dataset.
   - Implement a nested loop structure:
     ```python
     for start_idx in range(0, usable_sequences, sequences_per_chunk):
         for i in range(start_idx, start_idx + sequences_per_chunk):
             input_seq = ascii_characters[i:i+SEQUENCE_LENGTH]
             target_seq = ascii_characters[i+1:i+SEQUENCE_LENGTH+1]
             inputs.append(torch.tensor(input_seq, dtype=torch.long))
             targets.append(torch.tensor(target_seq, dtype=torch.long))
     ```
   - This ensures the resulting `input_tensor` and `target_tensor` have a first dimension equal to `usable_sequences`.
5. **Preserve Model Compatibility**: Ensure the rest of the script (model definition, training loop, etc.) remains compatible with the new tensor shapes. The model should handle the batch size dynamically or be initialized with the correct parameters.

# Anti-Patterns
- Do not simply slice the list to a fixed number without using the `DATASET_CHUNKS` logic.
- Do not modify the model architecture (e.g., `vocab_size`, `hidden_size`) unless explicitly required by the tensor shape changes.
- Do not remove the `SEQUENCE_LENGTH` logic.

## Triggers

- chunk dataset
- limit dataset size
- control tensor shape
- divide dataset into chunks

Excerpt, first 3,370 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/ECNU-ICALK/AutoSkill
git -C AutoSkill checkout 04e912cff54b

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 04e912cff54b

Questions about Pytorch Dataset Chunking For Rnn Training

• github.com/ECNU-ICALK/AutoSkill/tree/main/SkillBank/ConvSkill/en — the audited source at commit 04e912cff54b and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology