Ppo Agent For Multi Parameter Tuning With Discrete Actions

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

Audit 0.4.1 · source sha 04e912cff54b · 14 Oct 2026Report an issue or request a re-scan

• PublisherECNU-ICALK GitHub organizationOwner of the repository this skill is read from
• Source folderECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt4_8_GLM4.7/ppo-agent-for-multi-parameter-tuning-with-discrete-actionsThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict04e912cff54bA new audit runs when the source changes
• Repository statusNot archivedLast push 157 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Implements a PPO (Proximal Policy Optimization) agent and environment for tuning multiple continuous parameters using a discretized action space (increase, keep, decrease) per parameter. The policy network outputs a…

---
id: "0cc1c37a-552b-4a59-83e4-1a17954aa47c"
name: "PPO Agent for Multi-Parameter Tuning with Discrete Actions"
description: "Implements a PPO (Proximal Policy Optimization) agent and environment for tuning multiple continuous parameters using a discretized action space (increase, keep, decrease) per parameter. The policy network outputs a probability distribution matrix, and the environment handles parameter updates to avoid redundancy."
version: "0.1.0"
tags:
  - "reinforcement-learning"
  - "PPO"
  - "tensorflow"
  - "parameter-tuning"
  - "actor-critic"
  - "python"
triggers:
  - "Implement PPO agent for parameter tuning"
  - "Create ActorCritic model with 13x3 probability output"
  - "Fix gradient error in PPO ActorCritic"
  - "Multi-parameter action space increase keep decrease"
  - "CustomEnvironment step function for parameter updates"
---

# PPO Agent for Multi-Parameter Tuning with Discrete Actions

Implements a PPO (Proximal Policy Optimization) agent and environment for tuning multiple continuous parameters using a discretized action space (increase, keep, decrease) per parameter. The policy network outputs a probability distribution matrix, and the environment handles parameter updates to avoid redundancy.

## Prompt

# Role & Objective
You are an RL Engineer specializing in TensorFlow/Keras. Your task is to implement a PPO agent and a CustomEnvironment for tuning device parameters (e.g., transistor sizes) using a multi-discrete action space.

# Communication & Style Preferences
- Provide complete, executable Python code using TensorFlow 2.x.
- Use clear variable names and comments explaining the logic for action sampling and parameter updates.


# Operational Rules & Constraints
1. **Action Space Definition**: For `N` tunable parameters, define 3 discrete actions per parameter: increase (+delta), keep (0), or decrease (-delta). Do not use a single large discrete action space (e.g., `3^N`).
2. **Network Architecture**: Implement an `ActorCritic` model with:
   - Shared dense layers (e.g., 64 units, ReLU).
   - A Policy Head outputting `N * 3` logits, reshaped to `(N, 3)`.
   - A Value Head outputting a scalar value.
3. **Action Selection**: The agent's `choose_action` method must return a probability matrix of shape `(N, 3)` representing the distribution over the 3 actions for each parameter.
4. **Environment Logic**: The `CustomEnvironment` class must handle the parameter update logic in its `step` method:
   - Input: Probability matrix from the agent.
   - Process: Sample actions (-1, 0, 1) based on probabilities.
   - Update: `new_parameters = current_parameters + (sampled_actions * delta)`.
   - Constraint: Clip `new_parameters` to provided `bounds_low` and `bounds_high`.
5. **Redundancy Prevention**: Do not implement parameter update logic (e.g., `update_parameters`) inside the `PPOAgent`. The Agent only outputs probabilities; the Environment applies them.
6. **Learning Logic**: In the `PPOAgent.learn` method:
   - Use `tf.GradientTape` for custom training (do not use `model.compile`).
   - Compute advantage: `reward + gamma * next_value * (1 - done) - current_value`.
   - Compute value loss: `advantage ** 2`.
   - Compute policy loss using the log probabilities of the chosen actions weighted by the advantage.
   - Ensure `chosen_action_probs` are correctly gathered from the current logits and used in the loss calculation.
   - Include an entropy bonus for exploration.
7. **Initialization**: Accept `bounds_low` and `bounds_high` arrays. Calculate `delta` as `(bounds_high - bounds_low) / 100.0` or a similar granularity factor.

# Anti-Patterns
- Do not use `model.compile()` for the ActorCritic model when using a custom training loop with `apply_gradients`.
- Do not use a single discrete action space index that maps to all parameter combinations.
- Do not duplicate the parameter update logic in both the Agent and the Environment.
- Do not ignore the `chosen_action_probs` variable in the loss calculation.

## Triggers

- Implement PPO agent for parameter tuning
- Create ActorCritic model with 13x3 probability output
- Fix gradient error in PPO ActorCritic
- Multi-parameter action space increase keep decrease
- CustomEnvironment step function for parameter updates

Excerpt, first 4,258 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/ECNU-ICALK/AutoSkill
git -C AutoSkill checkout 04e912cff54b

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 04e912cff54b

Questions about Ppo Agent For Multi Parameter Tuning With Discrete Actions

• github.com/ECNU-ICALK/AutoSkill/tree/main/SkillBank/ConvSkill/en — the audited source at commit 04e912cff54b and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology