Radiation therapy treatment planning is one of the most critical steps in modern cancer care. A skilled human planner must define dozens of parameters that control how radiation is delivered, balancing maximum dose to the tumor against minimum damage to surrounding healthy organs.
These parameters, called treatment planning parameters (TPPs), include dose limits, weighting factors, and volume constraints for the tumor target and organs at risk (OARs) such as the bladder and rectum in prostate cancer.
The current process is labor-intensive and highly subjective. Plan quality depends heavily on the experience of the individual planner and how much time is available. The planner repeatedly adjusts parameters, re-runs the optimization engine, and evaluates the resulting dose distribution until satisfied with the result.
Fully automated treatment planning is therefore a major goal in radiation oncology, promising more consistent plan quality and freeing clinical staff for other tasks.
Researchers at UT Southwestern Medical Center had previously developed an Intelligent Automatic Treatment Planning (IATP) framework that trained a Virtual Treatment Planner Network (VTPN) to mimic a human planner's behavior when operating a treatment planning system.
However, the original VTPN had a critical limitation: its network size grew nearly linearly with the number of planning parameters. For simple problems with five parameters this was manageable, but for complex treatments requiring 48 or more parameters, the computational demands became prohibitive.
This paper introduces a new Hierarchical VTPN (HieVTPN) that mimics how a human planner actually thinks: first deciding which structure needs attention (bladder, rectum, or tumor target), then which specific parameter to adjust, then what direction to change it.
This hierarchical approach keeps the network size essentially constant regardless of how many parameters are involved, enabling the system to scale to complex, clinically realistic treatment scenarios.
The HieVTPN consists of three networks that activate one after another in each planning step. Structure-Net first reviews the current dose distribution and decides which anatomical structure (tumor, bladder, rectum) most needs adjustment.
Parameter-Net then receives the selected structure and determines which specific planning parameter for that structure should be changed, such as the dose limit or the weighting factor that controls how strongly that structure is prioritized in optimization.
Action-Net finally decides the direction of change: whether to increase or decrease the selected parameter. Together, these three decisions define one planning step, after which the system re-runs the optimization engine and evaluates the result.
The system was tested on two prostate cancer scenarios: a standard IMRT plan (conventional fractionation, 5 planning parameters) and a more complex SBRT plan (stereotactic body radiation therapy, hypofractionation, 48 planning parameters), the latter resembling what would be used in clinical practice with commercial systems.
The networks were trained using deep reinforcement learning (DRL), a technique where an AI learns by taking actions and receiving rewards based on the outcomes. A positive reward is given when a planning step improves the overall plan quality score, and a negative reward when it makes the plan worse.
The training algorithm follows a Q-learning framework, which teaches the system to choose actions that maximize total future reward, not just the immediate next step. This encourages the AI to make strategically smart decisions rather than short-sighted ones.
A key innovation was the hierarchical Q-learning (HieDRL) scheme that allows all three networks to be trained simultaneously in an end-to-end fashion, even though their outputs depend on each other. The three networks are updated alternately until they converge on stable decision-making behavior.
Training used only 10 patient cases, which was sufficient because the reinforcement learning process automatically generates a large number of training scenarios through its trial-and-error exploration, described using an epsilon-greedy algorithm that gradually shifts from random exploration to learned exploitation.
For prostate IMRT, HieVTPN achieved an average plan score of 8.62 out of 9 on 59 test patients, closely matching the previous VTPN (8.45) and comparable to an experienced human planner's average of approximately 8.5. The entire planning process took about 3 minutes per patient.
For the more complex prostate SBRT task involving 48 parameters, HieVTPN achieved an average plan score of 139.07 out of 150, outperforming the average score of 132.21 achieved by experienced human planners who competed in the 2016 international AAMD/RSS Plan Study challenge.
The starting plans (with all parameters set to default values) scored only 95.56 on average for SBRT, demonstrating that HieVTPN's parameter adjustments produced a major improvement in plan quality, moving from mediocre initial conditions to near-optimal outcomes.
Critically, the original VTPN could not even be tested on the SBRT problem because its computational demands exceeded the workstation's memory capacity, while HieVTPN handled the 48-parameter problem without difficulty, confirming its scalability advantage.
Researchers analyzed the AI's decision-making behavior by feeding it thousands of randomly generated treatment plans and observing its responses. The results showed that HieVTPN's reasoning aligned with experienced human planner intuition.
When the OAR score (bladder and rectum protection) was poor, the AI consistently prioritized adjusting OAR parameters first. Once the OAR score reached an acceptable level, it shifted focus to improving tumor coverage (PTV). This is exactly the sequence an experienced planner would follow.
When choosing between bladder and rectum, the AI tended to address whichever organ had the worse score, working to balance the two in a clinically sensible manner. At high OAR scores, it strategically allowed slight relaxation to unlock gains in tumor dose coverage.
These decision patterns were spontaneously learned during training with no explicit rules programmed in. The emergence of clinically sensible behavior from pure reinforcement learning validates the approach and builds confidence for future clinical deployment.
The original parallel VTPN required one dedicated sub-network for each adjustable parameter. With 5 parameters, this was manageable. With 48 parameters, the VTPN needed 16 times as many network components, making training practically infeasible.
HieVTPN's three-network hierarchical structure is largely independent of the number of parameters. Going from 5 to 48 parameters increased HieVTPN's total network size by only about 0.1%, compared to a 16-fold increase for the conventional VTPN.
This architectural advantage means that HieVTPN could theoretically handle planning problems far more complex than those tested, including treatments for sites like head and neck cancer that may involve many more planning structures and parameters than prostate cancer.
The study used in-house developed treatment planning systems rather than commercially available software like Varian Eclipse. Future work must demonstrate that the framework can interface with commercial systems used in real clinical settings.
The reward functions were derived from existing scoring criteria (the ProKnow system). These may not fully capture the nuanced clinical judgments that radiation oncologists make, such as patient-specific anatomical considerations or quality-of-life priorities beyond dose metrics.
The framework also currently lacks a roll-back mechanism, which would allow the system to undo poor decisions and return to a better previous plan state. Human planners routinely do this, and adding this capability is identified as an important future direction.
Despite these limitations, the successful demonstration across both standard IMRT and complex SBRT scenarios, combined with decision-making behavior that mirrors clinical practice, suggests strong potential for clinical translation of hierarchical AI-based treatment planning.