Understanding the Impact of Entropy on Policy Optimization

Entropy regularization shows up in almost every policy gradient codebase, and the standard explanation is exploration: a random policy keeps trying things. This paper asks what that means for the optimization itself. A trick like entropy can only help in two ways: it makes the gradient estimate less noisy, or it changes the objective being climbed. So the authors remove the noise entirely, with exact gradients in a small gridworld, and entropy still helps. Their answer is that a more random policy smooths the landscape. Flat spots get a slope, valleys between solutions fill in, and the curvature stops swinging so wildly that only tiny learning rates are safe.

It's an analysis paper: no new algorithm, a cheap probe for gradient and curvature, and a set of qualitative experiments. The probe runs on second-order calculus, which is where most of my time went, so that part is written out in full. My notes are on arXiv v5.

Setup

• Gridworld (exact gradients): a 5×5 grid, starting top-left, with a reward of 4.5 in the top-right corner and 5.0 in the bottom-left. The policy is a softmax, $\pi (a\mid s)\propto \exp ({\theta }^{\top }s)$, over four actions with one-hot states, so it's effectively tabular. The dynamics are known, so the gradient is computed exactly and there's no sampling noise at all. There are two local optima: always go right (${\pi }_{\text{sub}}$) or always go down (${\pi }_{\text{opt}}$). The objective with no entropy bonus, $\tau =0$, is called the true objective.
• MuJoCo (sampled gradients): Hopper, Walker2d, and HalfCheetah with a Gaussian policy whose mean is linear, ${\theta }^{\top }{s}_t$, and whose standard deviation $\sigma$ is fixed. A Gaussian's entropy depends only on $\sigma$, so $\sigma$ is the entropy knob, and $\sigma =0$ is the true objective. Gradients come from REINFORCE with 128 trajectories per batch and a batch-average baseline. The large batches keep the variance-reduction effect of a larger $\sigma$ out of the comparison, and performance is always evaluated deterministically. Learning rates are tuned separately for each $\sigma$, with five seeds each.
• Landscape probes: straight-line interpolation between two parameter vectors, and the random-perturbation method below, with about 1000 directions per parameter vector and 512 rollouts per evaluation.

Background: Jacobian, Hessian, eigenvalues

The probe is a second-order Taylor expansion in many dimensions. This is the calculus I needed straight before Section 2.1.

Start with one input. For $f(x)=x^2$ at $x=3$, the slope is $f^{\prime }(3)=6$ and the bend is $f^{\prime \prime }(3)=2$. Together they give the first- and second-order predictions:

$$f(x+\Delta )\approx f(x)+f^{\prime }(x)\Delta ,f(x+\Delta )\approx f(x)+f^{\prime }(x)\Delta +\frac{1}{2}{f}^{\prime \prime }(x){\Delta }^2$$

For $x^2$ the second one is exact: $(3+\Delta {)}^2=9+6\Delta +{\Delta }^2$. Everything below generalizes slope and bend to many inputs.

The Jacobian is a table of slopes. For outputs $f_1,\dots ,f_m$ and inputs $x_1,\dots ,x_n$, it's the $m\times n$ matrix whose row $i$ holds the slopes of $f_i$:

$$J=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \cdots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

Take a rectangle with inputs width $w$ and height $h$, and two outputs, area $A=wh$ and perimeter $P=2w+2h$:

$$J(w,h)=\left[\begin{array}{cc}\frac{\partial A}{\partial w}& \frac{\partial A}{\partial h}\\ \frac{\partial P}{\partial w}& \frac{\partial P}{\partial h}\end{array}\right]=\left[\begin{array}{cc}h& w\\ 2& 2\end{array}\right],J(3,2)=\left[\begin{array}{cc}2& 3\\ 2& 2\end{array}\right]$$

Its use is prediction: $f(p+\Delta )\approx f(p)+J\Delta$, with $\Delta =(\Delta w,\Delta h)$. Growing both sides of a 3×2 rectangle by 0.1 gives $J\Delta =(0.5,0.4)$, so the area comes out at about 6.5 (the true value is 6.51) and the perimeter at 10.4, exactly, because $P$ is linear.

The gradient is the Jacobian of a single output, written as a column: $\nabla A=(\partial A/\partial w,\partial A/\partial h)=(h,w)$.

The Hessian is every second derivative of a single output, an $n\times n$ matrix with $H_{ij}={\partial }^{2}f/\partial x_i\partial x_j$:

$$H=\left[\begin{array}{ccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right]$$

Mixed partials don't depend on order, so $H$ is symmetric. For the area,

$$H_A=\left[\begin{array}{cc}\frac{{\partial }^{2}A}{\partial w^2}& \frac{{\partial }^{2}A}{\partial w\partial h}\\ \frac{{\partial }^{2}A}{\partial h\partial w}& \frac{{\partial }^{2}A}{\partial h^2}\end{array}\right]=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]$$

and the second-order prediction is

$$f(p+\Delta )\approx f(p)+\nabla f(p{)}^{\top }\Delta +\frac{1}{2}{\Delta }^{\top }H(p)\Delta$$

That recovers the missing 0.01 exactly: $\frac{1}{2}{\Delta }^{\top }{H}_A\Delta =\Delta w\Delta h$, the corner piece of $(w+\Delta w)(h+\Delta h)$. The short version: the Jacobian is slope, the Hessian is bend.

Eigenvectors are the directions a matrix doesn't turn. A matrix takes an arrow in and gives an arrow out, usually pointing somewhere new. For a few special directions it only stretches, shrinks, or flips the arrow: $Mv=\lambda v$, the same arrow times $\lambda$. For example,

$$M=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]:(1,0)\mapsto (2,1),(1,1)\mapsto (3,3),(1,-1)\mapsto (1,-1)$$

The first arrow gets turned. The second comes back 3 times longer, so $(1,1)$ is an eigenvector with eigenvalue 3, and $(1,-1)$ comes back unchanged, with eigenvalue 1.

Eigenvalues are the extreme bends. That $M$ happens to be the Hessian of $f(x,y)=x^2+xy+y^2$. The bend along a unit direction $d$ is $d^{\top }Hd$, and for a symmetric $H$ with orthonormal eigenvectors $v_i$, it's a weighted average of the eigenvalues:

$$d^{\top }Hd=\sum _i{\lambda }_i(v_i^{\top }d{)}^2,\sum _i(v_i^{\top }d{)}^2=1$$

So every direction's bend lies between ${\lambda }_{\min }$ and ${\lambda }_{\max }$, reaching them only along the eigenvectors. For $M$ that's 3 along $(1,1)/\sqrt{2}$, 1 along $(1,-1)/\sqrt{2}$, and 2 along $(1,0)$, an even mix of the two. Where the gradient is zero, the signs classify the point: all eigenvalues negative is a local maximum, all positive is a local minimum, mixed signs is a saddle (up in some directions, down in others), and zero eigenvalues mark flat directions.

The random-perturbation probe

Plotting the objective along one straight line (Section 2.1.1) has a known weakness: two optima that look separated by a valley on the line can be joined by a curved path of equal value. So the paper adds a local probe (Section 2.1.2) that classifies a point as a local optimum, saddle, or flat region, and tracks curvature during training.

Sample directions $d$ at random. The paper says "on the unit ball," but the unit ball is every vector with $\Vert d\Vert \le 1$, and a direction has length exactly 1, so it's really the unit sphere, the surface of the ball. Step both ways from ${\theta }_0$ by some $\alpha$ and record the change:

$${\Delta }_d^{\mathcal{O}+}=\mathcal{O}({\theta }_0+\alpha d)-\mathcal{O}({\theta }_0),{\Delta }_d^{\mathcal{O}-}=\mathcal{O}({\theta }_0-\alpha d)-\mathcal{O}({\theta }_0)$$

Each direction becomes one dot, with ${\Delta }^{+}$ on the x-axis and ${\Delta }^{-}$ on the y-axis (dropping the extra sub- and superscripts from here on). To read the dots, expand $\mathcal{O}$ to second order around ${\theta }_0$ with steps $\pm \alpha d$:

$$\begin{array}{rl}{\Delta }^{+}& \approx \alpha \nabla \mathcal{O}({\theta }_0{)}^{\top }d+\frac{1}{2}{\alpha }^2{d}^{\top }Hd\\ {\Delta }^{-}& \approx -\alpha \nabla \mathcal{O}({\theta }_0{)}^{\top }d+\frac{1}{2}{\alpha }^2{d}^{\top }Hd\end{array}$$

The slope term flips sign with the step and the bend term doesn't, so subtracting keeps only the slope and adding keeps only the bend:

$${\Delta }^{+}-{\Delta }^{-}=2\alpha \nabla \mathcal{O}({\theta }_0{)}^{\top }d(1)$$ $${\Delta }^{+}+{\Delta }^{-}={\alpha }^2{d}^{\top }Hd(2)$$

These are central finite differences along $d$. On the scatter plot, a dot's position along the $y=-x$ diagonal is the slope in that direction, and its position along $y=x$ is the bend. A histogram of (2) over many directions is the curvature spectrum, which by the eigenvalue fact above lives between ${\alpha }^2{\lambda }_{\min }$ and ${\alpha }^2{\lambda }_{\max }$.

What the dots do	Point type	In Hessian terms
Both negative for every $d$	Local maximum	All eigenvalues negative
Both positive for every $d$	Local minimum	All eigenvalues positive
Both negative for some $d$, both positive for others	Saddle point	Eigenvalues of both signs
${\Delta }^{+}\approx -{\Delta }^{-}$	Almost linear region	Slope, but little bend
${\Delta }^{+}\approx {\Delta }^{-}\approx 0$	Almost flat region	Almost no slope or bend

With finitely many directions, these are probabilities rather than certainties.

The paper's toy, checked by hand. The running example is $\mathcal{O}(x,y)=-(1-xy{)}^2$. It's never positive and equals 0 on the whole curve $xy=1$, so its maxima form a curve, a manifold, and there's a saddle at the origin. The gradient is $2(1-xy)(y,x)$, which vanishes at both. The Hessian is

$$H(x,y)=\left[\begin{array}{cc}-2y^2& 2-4xy\\ 2-4xy& -2x^2\end{array}\right]$$

and at the two points the paper probes,

$$H(0,0)=\left[\begin{array}{cc}0& 2\\ 2& 0\end{array}\right],H(-0.5,-2)=\left[\begin{array}{cc}-8& -2\\ -2& -0.5\end{array}\right]$$

At the origin the eigenvalues are $\pm 2$: up along $(1,1)$, down along $(1,-1)$, a saddle. At the maximum $(-0.5,-2)$ they're $-8.5$ and $0$. The zero eigenvalue's eigenvector is $(1,-4)$, which is exactly the tangent to $xy=1$ at that point, so the flat direction runs along the manifold of maxima while the steep one, $(4,1)$, cuts across it. That's what Figures 1 and 2 show: at the maximum the dots sit in the bottom-left with some reaching zero and the curvatures are zero or negative, while at the saddle the curvatures come in both signs.

Policy optimization and the entropy bonus

${\mathcal{O}}_{ER}$ is expected return, the book's $J(\theta )$, in the notation of Nachum et al.'s path consistency learning paper, which shares two authors with this one. Splitting the expectation into the start state and everything after it turns it into the value of the start state, averaged over where you start:

$$\begin{array}{rl}{\mathcal{O}}_{ER}(\theta )& ={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=1}^{\infty }{\gamma }^t{r}_t\right]={\mathbb{E}}_{\xi }[R(\xi )]\\ & ={\mathbb{E}}_{s_0}\left[\mathbb{E}\right[R(\xi )\mid s_0\left]\right]={\mathbb{E}}_{s_0}[V^{\pi }(s_0)]\end{array}$$

Here $\xi$ is a whole trajectory, since $\tau$ is taken by the entropy temperature, and the sum starting at $t=1$ with ${\gamma }^t$ is an indexing quirk, so read it as the usual discounted return. The gradient is the policy gradient theorem, written with integrals so it covers continuous states and actions:

$${\nabla }_{\theta }{\mathcal{O}}_{ER}(\theta )={\int }_s{d}^{{\pi }_{\theta }}(s){\int }_a{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)Q^{{\pi }_{\theta }}(s,a)dads$$

With sums instead of integrals, that's the book's ${\sum }_s\mu (s){\sum }_a{q}_{\pi }(s,a)\nabla \pi (a\mid s,\theta )$, which keeps the same form in the episodic and continuing cases.

Entropy is the expected negative log-probability of the policy's own actions: large when the policy spreads out, 0 when it always picks one action. The paper adds it to the reward at every step, weighted by a temperature $\tau$:

$$\mathbb{H}(\pi (\cdot \mid s_t))={\mathbb{E}}_{a\sim \pi (\cdot \mid s_t)}[-\log \pi (a\mid s_t)],r_t^{\tau }=r_t+\tau \mathbb{H}(\pi (\cdot \mid s_t))$$

which gives

$$\begin{array}{rl}{\nabla }_{\theta }{\mathcal{O}}_{ENT}(\theta )={\int }_s{d}^{{\pi }_{\theta }}(s){\int }_a\pi (a\mid s)[& Q^{\tau ,{\pi }_{\theta }}(s,a){\nabla }_{\theta }\log \pi (a\mid s)\\ & +\tau {\nabla }_{\theta }\mathbb{H}(\pi (\cdot \mid s))]dads(3)\end{array}$$

Entropy regularization comes in two flavors that look alike:

• Bonus in the loss (A3C, PPO): add $\beta \nabla \mathbb{H}$ at each sampled state, with the return built from plain rewards. Entropy only counts at the current state.
• Bonus in the reward (maximum-entropy RL, SAC, this paper): $Q^{\tau }$ is a soft value that includes all future entropy, so the agent is also paid for heading toward states where it can stay random. SAC calls the temperature $\alpha$.

In expectation, A3C's update has the same shape as (3), and the only real difference is which $Q$. The $\tau \nabla \mathbb{H}$ term sits inside ${\int }_a\pi (a\mid s)$ only for tidiness, since it doesn't depend on $a$.

The paper's framing comes from one observation. In Q-learning, exploration is a separate module like ε-greedy, but a policy gradient method explores with the very policy it's optimizing, so any change to the policy is a change to the objective. A trick like entropy can therefore affect learning in only two ways: it reduces noise in the gradient estimate, or it changes the landscape. Changing $\theta$ also changes $d^{{\pi }_{\theta }}$, which states get visited, but the gradient never differentiates through $d^{{\pi }_{\theta }}$ directly; the policy gradient theorem folds that effect into $Q$.

Gridworld: hard even with the exact gradient

With exact gradients, about 25% of random initializations still end at the worse corner. Sampling noise can't be the cause, so the landscape is.

Figure 4 shows why. At ${\theta }_{\text{sub}}$, every sampled direction lands at the origin, with a negligible gradient and slightly negative bend (4a, black): a plateau with nothing to follow. On the straight line from ${\theta }_{\text{sub}}$ to ${\theta }_{\text{opt}}$, the true objective has two plateaus with a sharp valley between them (4b, black). The valley matters not because you're in it but because it sits between you and the better solution. Gradient ascent only moves uphill and can't see what's on the other side.

Adding entropy and decaying it during training drops the rate of bad solutions to zero (Figure S5), and Figure 4 shows two changes. With $\tau =0.01$, the dots at ${\theta }_{\text{sub}}$ spread along $y=-x$ (4a, blue and orange), so every direction now has an uphill side and it's no longer a flat solution. Along the interpolation, entropy alone shrinks the valley (blue), and entropy plus a more random policy nearly erases it (orange).

"A more random policy" is the Mix knob. The evaluated policy is $(1-\text{Mix}){\pi }_{\theta }+\text{Mix}/|\mathcal{A}|$, so every action keeps at least $\text{Mix}/|\mathcal{A}|$ probability; with four actions and Mix = 0.8, that's 20% each. $\tau$ rewards randomness, while Mix forces it.

One thing tripped me up in 4a: orange isn't better than blue there, just tighter, and the caption groups the two together. With Mix = 0.8, $\theta$ controls only 20% of the policy, so any step in $\theta$ moves the objective less. Orange's real advantage is in 4b.

MuJoCo: more randomness, smoother objective

Here $\sigma$ plays the role that $\tau$ and Mix played in the gridworld.

Faster learning. Higher $\sigma$ learns faster in all three tasks, and in Hopper and Walker2d the final reward is 2 to 8 times that of $\sigma =0.1$ (Figure 5). It isn't a clean ranking, though. $\sigma =0.75$ beats $1.0$ in Hopper and HalfCheetah, and in HalfCheetah $0.1$ isn't the worst. The authors only claim the rough trend.

Why faster: steadier curvature. The clue is the learning rate. In Hopper and Walker2d the best constant rate grows with $\sigma$, ten times larger at $\sigma =1$ than at $0.1$. The safe step size depends on the bend, and a constant rate has to survive the sharpest bend along the way, like driving at one fixed speed on a road with a single hairpin turn. Rescaling the parameters can't fully explain it, since the reward also climbs faster at higher $\sigma$. Figure 6 measures the bend in the best improvement direction during training: for $\sigma \in \{0,0.1\}$ it swings down to around $-90$ and back, and for $\sigma =1$ it stays in a narrow band. HalfCheetah's swings barely change with $\sigma$, which fits its weak response.

More randomness allowing bigger steps felt backwards to me at first. It makes sense once you see that a random policy's objective averages reward over many nearby actions. Averaging blurs the landscape, so cliffs like "the robot falls or it doesn't" become slopes. The randomness that makes big steps risky is a different kind: noise in the gradient estimate, which the large batches hold down.

Better final solutions: fewer traps. Final Hopper solutions at $\sigma =0.1$ have about 3 times as many downward-bending directions as at $\sigma =1$. At one poor solution (Figure 7a), 84% of directions bend down under $\sigma =0$, so it looks like a hilltop, while under $\sigma =1$ almost every curvature is within noise, closer to a slope. Fewer downward bends don't prove a way up exists, so they check one: the straight line to a good solution, trained with $\sigma =1$ from the same initialization, climbs steadily under the $\sigma =1$ objective (7b). Under the true objective it's flat near the poor solution and then jumps. This holds for some pairs in Hopper and Walker2d (Figures S14 to S16), not all (S17), and not in HalfCheetah, where the local landscape barely changes with $\sigma$ (S18).

Exploration or smoothing?

In policy gradient methods these are two descriptions of the same randomness. Exploration describes behavior: the agent tries more things. Smoothing describes the math: averaging over those tries makes the objective easier to climb. The exact-gradient gridworld removes the data-gathering side entirely, since nothing is sampled, and entropy still helps, so smoothing is doing real work.

Later theory agrees. Mei et al. (2020) show that with exact gradients, softmax policy gradient converges at an $O(1/t)$ rate, while the entropy-regularized version converges linearly to its regularized optimum. Smoothing isn't the whole story either. In sampled settings, especially with sparse rewards, you can't learn from a reward you never reach, and HalfCheetah doesn't fit the smoothing story at all. I read the paper as adding a second lens, not replacing the first.

A nice side idea from the related work: Ilyas et al. (2018) found that ascent on PPO's surrogate objective doesn't always mean ascent on the true objective. This paper's lens suggests why that can still work: a surrogate, like an entropy-smoothed objective, can connect regions that the true objective separates with a dip.

Connection to Sutton & Barto, Chapter 13

I read this alongside the policy gradient chapter, and the two fit together well:

• Same theorem, different notation. ${\mathcal{O}}_{ER}$ is $J(\theta )$, $d^{\pi }$ is $\mu$, and $Q^{\pi }$ is $q_{\pi }$. The book's form also covers the continuing case, where $\mu$ becomes the steady-state distribution, $q_{\pi }$ the differential value, and $\propto$ becomes $=$.
• The book fixes noise; this paper points at geometry. The book's remedies for slow policy gradients are variance reduction: the baseline in REINFORCE with baseline, then actor-critic. This paper says that's half the problem, since even the exact gradient stalls on plateaus and can't cross valleys. The authors call for optimization techniques "orthogonal to variance reduction."
• The ${\gamma }^t$ is the discounted state weighting. The book's REINFORCE box multiplies each update by ${\gamma }^t$. The paper's main text calls $d^{\pi }$ the stationary distribution, but the exact gradient in Appendix S.2 (eq. 23) weights states by ${\sum }_t{\gamma }^t\mathrm{Pr}(s_t=s\mid s_0)$, the discounted version, which is exactly what the ${\gamma }^t$ does.
• Mix is an ε-soft policy. Chapter 5's ε-soft policies give every action at least $\epsilon /|\mathcal{A}|$ probability. Mix does the same with $\epsilon =\text{Mix}$.
• The price of approaching determinism. Section 13.1 sells softmax policies because, unlike ε-greedy, they can approach a deterministic policy. This paper shows the cost: a near-deterministic softmax sits on a plateau where pushing its preferences further barely changes the policy, so the gradient vanishes. For exactly these plateaus, the authors point to the natural gradient and methods like TRPO and PPO.

My take

The framing and the exact-gradient gridworld are the strongest parts: they cleanly separate noise from geometry, and the probe is cheap, needing only forward evaluations that parallelize trivially. Past that, the evidence is thinner than the abstract suggests:

• The probe is central differences. Equations (1) and (2) are central finite differences along random directions, and the novelty is in summarizing many directions as a scatter plot and two histograms. Expanding around ${\theta }_0$, as above, is also shorter than the appendix's derivation from a global quadratic, which drops and misplaces a few $\alpha$'s in its eqs. (4) and (5).
• Random directions understate the extremes. In $d^{\top }Hd={\sum }_i{\lambda }_i(v_i^{\top }d{)}^2$, each weight averages $1/n$ in $n$ dimensions, so random directions land near the average eigenvalue. The claim that the sampled extremes are close to ${\lambda }_{\min }$ and ${\lambda }_{\max }$ gets less true as $n$ grows, and a few sharp or uphill directions can go unseen, which the authors' own Figure S3 shows.
• Figure 4a has a confound. Because Mix = 0.8 leaves $\theta$ controlling 20% of the policy, orange's tighter cluster is partly mechanical, and blue versus orange isn't a fair comparison there. The clean signals are black versus everything in 4a and the depth of the valley in 4b.
• Entropy also moves the optimum. In Figure S5, a constant $\tau$ of 0.5 or more makes every run count as suboptimal, presumably because the regularized optimum stays too random to beat the deterministic 4.5. Decaying $\tau$ is what makes it work, so entropy acts as a continuation method: optimize a smoothed objective, then sharpen it. The discussion says as much.
• The learning curves are weak evidence. Five seeds, shading of half a standard deviation, learning rates tuned per $\sigma$, and no clean ordering among the higher $\sigma$ values. The authors concede the statistical power is limited.
• Scope. A tabular gridworld and linear Gaussian policies with at most a few hundred parameters, no neural networks, high-$\sigma$ policies that lean on action clipping (their Appendix S.4), and HalfCheetah showing no effect. Environment dependence is a finding, but it also bounds the claim.
• Small errors. "Stationary distribution" for what is really discounted state weighting (the appendix gets it right), "unit ball" for unit sphere, 84% in the text versus 85% in the caption for the same statistic, and appendix eqs. (21) and (22) label the gradient vector $q^{\pi }$ and write a matrix inverse as a division.

How I'd cite it: evidence that, in small tabular and linear-policy settings, a more random policy helps policy gradient partly by reshaping the objective (flat spots get a slope, curvature swings calm down, some local optima connect) rather than only through exploration, plus a cheap random-perturbation probe for gradient and curvature. It isn't evidence about deep policies, and it doesn't replace the exploration story.