AAAI Publications, Twenty-Ninth AAAI Conference on Artificial Intelligence

Font Size: 
Optimizing the CVaR via Sampling
Aviv Tamar, Yonatan Glassner, Shie Mannor

Last modified: 2015-02-21

Abstract


Conditional Value at Risk (CVaR) is a prominent risk measure that is being used extensively in various domains. We develop a new formula for the gradient of the CVaR in the form of a conditional expectation. Based on this formula, we propose a novel sampling-based estimator for the gradient of the CVaR, in the spirit of the likelihood-ratio method. We analyze the bias of the estimator, and prove the convergence of a corresponding stochastic gradient descent algorithm to a local CVaR optimum. Our method allows to consider CVaR optimization in new domains. As an example, we consider a reinforcement learning application, and learn a risk-sensitive controller for the game of Tetris.

Keywords


CVaR; Likelihood Ratio Method; Reinforcement Learning; MDP

Full Text: PDF