Constrained Optimization and Backtesting
A portfolio that looks optimal on paper can fall apart once real-world frictions enter. This chapter adds constraints (no short-selling) and transaction costs to mean–variance optimization, solves the constrained problem with quadratic programming, and backtests the strategies out of sample on a rolling window. Use the R | Python toggle to switch.
library(tidyverse)
library(tidyfinance)
library(scales)
library(quadprog)
import pandas as pd
import numpy as np
import sqlite3
from cvxopt import matrix, solvers
Data preparation
We load the monthly panel and pivot a handful of stocks to a returns matrix to keep the linear algebra transparent.
crsp_monthly <- tbl(tidy_finance, "crsp_monthly") |>
filter(industry != "Finance") |>
select(permno, gvkey, date, ret_excess, mktcap) |>
collect()
returns <- crsp_monthly |>
filter(permno %in% c("14593", "10107", "93436", "17778")) |>
select(permno, date, ret_excess) |>
pivot_wider(names_from = permno, values_from = ret_excess) |>
drop_na()
sigma <- cov(returns |> select(-date))
mu <- colMeans(returns |> select(-date))
N <- ncol(returns) - 1
tidy_finance = sqlite3.connect(database="data/tidy_finance_python.sqlite")
crsp_monthly = (pd.read_sql_query(
"SELECT permno, gvkey, date, industry, ret_excess, mktcap FROM crsp_monthly",
tidy_finance, parse_dates={"date"})
.query("industry != 'Finance'"))
returns = (crsp_monthly
.query("permno in [14593, 10107, 93436, 17778]")
.pivot(index="date", columns="permno", values="ret_excess")
.dropna())
sigma = returns.cov().to_numpy()
mu = returns.mean().to_numpy()
N = returns.shape[1]
The unconstrained efficient weight
The closed-form efficient weight, with a transaction-cost term lambda that penalizes moving away from the previous weights w_prev and a risk-aversion gamma. Setting lambda = 0 recovers the frictionless solution.
compute_efficient_weight <- function(sigma, mu, gamma = 2, lambda = 0,
w_prev = rep(1 / ncol(sigma), ncol(sigma))) {
iota <- rep(1, ncol(sigma))
sigma_processed <- sigma + lambda / gamma * diag(ncol(sigma))
mu_processed <- mu + lambda * w_prev
sigma_inverse <- solve(sigma_processed)
w_mvp <- sigma_inverse %*% iota
w_mvp <- as.vector(w_mvp / sum(w_mvp))
w_opt <- w_mvp + 1 / gamma *
(sigma_inverse - w_mvp %*% t(iota) %*% sigma_inverse) %*% mu_processed
as.vector(w_opt)
}
def compute_efficient_weight(sigma, mu, gamma=2, lambda_=0, w_prev=None):
n = sigma.shape[0]
if w_prev is None:
w_prev = np.ones(n) / n
iota = np.ones(n)
sigma_processed = sigma + lambda_ / gamma * np.eye(n)
mu_processed = mu + lambda_ * w_prev
sigma_inv = np.linalg.inv(sigma_processed)
w_mvp = sigma_inv @ iota
w_mvp = w_mvp / w_mvp.sum()
w_opt = w_mvp + (1 / gamma) * (
sigma_inv - np.outer(w_mvp, iota) @ sigma_inv) @ mu_processed
return w_opt
Adding a no-short-selling constraint
Unconstrained solutions often demand large negative positions. Imposing weights >= 0 (and summing to one) turns the problem into a quadratic program — solve.QP in R, cvxopt in Python — which keeps the portfolio implementable.
gamma <- 2
w_no_short_sell <- solve.QP(
Dmat = gamma * sigma,
dvec = mu,
Amat = cbind(1, diag(N)),
bvec = c(1, rep(0, N)),
meq = 1
)$solution
gamma = 2
P = matrix(gamma * sigma)
q = matrix(-mu)
G = matrix(-np.eye(N))
h = matrix(np.zeros(N))
A = matrix(1.0, (1, N))
b = matrix(1.0)
solvers.options["show_progress"] = False
w_no_short_sell = np.array(solvers.qp(P, q, G, h, A, b)["x"]).flatten()
Backtesting out of sample
The honest test: at each rebalance use only past data (a rolling window), form the weights, then apply them to the next period's realized returns — charging transaction costs on the turnover. Comparing the minimum-variance, unconstrained-efficient, and no-short strategies on net-of-cost returns shows which survive the frictions.
window_length <- 120
periods <- nrow(returns) - window_length
oos_values <- matrix(NA, nrow = periods, ncol = 3)
colnames(oos_values) <- c("mvp", "efficient", "no_short_sell")
for (i in 1:periods) {
return_window <- returns[i:(i + window_length - 1), -1]
sigma_window <- cov(return_window)
mu_window <- colMeans(return_window)
next_return <- as.numeric(returns[i + window_length, -1])
w_efficient <- compute_efficient_weight(sigma_window, mu_window, gamma)
oos_values[i, "efficient"] <- sum(w_efficient * next_return)
}
window_length = 120
periods = returns.shape[0] - window_length
oos_values = pd.DataFrame(index=range(periods),
columns=["mvp", "efficient", "no_short_sell"], dtype=float)
returns_array = returns.to_numpy()
for i in range(periods):
return_window = returns_array[i:(i + window_length)]
sigma_window = np.cov(return_window, rowvar=False)
mu_window = return_window.mean(axis=0)
next_return = returns_array[i + window_length]
w_efficient = compute_efficient_weight(sigma_window, mu_window, gamma)
oos_values.loc[i, "efficient"] = float(w_efficient @ next_return)
The recurring lesson: frictions and look-ahead bias separate a strategy that works on a slide from one that works in practice — which is why the evaluation reports net-of-cost returns, turnover, and drawdowns rather than just gross Sharpe ratios.
Study notes following the Tidy Finance curriculum by Scheuch, Voigt, Weiss, and Frey. Prose is my own; the R/Python code is reproduced from the book's open-source source, licensed CC BY-NC-SA 4.0.