In the current series on Advanced Trading Infrastructure we have described both the Position Class and the Portfolio Class - two essential components of a robust backtesting and live trading system. In this article we are going to extend our discussion to the Portfolio Handler Class, which will round out the description of the portfolio Order Management System (OMS).
The OMS is the backbone of any quantitative trading infrastructure. It needs to keep track of open (and closed) positions in assets, it needs to group those positions together into a portfolio (with cash) and must modify that portfolio with new trading signals, risk management overlays and position sizing rules.
In this article we will discuss the PortfolioHandler. This particular class is tasked with managing a Portfolio object, telling it whether to open/close positions based on information it receives from the Strategy, PositionSizer, RiskManager and ExecutionHandler. This class is extremely important as it ties the rest of the components together.
The following code presented in this article comes from the QSTrader open-source backtesting and live trading engine. I have released it under a liberal open-source MIT license and the latest version can always be found at https://github.com/mhallsmoore/qstrader. If you would like to keep up fully to date with the project, please take a look at that link.
In the previous article I listed out a current Component Reminder that detailed how all of the components of QSTrader fit together. Please take a look at it to remind yourself of how they all interact.
Let's now turn our attention to the PortfolioHandler class and see how it interacts with the Portfolio object.
PortfolioHandler
The first issue to discuss is why the older style of Portfolio class from QSForex has now been replaced with a calculation-heavy Portfolio class carrying out Position tracking, as well as the lighter PortfolioHandler class.
I made this choice because I felt it was cleaner to have a separate Portfolio object that was only tasked with keeping track of cash and open positions. The primary purpose of such an approach is to allow "theoretical" portfolio objects to be created (i.e. by the PositionSizer or RiskManager), and subsequently create a set of necessary trades to get from the current portfolio to the theoretically desired one.
This process is a lot more straightforward if the Portfolio is simply a grouping of Position objects and a cash balance.
This leaves the interaction with the events queue, the PositionSizer, RiskManager and PriceHandler. These interactions are handled by the new object, the PortfolioHandler.
I've created the PortfolioHandler in the file portfolio_handler.py and I've provided the full listing below. I'll break down the listing afterwards.
Note that any of these listings are subject to change, since I will be continually making changes to this project. Eventually I hope others will collaborate by providing Pull Requests to the codebase.
portfolio_handler.py
from qstrader.order.order import SuggestedOrder
from qstrader.portfolio.portfolio import Portfolio
class PortfolioHandler(object):
def __init__(
self, initial_cash, events_queue,
price_handler, position_sizer, risk_manager
):
"""
The PortfolioHandler is designed to interact with the
backtesting or live trading overall event-driven
architecture. It exposes two methods, on_signal and
on_fill, which handle how SignalEvent and FillEvent
objects are dealt with.
Each PortfolioHandler contains a Portfolio object,
which stores the actual Position objects.
The PortfolioHandler takes a handle to a PositionSizer
object which determines a mechanism, based on the current
Portfolio, as to how to size a new Order.
The PortfolioHandler also takes a handle to the
RiskManager, which is used to modify any generated
Orders to remain in line with risk parameters.
"""
self.initial_cash = initial_cash
self.events_queue = events_queue
self.price_handler = price_handler
self.position_sizer = position_sizer
self.risk_manager = risk_manager
self.portfolio = Portfolio(price_handler, initial_cash)
def _create_order_from_signal(self, signal_event):
"""
Take a SignalEvent object and use it to form a
SuggestedOrder object. These are not OrderEvent objects,
as they have yet to be sent to the RiskManager object.
At this stage they are simply "suggestions" that the
RiskManager will either verify, modify or eliminate.
"""
order = SuggestedOrder(
signal_event.ticker, signal_event.action
)
return order
def _place_orders_onto_queue(self, order_list):
"""
Once the RiskManager has verified, modified or eliminated
any order objects, they are placed onto the events queue,
to ultimately be executed by the ExecutionHandler.
"""
for order_event in order_list:
self.events_queue.put(order_event)
def _convert_fill_to_portfolio_update(self, fill_event):
"""
Upon receipt of a FillEvent, the PortfolioHandler converts
the event into a transaction that gets stored in the Portfolio
object. This ensures that the broker and the local portfolio
are "in sync".
In addition, for backtesting purposes, the portfolio value can
be reasonably estimated in a realistic manner, simply by
modifying how the ExecutionHandler object handles slippage,
transaction costs, liquidity and market impact.
"""
action = fill_event.action
ticker = fill_event.ticker
quantity = fill_event.quantity
price = fill_event.price
commission = fill_event.commission
# Create or modify the position from the fill info
self.portfolio.transact_position(
action, ticker, quantity,
price, commission
)
def on_signal(self, signal_event):
"""
This is called by the backtester or live trading architecture
to form the initial orders from the SignalEvent.
These orders are sized by the PositionSizer object and then
sent to the RiskManager to verify, modify or eliminate.
Once received from the RiskManager they are converted into
full OrderEvent objects and sent back to the events queue.
"""
# Create the initial order list from a signal event
initial_order = self._create_order_from_signal(signal_event)
# Size the quantity of the initial order
sized_order = self.position_sizer.size_order(
self.portfolio, initial_order
)
# Refine or eliminate the order via the risk manager overlay
order_events = self.risk_manager.refine_orders(
self.portfolio, sized_order
)
# Place orders onto events queue
self._place_orders_onto_queue(order_events)
def on_fill(self, fill_event):
"""
This is called by the backtester or live trading architecture
to take a FillEvent and update the Portfolio object with new
or modified Positions.
In a backtesting environment these FillEvents will be simulated
by a model representing the execution, whereas in live trading
they will come directly from a brokerage (such as Interactive
Brokers).
"""
self._convert_fill_to_portfolio_update(fill_event)
The PortfolioHandler requires the SuggestedOrder object as well as the Portfolio. The former is a separate object to an OrderEvent as it has not been through a position sizing or risk management process. Once an order has been through both of those processes it becomes a full OrderEvent.
To initialise a PortfolioHandler we require an initial cash balance and references to the events queue, the price handler, the position sizer and the risk manager. Finally we create the inner associated Portfolio object. Notice that it itself requires access to the price handler and the initial cash balance:
from qstrader.order.order import SuggestedOrder
from qstrader.portfolio.portfolio import Portfolio
class PortfolioHandler(object):
def __init__(
self, initial_cash, events_queue,
price_handler, position_sizer, risk_manager
):
"""
The PortfolioHandler is designed to interact with the
backtesting or live trading overall event-driven
architecture. It exposes two methods, on_signal and
on_fill, which handle how SignalEvent and FillEvent
objects are dealt with.
Each PortfolioHandler contains a Portfolio object,
which stores the actual Position objects.
The PortfolioHandler takes a handle to a PositionSizer
object which determines a mechanism, based on the current
Portfolio, as to how to size a new Order.
The PortfolioHandler also takes a handle to the
RiskManager, which is used to modify any generated
Orders to remain in line with risk parameters.
"""
self.initial_cash = initial_cash
self.events_queue = events_queue
self.price_handler = price_handler
self.position_sizer = position_sizer
self.risk_manager = risk_manager
self.portfolio = Portfolio(price_handler, initial_cash)
In the following method, _create_order_from_signal, we simply create the SuggestedOrder from the ticker and action. At this stage we are only supporting market orders. Limit orders, and more exotic forms of execution, will come later:
# portfolio_handler.py
def _create_order_from_signal(self, signal_event):
"""
Take a SignalEvent object and use it to form a
SuggestedOrder object. These are not OrderEvent objects,
as they have yet to be sent to the RiskManager object.
At this stage they are simply "suggestions" that the
RiskManager will either verify, modify or eliminate.
"""
order = SuggestedOrder(
signal_event.ticker, signal_event.action
)
return order
_place_orders_onto_queue is a simple helper method that takes in a list of OrderEvent objects and adds them to the events queue:
# portfolio_handler.py
def _place_orders_onto_queue(self, order_list):
"""
Once the RiskManager has verified, modified or eliminated
any order objects, they are placed onto the events queue,
to ultimately be executed by the ExecutionHandler.
"""
for order_event in order_list:
self.events_queue.put(order_event)
The following method, _convert_fill_to_portfolio_update, takes in a FillEvent message and then adjusts the inner Portfolio object to account for the fill transaction. As can be seen, it shows that the PortfolioHandler does no mathematical calculation of its own, rather it delegates the calculations to the Portfolio class:
# portfolio_handler.py
def _convert_fill_to_portfolio_update(self, fill_event):
"""
Upon receipt of a FillEvent, the PortfolioHandler converts
the event into a transaction that gets stored in the Portfolio
object. This ensures that the broker and the local portfolio
are "in sync".
In addition, for backtesting purposes, the portfolio value can
be reasonably estimated in a realistic manner, simply by
modifying how the ExecutionHandler object handles slippage,
transaction costs, liquidity and market impact.
"""
action = fill_event.action
ticker = fill_event.ticker
quantity = fill_event.quantity
price = fill_event.price
commission = fill_event.commission
# Create or modify the position from the fill info
self.portfolio.transact_position(
action, ticker, quantity,
price, commission
)
The on_signal method ties together some of the previous methods. It creates the initial suggested order, then sends it to the PositionSizer object (along with the portfolio) to be refined. Once the sized order is returned, the RiskManager is then sent the order to manage any risk associated with how the new order will affect the current portfolio.
The risk manager then sends back a list of orders. Why a list? Well, consider the fact that a generated trade may induce the risk manager to create a hedging order in another security. Hence there is a need to possibly return more than one order.
Once the list of orders has been created they are all placed onto the events queue:
# portfolio_handler.py
def on_signal(self, signal_event):
"""
This is called by the backtester or live trading architecture
to form the initial orders from the SignalEvent.
These orders are sized by the PositionSizer object and then
sent to the RiskManager to verify, modify or eliminate.
Once received from the RiskManager they are converted into
full OrderEvent objects and sent back to the events queue.
"""
# Create the initial order list from a signal event
initial_order = self._create_order_from_signal(signal_event)
# Size the quantity of the initial order
sized_order = self.position_sizer.size_order(
self.portfolio, initial_order
)
# Refine or eliminate the order via the risk manager overlay
order_events = self.risk_manager.refine_orders(
self.portfolio, sized_order
)
# Place orders onto events queue
self._place_orders_onto_queue(order_events)
The final method in the PortfolioHandler is on_fill. It simply calls the previous method _convert_fill_to_portfolio_update. These two methods have been separated, as in later versions of QSTrader there might be a need for more sophisticated logic to exist. We don't wish to change the on_fill interface to the PortfolioHandler unless absolutely necessary. This helps maintain backward compatibility:
# portfolio_handler.py
def on_fill(self, fill_event):
"""
This is called by the backtester or live trading architecture
to take a FillEvent and update the Portfolio object with new
or modified Positions.
In a backtesting environment these FillEvents will be simulated
by a model representing the execution, whereas in live trading
they will come directly from a brokerage (such as Interactive
Brokers).
"""
self._convert_fill_to_portfolio_update(fill_event)
This completes the PortfolioHandler class description. For completeness you can find the full code for the PortfolioHandler class on Github at portfolio_handler.py.
portfolio_handler_test.py
Now that we've created the PortfolioHandler we need to test it. Thankfully, the majority of the mathematical tests occur in the Position and Portfolio classes. However, it is still necessary to test that the PortfolioHandler "does the right thing" when it receives strategy-generated signals and execution-generated fills.
While the following tests may seem "trivial", I can assure you that even though it can be quite tedious to write out unit testing code, it is absolutely vital to ensure that you have a functioning system as more complexity is added. One of the most frustrating aspects of software development is not unit testing to "get an answer quickly" and then realising you have a bug and no idea where it is in a large collection of modules!
By carrying out unit testing while we write the individual modules we avoid this problem as much as possible. If a bug is discovered, it is usually much more straightforward to track it down. Time spent unit testing is never wasted!
The following is a full listing of portfolio_handler_test.py. After the listing I will break down the individual objects and methods, as before:
import datetime
from decimal import Decimal
import queue
import unittest
from qstrader.event.event import FillEvent, OrderEvent, SignalEvent
from qstrader.portfolio_handler.portfolio_handler import PortfolioHandler
class PriceHandlerMock(object):
def __init__(self):
pass
def get_best_bid_ask(self, ticker):
prices = {
"MSFT": (Decimal("50.28"), Decimal("50.31")),
"GOOG": (Decimal("705.46"), Decimal("705.46")),
"AMZN": (Decimal("564.14"), Decimal("565.14")),
}
return prices[ticker]
class PositionSizerMock(object):
def __init__(self):
pass
def size_order(self, portfolio, initial_order):
"""
This PositionSizerMock object simply modifies
the quantity to be 100 of any share transacted.
"""
initial_order.quantity = 100
return initial_order
class RiskManagerMock(object):
def __init__(self):
pass
def refine_orders(self, portfolio, sized_order):
"""
This RiskManagerMock object simply lets the
sized order through, creates the corresponding
OrderEvent object and adds it to a list.
"""
order_event = OrderEvent(
sized_order.ticker,
sized_order.action,
sized_order.quantity
)
return [order_event]
class TestSimpleSignalOrderFillCycleForPortfolioHandler(unittest.TestCase):
"""
Tests a simple Signal, Order and Fill cycle for the
PortfolioHandler. This is, in effect, a sanity check.
"""
def setUp(self):
"""
Set up the PortfolioHandler object supplying it with
$500,000.00 USD in initial cash.
"""
initial_cash = Decimal("500000.00")
events_queue = queue.Queue()
price_handler = PriceHandlerMock()
position_sizer = PositionSizerMock()
risk_manager = RiskManagerMock()
# Create the PortfolioHandler object from the rest
self.portfolio_handler = PortfolioHandler(
initial_cash, events_queue, price_handler,
position_sizer, risk_manager
)
def test_create_order_from_signal_basic_check(self):
"""
Tests the "_create_order_from_signal" method
as a basic sanity check.
"""
signal_event = SignalEvent("MSFT", "BOT")
order = self.portfolio_handler._create_order_from_signal(signal_event)
self.assertEqual(order.ticker, "MSFT")
self.assertEqual(order.action, "BOT")
self.assertEqual(order.quantity, 0)
def test_place_orders_onto_queue_basic_check(self):
"""
Tests the "_place_orders_onto_queue" method
as a basic sanity check.
"""
order = OrderEvent("MSFT", "BOT", 100)
order_list = [order]
self.portfolio_handler._place_orders_onto_queue(order_list)
ret_order = self.portfolio_handler.events_queue.get()
self.assertEqual(ret_order.ticker, "MSFT")
self.assertEqual(ret_order.action, "BOT")
self.assertEqual(ret_order.quantity, 100)
def test_convert_fill_to_portfolio_update_basic_check(self):
"""
Tests the "_convert_fill_to_portfolio_update" method
as a basic sanity check.
"""
fill_event_buy = FillEvent(
datetime.datetime.utcnow(), "MSFT", "BOT",
100, "ARCA", Decimal("50.25"), Decimal("1.00")
)
self.portfolio_handler._convert_fill_to_portfolio_update(fill_event_buy)
# Check the Portfolio values within the PortfolioHandler
port = self.portfolio_handler.portfolio
self.assertEqual(port.cur_cash, Decimal("494974.00"))
def test_on_signal_basic_check(self):
"""
Tests the "on_signal" method as a basic sanity check.
"""
signal_event = SignalEvent("MSFT", "BOT")
self.portfolio_handler.on_signal(signal_event)
ret_order = self.portfolio_handler.events_queue.get()
self.assertEqual(ret_order.ticker, "MSFT")
self.assertEqual(ret_order.action, "BOT")
self.assertEqual(ret_order.quantity, 100)
if __name__ == "__main__":
unittest.main()
The first task is to import the correct modules. We make use of decimal, as in prior articles, as well as the unittest module. We also need to import various Event objects that the PortfolioHandler uses to communicate. Finally we import the PortfolioHandler itself:
import datetime
from decimal import Decimal
import queue
import unittest
from qstrader.event.event import FillEvent, OrderEvent, SignalEvent
from qstrader.portfolio_handler.portfolio_handler import PortfolioHandler
We need to create three "mock" objects (see the previous article for a description of mock objects), one each for the PriceHandler, PositionSizer and RiskManager. The first, PriceHandlerMock, provides us with static bid/ask prices for three shares: MSFT, GOOG and AMZN. Essentially we want to simulate the get_best_bid_ask method for our repeatable unit tests:
class PriceHandlerMock(object):
def __init__(self):
pass
def get_best_bid_ask(self, ticker):
prices = {
"MSFT": (Decimal("50.28"), Decimal("50.31")),
"GOOG": (Decimal("705.46"), Decimal("705.46")),
"AMZN": (Decimal("564.14"), Decimal("565.14")),
}
return prices[ticker]
The second mock object is PositionSizerMock. It simply sets the order quantity to be 100, which is an arbitrary choice, but is necessary to be fixed for our unit tests. It simulates the size_order method that will be found on the "real" PositionSizer class when it is complete:
class PositionSizerMock(object):
def __init__(self):
pass
def size_order(self, portfolio, initial_order):
"""
This PositionSizerMock object simply modifies
the quantity to be 100 of any share transacted.
"""
initial_order.quantity = 100
return initial_order
The final mock object is RiskManagerMock. It doesn't do anything beyond creating an OrderEvent object and placing this in a list. Crucially, there is no real risk management! While this may seem contrived, it does allow us to perform a basic "sanity check" that the PortfolioHandler can simply transact the most basic of orders, fills and signals. As we create more sophisticated RiskManager objects, our unit test list will grow, in order to test the new functionality. This way we continually ensure that the codebase is working as expected:
class RiskManagerMock(object):
def __init__(self):
pass
def refine_orders(self, portfolio, sized_order):
"""
This RiskManagerMock object simply lets the
sized order through, creates the corresponding
OrderEvent object and adds it to a list.
"""
order_event = OrderEvent(
sized_order.ticker,
sized_order.action,
sized_order.quantity
)
return [order_event]
Now that we have the three mock objects defined, we can create the unit tests themselves. The class that carries this out is called TestSimpleSignalOrderFillCycleForPortfolioHandler. While verbose, it does tell us exactly what the test class is designed to do, namely test a simple signal-order-fill cycle within the portfolio handler.
In order to do this, we create an initial cash balance of 500,000 USD, an events queue and the three aforementioned mock objects. Finally, we create the PortfolioHandler itself and attach it to the test class:
class TestSimpleSignalOrderFillCycleForPortfolioHandler(unittest.TestCase):
"""
Tests a simple Signal, Order and Fill cycle for the
PortfolioHandler. This is, in effect, a sanity check.
"""
def setUp(self):
"""
Set up the PortfolioHandler object supplying it with
$500,000.00 USD in initial cash.
"""
initial_cash = Decimal("500000.00")
events_queue = queue.Queue()
price_handler = PriceHandlerMock()
position_sizer = PositionSizerMock()
risk_manager = RiskManagerMock()
# Create the PortfolioHandler object from the rest
self.portfolio_handler = PortfolioHandler(
initial_cash, events_queue, price_handler,
position_sizer, risk_manager
)
The first test simply generates a fake SignalEvent to buy Microsoft. We then test that the correct order has been generated. Note that a quantity has not been set at this stage (it is zero). We check for all properties to ensure that the order has been created correctly:
# test_portfolio_handler.py
def test_create_order_from_signal_basic_check(self):
"""
Tests the "_create_order_from_signal" method
as a basic sanity check.
"""
signal_event = SignalEvent("MSFT", "BOT")
order = self.portfolio_handler._create_order_from_signal(signal_event)
self.assertEqual(order.ticker, "MSFT")
self.assertEqual(order.action, "BOT")
self.assertEqual(order.quantity, 0)
The next test is checking to see whether the orders are correctly placed on the queue (and retrieved). Note that we have to wrap the OrderEvent into a list, as the RiskManager produces a list of orders, due to the aforementioned need to possibly hedge or add additional orders beyond those suggested by the Strategy. Finally, we assert that the returned order (which is taken from the queue) contains the appropriate information:
# test_portfolio_handler.py
def test_place_orders_onto_queue_basic_check(self):
"""
Tests the "_place_orders_onto_queue" method
as a basic sanity check.
"""
order = OrderEvent("MSFT", "BOT", 100)
order_list = [order]
self.portfolio_handler._place_orders_onto_queue(order_list)
ret_order = self.portfolio_handler.events_queue.get()
self.assertEqual(ret_order.ticker, "MSFT")
self.assertEqual(ret_order.action, "BOT")
self.assertEqual(ret_order.quantity, 100)
The following test creates a FillEvent, as if one had just been received from an ExecutionHandler object. The portfolio handler is then told to convert the fill to an actual portfolio update (i.e. register the transaction with the inner Portfolio object).
The test is to actually check that the current cash balance within the inner Portfolio is actually correct:
# test_portfolio_handler.py
def test_convert_fill_to_portfolio_update_basic_check(self):
"""
Tests the "_convert_fill_to_portfolio_update" method
as a basic sanity check.
"""
fill_event_buy = FillEvent(
datetime.datetime.utcnow(), "MSFT", "BOT",
100, "ARCA", Decimal("50.25"), Decimal("1.00")
)
self.portfolio_handler._convert_fill_to_portfolio_update(fill_event_buy)
# Check the Portfolio values within the PortfolioHandler
port = self.portfolio_handler.portfolio
self.assertEqual(port.cur_cash, Decimal("494974.00"))
The final test simply tests the on_signal method by creating a SignalEvent object, placing it on the queue and then retrieving it to check that the order values are as expected. This tests the "end to end" basic handling of the PositionSizer and RiskManager objects:
# test_portfolio_handler.py
def test_on_signal_basic_check(self):
"""
Tests the "on_signal" method as a basic sanity check.
"""
signal_event = SignalEvent("MSFT", "BOT")
self.portfolio_handler.on_signal(signal_event)
ret_order = self.portfolio_handler.events_queue.get()
self.assertEqual(ret_order.ticker, "MSFT")
self.assertEqual(ret_order.action, "BOT")
self.assertEqual(ret_order.quantity, 100)
We can clearly see that there is a lot more testing to be done here. We've only scratched the surface with the sort of situations that can occur. However, it is always good to have a basic set of sanity checks in place. The unit testing framework is highly extensible and as we come across new situations/bugs, we can simply write new tests and correct for the problem.
For completeness you can find the full code for the testing of PortfolioHandler on Github at portfolio_handler_test.py.
Next Steps
We've now covered three of the main objects for the Order Management System, namely the Position, the Portfolio and the PortfolioHandler. These are the "core" mathematical calculation aspects of the code and as such we need to be sure that they work as expected.
While discussing these objects is not as exciting as building a Strategy object, or even a RiskManager, it is vital that they work, otherwise the rest of the backtesting and live trading infrastructure will be, at best, useless and at worst, highly unprofitable!
We've got a lot of remaining componets to cover in addition to those mentioned above, including the PriceHandler, the Backtest class, the various ExecutionHandlers that might tie into Interactive Brokers or OANDA, as well as the implementation of a non-trivial Strategy object.
In the next article I will discuss one or more of these classes.
