Developing a Real Life Fully Automated Trading Algorithm Using Interactive Brokers and Python

In this article I describe how my backtested algorithm can be used in live algorithmic trading. My broker provides me with the TWS (Trader WorkStation) API which is the solution that I use to build my trading application.

The basic idea of my trading strategy is to rank hundreds of stocks by value (\(value = \frac{1}{(PB \times PE)}\)) in descending order. That way a company with a high PB and PE ratio will land at the bottom. I then short the bottom ranking 30 stocks and go long the top ranking 30 stocks. The algorithm will check if rebalancing is needed and will do so using an interactive mode as a precautionary measure every time it runs. How often it runs is up to you. You can reschedule it to run daily, monthly or quarterly using a simple cron job.

For the value factor I need price and fundamental data. The price data I request from my broker. I have a market data subscription for that. For fundamental data such as earnings and equity I use my own data source. For the later stay tuned as I will be making this fundamental data available sometime in the future.

I used Python to develop this strategy. You might need to signup for a paper account at your broker and make sure they are using TWS. For a full TWS API please refer to the official documentation.

The TWS API Architecture

I will not go into all the API details since the official documentation describes everything pretty well. But I believe it is good to at least describe the idea behind the TWS API architecture. Some of the text below is taken straight out of the official documentation.

First you need to start TWS on your machine to allow for incoming connections. TWS acts as a server to receive requests from the API application (our Python client/strategy) and responds by taking appropriate actions. The first step is for the API client to initiate a connection to TWS on a socket port where TWS is already listening.

Once the TWS is up and running and actively listening for incoming connections we are ready to write our code. We have to implement TWS API’s two major interfaces: EWrapper interface and the EClientSocket.

The EWrapper interface is the mechanism through which the TWS delivers information to the API client application (our strategy). By implementing this interface the client application will be able to receive and handle the information coming from the TWS.

The class used to send messages to TWS is EClientSocket. Unlike EWrapper, this class is not overriden as the provided functions in EClientSocket are invoked to send messages to TWS. To use EClientSocket, first it may be necessary to implement the EWrapper interface as part of its constructor parameters so that the application can handle all returned messages. Messages sent from TWS as a response to function calls in EClientSocket require a EWrapper implementation so they can processed to meet the needs of the API client.

TWS API programs always have at least two threads of execution. One thread is used for sending messages to TWS, and another thread is used for reading returned messages. The second thread uses the API EReader class to read from the socket and add messages to a queue. Every time a new message is added to the message queue, a notification flag is triggered to let other threads know that there is a message waiting to be processed. In the two-thread design of an API program, the message queue is also processed by the first thread. The thread responsible for the message queue will decode messages and invoke the appropriate functions in EWrapper.

In Python TWS API the EReader thread is automatically started upon connection to TWS. There is no need for user to start the reader.

Let us go ahead and implement the EWrapper interface first since that is where the data from TWS will be coming in. The code snippet below does not contain all the methods implemented for EWrapper as my intention is to illustrate the basic idea. The full implementation will be listed at the end of this article.

class Wrapper(EWrapper):
    def __init__(self):
        wrapper.EWrapper.__init__(self)

    def accountSummary(self, reqId: int, account: str, tag: str, value: str, currency: str):
        super().accountSummary(reqId, account, tag, value, currency)
        global account_summary
        account_summary[tag] = value

    def accountSummaryEnd(self, reqId: int):
        super().accountSummaryEnd(reqId)
        global account_summary_end
        account_summary_end = True

    def tickPrice(self, reqId: TickerId, tickType: TickType, price: float, attrib: TickAttrib):
        super().tickPrice(reqId, tickType, price, attrib)
        if tickType == TickTypeEnum.CLOSE:
            global close_price
            close_price = price

    def nextValidId(self, orderId: int):
        global next_valid_order_id
        next_valid_order_id = orderId

It is important to understand that the methods implemented in EWrapper will be called when the respective data (e.g.: account summary) that has been requested by the client will become available. Depending on the data requested the method might get called repeatedly as can be the case with tick data or it can be called as low as once. For the later the end is usually signaled using the counterpart *End method.

Let’s look at our EClient subclass that will be requesting our data from TWS. Again I am not showing the entire client implementation at this stage.

class Client(EClient):
    def __init__(self, wrapper):
        EClient.__init__(self, wrapper)

    def get_close_price(self, symbol):
        contract = Contract()
        contract.symbol = symbol
        contract.secType = "STK"
        contract.currency = "USD"
        contract.exchange = "SMART"
        contract.primaryExchange = "NASDAQ"

        global close_price
        close_price = None

        reqId = self.get_next_request_id()

        self.reqMarketDataType(4)  # 1 for live, 4 for delayed-frozen data if live is not available
        self.reqMktData(reqId, contract, "", False, False, [])

        while close_price is None:
            sleep(0.1)

        self.cancelMktData(reqId)
        return close_price

    def get_portfolio_value(self):
        request_id = int(time())
        global account_summary_end
        account_summary_end = False

        self.reqAccountSummary(request_id, "All", "$LEDGER:ALL")

        while not account_summary_end:
            sleep(0.1)

        global account_summary
        return float(account_summary["TotalCashBalance"])

    def get_valid_order_id(self):
        global next_valid_order_id
        next_valid_order_id = None

        self.reqIds(numIds=1)

        while next_valid_order_id is None:
            sleep(0.1)

        return next_valid_order_id

Notice how the client receives an EWrapper object in its constructor. In the above snippet see how we are calling the TWS API methods using the keyword self(e.g.: self.reqAccountSummary(...)) from within our methods (e.g.: get_portfolio_value(...)). The API methods are available to us since our client is a subclass of EClient. In order to make it easier to read the code and distinguish between API provided methods and method written by me I will use snake_case as my naming convention for my methods.

Finally we create a new class called Strategy that is both an EClient and EWrapper that will be connecting to TWS.

Once the client is connected, an EReader thread will be automatically created to handle incoming messages and put the messages into a message queue for further process. If you are using the TWS API with other programming languages you might need to pass an EReaderSignal object to the EClientSocket’s constructor. This object is used to signal that a message is ready for processing in the queue. In Python the Queue class handles this task directly so we don’t have to pass in any EReaderSignal objects to our client’s constructor.

Looking inside the EClient’s connect(..) method we see that the EReader thread is created for us. This thread is used to read from the socket and add messages to the queue.

def connect(self, host, port, clientId):
    ...
    self.reader = reader.EReader(self.conn, self.msg_queue)
    self.reader.start()   # start thread
    ...

We also need to trigger Client::run(), where the message queue is processed in an infinite loop and the EWrapper call-back functions are then called. Let us briefly look inside the run() implementation of EClient. You don’t have to worry about this since this is already made available to you.

def run(self):
    """This is the function that has the message loop."""
    try:
        while not self.done and (self.isConnected() or not self.msg_queue.empty()):
            try:
                try:
                    text = self.msg_queue.get(block=True, timeout=0.2)
                    if len(text) > MAX_MSG_LEN:
                        self.wrapper.error(NO_VALID_ID, BAD_LENGTH.code(), 
                            "%s:%d:%s" % (BAD_LENGTH.msg(), len(text), text))
                        self.disconnect()
                        break
                except queue.Empty:
                    logger.debug("queue.get: empty")
                else:
                    fields = comm.read_fields(text)
                    logger.debug("fields %s", fields)
                    self.decoder.interpret(fields)

            except (KeyboardInterrupt, SystemExit):
                logger.info("detected KeyboardInterrupt, SystemExit")
                self.keyboardInterrupt()
                self.keyboardInterruptHard()
            except BadMessage:
                logger.info("BadMessage")
                self.conn.disconnect()

            logger.debug("conn:%d queue.sz:%d", 
                        self.isConnected(), 
                        self.msg_queue.qsize())
    finally:
        self.disconnect()

In order for the client to be able to trigger EWrapper call-back functions we need to provide it with a concrete EWrapper implementation – a Strategy instance in our case. We do that using constructor dependency injection.

class Strategy(Wrapper, Client):
    def __init__(self, ipaddress, portid, clientid):
        Wrapper.__init__(self)
        Client.__init__(self, wrapper=self)
        self.connect(ipaddress, portid, clientid)
        thread = Thread(target=self.run)
        thread.start()
        setattr(self, "_thread", thread)

The wrapper methods are then triggered in the infinite loop listed in the run() method above using the Decoder object as soon as a message is available in the queue:

self.decoder.interpret(fields)

The decoder interprets the fields and knows what method to call on the EWrapper object.

That is basically the high level overview of the TWS API architecture. Next we will be focusing on the concrete trading strategy.

The Strategy

The first thing to do is instantiate a new Strategy and connect to TWS:

app = Strategy(ipaddress="localhost", portid=1234, clientid=0)

Next we will be fetching some stocks from our database that we will be processing:

fundamentals = Fundamentals('database.ini')
y = year()
q = last_quarter()
stocks = fundamentals.get_stock_profiles(y, q)

Fundamentals is my proprietary API that communicates with my fundamentals database. I am currently working on making this publicly available in the future. But for now you can assume that this returns a list of StockProfile objects from a MySQL database. Each StockProfile represents a company and has properties such as price_earning, price_book and industry. year() and last_quarter() just return the current year and quarter and are passed in get_stock_profiles(…) to fetch fundamental data for that reporting period.

We will now request the latest close prices for each stock in our list from TWS:

for stock in stocks:
    stock.price = app.get_close_price(stock.symbol)

Filter out companies for which we don’t have enough data and companies whose industry is either Financials or Utilities:

def filter_stock_profiles(stock_profiles):
    filtered_stock_profiles = list(filter(lambda stock:
                                          (stock.price_earnings is not None)
                                          and (not math.isnan(stock.price_earnings))
                                          and (stock.price_book is not None)
                                          and (not math.isnan(stock.price_book)), stock_profiles))

    filtered_stock_profiles = list(filter(lambda stock: (stock.industry is not None) and (
        stock.industry != 'Financials') and (stock.industry != 'Utilities'), filtered_stock_profiles))
        
        
stocks = filter_stock_profiles(stocks)

Once we have filtered out all the unwanted companies we proceed with ranking them by value in descending order and fetching the top and bottom stocks for which we want to place an order:

def rank_stocks(stock_profiles):
    return sorted(stock_profiles, key=lambda stock: 1.0 / (float(stock.price_earnings) + float(stock.price_book)), reverse=True)

stocks = rank_stocks(stocks)
longs = stocks[:MAX_LONG_POSITION_SIZE]
shorts = stocks[-MAX_SHORT_POSITION_SIZE:]

Next we calculate the weights that will be used to rebalance our portfolio later. We want the top ranking stocks to have a heavier weight in our long position and for the shorts we want the lowest ranking companies to carry a greater weight. That is, we buy more stocks of our top companies and sell more of our worst companies. Notice also that I don’t check a stock’s liquidity or if I can actually trade a particular stock. It is probably a good idea to do that in order to diminish market impact. It might also be a good idea to invest less in stocks with very low liquidity. I don’t do that here for the sake of simplicity.

portfolio_value = app.get_portfolio_value()
amount = portfolio_value // TARGET_NUMBER_OF_POSITIONS

my_positions = app.get_positions()

amount_available_for_long = (portfolio_value // 2)
number_of_long_units = (MAX_LONG_POSITION_SIZE * (MAX_LONG_POSITION_SIZE + 1)) / 2
amount_per_long_unit = amount_available_for_long // number_of_long_units

for rank, stock in enumerate(longs):
        dollar_amount = (MAX_LONG_POSITION_SIZE - rank) * amount_per_long_unit
        amount = dollar_amount // stock.price
        stock.portfolio_weight = (amount * stock.price) / portfolio_value
        stock.order_amount = amount

amount_available_for_short = (portfolio_value // 2)
number_of_short_units = (MAX_SHORT_POSITION_SIZE * (MAX_SHORT_POSITION_SIZE + 1)) / 2
amount_per_short_unit = amount_available_for_short // number_of_short_units

for rank, stock in enumerate(shorts):
        dollar_amount = (rank + 1) * amount_per_short_unit
        amount = dollar_amount // stock.price
        stock.portfolio_weight = (amount * stock.price) / portfolio_value
        stock.order_amount = -1 * amount

Before we go about rebalancing our portfolio we will be exiting any positions that we have in companies that are not included in our ranking.

order_book = []

# Exit positions that are not ranking any more
for symbol in my_positions:
    position = my_positions[symbol]
    if position.position > 0 and not any(stock.symbol == position.symbol for stock in longs):
        order = MyOrder(position.symbol)
        order.description = "Closing long position."
        order.type = "EXIT"
        order.amount = -1 * position.position
        order.currency = position.currency
        order_book.append(order)

for symbol in my_positions:
    position = my_positions[symbol]
    if position.position < 0 and not any(stock.symbol == position.symbol for stock in shorts):
        order = MyOrder(position.symbol)
        order.description = "Closing short position."
        order.type = "EXIT"
        order.amount = -1 * position.position
        order.currency = position.currency
        order_book.append(order)

As you can see I am storing any potential orders in a list called order_book and don’t actually place them yet. The reason for that is that I want to display them on screen and ask the user to confirm them before sending them off to TWS for execution. I also don’t want this strategy to continue running if there are any open orders still existent. I do this as a precautionary measure:

if len(order_book) > 0:
    confirm = input("Do you want to place the above exit orders (yes/no)? ")
    if confirm == 'yes':
        open_orders = app.get_open_orders()
        if open_orders:
            log.debugv("There are pending orders in the system.")
            log.debugv(print_open_orders(open_orders))
            log.debugv("Not executing. Bye!")
            app.disconnect()
            exit()
        else:
            for order in order_book:
                if order.currency is not None:
                    app.order_target_value(symbol=order.symbol, amount=order.amount, currency=order.currency)
                else:
                    app.order_target_value(symbol=order.symbol, amount=order.amount)
    else:
        log.debugv("Decline execution accepted. Not executing. Bye!")
        app.disconnect()
        exit()
else:
    log.debugv("Portfolio seems up-to-date. Not executing. Bye!")
    app.disconnect()
    exit()

_config.yml

Once I have exited all my unwanted positions I go ahead and calculate any adjustments that I need to undertake on any current positions and prepare the order for any new positions. Again I am storing any orders in the order_book list and ask the user to confirm them before sending the off to TWS:

order_book = []
log.debugv("No remaining pending orders.")

# Buy/Sell/Adjust
for stock in longs:
    if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0):
        order = MyOrder(stock.symbol)
        order.description = "New long position {0} x {1}".format(stock.order_amount, stock.symbol)
        order.type = "NEW"
        order.amount = stock.order_amount
        order_book.append(order)
    else:
        position = my_positions[stock.symbol]
        target_position = stock.order_amount
        if position.position < 0:
            stock.order_amount = target_position + abs(position.position)
        elif position.position > 0:
            if position.position != target_position:
                stock.order_amount = target_position - position.position
            else:
                stock.order_amount = 0

        if stock.order_amount != 0:
            order = MyOrder(stock.symbol)
            order.description = "Adjusting existing long position for {0} from {1} -> {2}".format(
                stock.symbol, my_positions[stock.symbol].position, target_position)
            order.type = "ADJUSTMENT"
            order.amount = stock.order_amount
            order_book.append(order)

for stock in shorts:
    if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0):
        order = MyOrder(stock.symbol)
        order.description = "New short position {0} x {1}".format(stock.order_amount, stock.symbol)
        order.type = "NEW"
        order.amount = stock.order_amount
        order_book.append(order)
    else:
        position = my_positions[stock.symbol]
        target_position = stock.order_amount
        if position.position > 0:
            stock.order_amount = target_position - position.position
        elif position.position < 0:
            if position.position != target_position:
                stock.order_amount = target_position - position.position
            else:
                stock.order_amount = 0

        if stock.order_amount != 0:
            order = MyOrder(stock.symbol)
            order.description = "Adjusting existing short position for {0} from {1} -> {2}".format(
                stock.symbol, my_positions[stock.symbol].position, target_position)
            order.type = "ADJUSTMENT"
            order.amount = stock.order_amount
            order_book.append(order)

In the above code snippet you can see that I am also storing whether a particular trade is caused by a new position or adjustment. It might have been more simple to just exit all positions in my current portfolio and then just go long the top and short the bottom companies but if a position doesn’t change or just needs a minor adjustment this would lead to unnecessary transaction costs. You might even ignore an adjustment if it is too small.

Finally, I again do some checking for any open orders and ask the user to confirm the new portfolio before placing the orders with my broker:

if len(order_book) > 0:
    confirm = input("Do you want to place the above orders (yes/no)? ")
    if confirm == 'yes':
        open_orders = app.get_open_orders()
        if open_orders:
            log.debugv("There are pending orders in the system.")
            log.debugv(print_open_orders(open_orders))
            log.debugv("Not executing. Bye!")
            app.disconnect()
            exit()
        else:
            for order in order_book:
                if order.currency is not None:
                    app.order_target_value(symbol=order.symbol, amount=order.amount, currency=order.currency)
                else:
                    app.order_target_value(symbol=order.symbol, amount=order.amount)
            open_orders = app.get_open_orders()
            log.debugv("Waiting for any pending orders to be filled.")
            while open_orders:
                sleep(0.1)
            log.debugv("No remaining pending orders. Exiting. Bye!")
            app.disconnect()
            exit()
    else:
        log.debugv("Decline execution accepted. Not executing. Bye!")
        app.disconnect()
        exit()
else:
    log.debugv("Portfolio seems up-to-date. Not executing. Bye!")
    app.disconnect()
    exit()
else:
log.debugv("Filter resulted in less than %d stocks." % (TARGET_NUMBER_OF_POSITIONS))

Below you will find the full code for the strategy. You simply have to run it using:

python strategy.py
# Import ibapi deps
from ibapi import wrapper
from ibapi.client import EClient
from ibapi.contract import *
from ibapi.common import BarData
from ibapi.common import TickerId
from ibapi.common import TickAttrib
from ibapi.ticktype import TickType
from ibapi.ticktype import TickTypeEnum
from ibapi.order import *
from ibapi.order_state import OrderState
from threading import Thread
from datetime import datetime
from datetime import timedelta
from time import time
from time import sleep
import pandas as pd
from fundamentalslib.fundamentals import Fundamentals
from fundamentalslib.fundamentals import StockProfile
import random
from termcolor import colored
import math
import io
from logging import getLoggerClass, addLevelName, setLoggerClass, NOTSET
import logging
import logging.handlers

# Use epoch as first orderId at api app startup, and increment it with each order, 
# this way it ensures uniqueness and bigger across multiple clientId and each app invocation
next_valid_order_id = 0
MARKET_ID = 19004
TARGET_NUMBER_OF_POSITIONS = 10
MAX_SHORT_POSITION_SIZE = TARGET_NUMBER_OF_POSITIONS // 2
MAX_LONG_POSITION_SIZE = TARGET_NUMBER_OF_POSITIONS // 2
# Create a global DataFrame to insert the data in
historical_data = pd.DataFrame(
    columns=['Date', 'Open', 'Close', 'High', 'Low'])
positions = {}
position_end = False
open_orders = {}
open_orders_end = False
account_summary_end = False
account_summary = {}
close_price = None
log = None
DEBUG_LEVELV_NUM = 9


class MyLogger(getLoggerClass()):
    def __init__(self, name, level=NOTSET):
        super().__init__(name, level)
        addLevelName(DEBUG_LEVELV_NUM, "STRATEGY_DEBUG")

    def debugv(self, msg, *args, **kwargs):
        if self.isEnabledFor(DEBUG_LEVELV_NUM):
            self._log(DEBUG_LEVELV_NUM, msg, args, **kwargs)


class Position(object):
    def __init__(self, symbol=None):
        self.symbol = symbol
        self.currency = None
        self.sec_type = None
        self.position = 0

    def __str__(self):
        return "Symbol: {0}  Currency: {1}  Position: {2}  Sec Type: {3}".format(self.symbol, self.currency, self.position, self.sec_type)


class MyOrder(object):
    def __init__(self, symbol=None):
        self.symbol = symbol
        self.currency = None
        self.amount = 0
        self.description = None
        self.type = None

    def __str__(self):
        return "Symbol: {0}  Currency: {1}  Amount: {2}  Description: {3}  Type: {4}".format(self.symbol, self.currency, self.amount, self.description, self.type)


class Wrapper(wrapper.EWrapper):
    def __init__(self):
        wrapper.EWrapper.__init__(self)

    def historicalData(self, reqId: int, bar: BarData):
        global historical_data
        historical_data = historical_data.append({'Date': str(bar.date), 'Open': float(bar.open), 'Close': float(
            bar.close), 'High': float(bar.high), 'Low': float(bar.low)}, ignore_index=True)

    def historicalDataEnd(self, reqId: int, start: str, end: str):
        log.debugv("Done downloading historical data (request_id=%s)." % (reqId))

    def accountSummary(self, reqId: int, account: str, tag: str, value: str, currency: str):
        super().accountSummary(reqId, account, tag, value, currency)
        global account_summary
        account_summary[tag] = value

    def accountSummaryEnd(self, reqId: int):
        super().accountSummaryEnd(reqId)
        log.debugv("Done fetching account summary (request_id=%s)." % (reqId))
        global account_summary_end
        account_summary_end = True

    # https://api.lynx.academy/Account&PortfolioData?id=positions
    def position(self, account: str, contract: Contract, position: float, avgCost: float):
        super().position(account, contract, position, avgCost)
        global positions
        pos = Position(contract.symbol)
        pos.position = position
        pos.sec_type = contract.secType
        pos.currency = contract.currency
        positions[contract.symbol] = pos

    def positionEnd(self):
        super().positionEnd()
        log.debugv("Done fetching positions.")
        global position_end
        position_end = True

    def tickPrice(self, reqId: TickerId, tickType: TickType, price: float, attrib: TickAttrib):
        super().tickPrice(reqId, tickType, price, attrib)
        if tickType == TickTypeEnum.CLOSE:
            global close_price
            close_price = price
            log.debugv("Got close price %s for request_id=%s." %
                       (close_price, reqId))

    def openOrder(self, orderId: int, contract: Contract, order: Order, orderState: OrderState):
        log.debugv("Open %s order with ID=%s Symbol=%s State=%s." %
                   (order.action, orderId, contract.symbol, orderState.status))
        global open_orders

        # https://interactivebrokers.github.io/tws-api/order_submission.html#order_status
        if orderState.status == 'PreSubmitted':
            open_orders[orderId] = order
        elif orderState.status == 'Filled' and orderId in open_orders:
            del open_orders[orderId]
        elif orderState.status == 'PendingCancel' and orderId in open_orders:
            del open_orders[orderId]

    def openOrderEnd(self):
        super().openOrderEnd()
        log.debugv("Done fetching open orders.")
        global open_orders_end
        open_orders_end = True

    def nextValidId(self, orderId: int):
        """
        Receives next valid order id.
        The LYNXApi.EWrapper.nextValidID callback is commonly used to indicate that the connection is completed
        and other messages can be sent from the API client to TWS. There is the possibility that function calls
        made prior to this time could be dropped by TWS.
        https://api.lynx.academy/Connectivity
        """
        log.debugv("Got next_valid_order_id=%d." % orderId)
        global next_valid_order_id
        next_valid_order_id = orderId


class Client(EClient):
    def __init__(self, wrapper):
        EClient.__init__(self, wrapper)

    def get_historicalData(self, contract, duration="1 M", barSize="1 hour", reqId=1):
        log.debugv("Requesting historical data for '%s'." % (contract.symbol))
        # Define the end date of the query
        queryTime = (datetime.today() - timedelta(days=0)).strftime("%Y%m%d %H:%M:%S")

        # Here we are requesting historical bar data for the the contract
        self.reqHistoricalData(reqId, contract, queryTime,
                               duration, barSize, "MIDPOINT", 1, 1, False, [])
        MAX_WAITED_SECONDS = 5
        log.debugv(
            "Getting historical data from the server... can take %d second to complete." % MAX_WAITED_SECONDS)
        sleep(MAX_WAITED_SECONDS)
        global historical_data
        return historical_data

    def get_close_price(self, symbol):
        contract = Contract()
        contract.symbol = symbol
        contract.secType = "STK"
        contract.currency = "USD"
        contract.exchange = "SMART"
        contract.primaryExchange = "NASDAQ"
        global close_price
        close_price = None
        reqId = self.get_next_request_id()

        # 1 for live, 4 for delayed-frozen data if live is not available
        self.reqMarketDataType(4)
        self.reqMktData(reqId, contract, "", False, False, [])

        log.debugv("Getting close price for '%s' (request_id=%d) from the server." % (
            contract.symbol, reqId))

        while close_price is None:
            sleep(0.1)
        self.cancelMktData(reqId)
        return close_price

    def get_positions(self):
        global position_end
        position_end = False
        self.reqPositions()
        while not position_end:
            sleep(0.1)
        global positions
        return positions

    def get_portfolio_value(self):
        request_id = int(time())
        global account_summary_end
        account_summary_end = False
        self.reqAccountSummary(request_id, "All", "$LEDGER:ALL")
        while not account_summary_end:
            sleep(0.1)
        global account_summary
        return float(account_summary["TotalCashBalance"])

    def order_target_value(self, symbol, amount, currency="USD"):
        if amount == 0:
            log.debugv("Error: amount cannot be zero.")
            return
        contract = Contract()
        contract.symbol = symbol
        contract.secType = "STK"
        contract.currency = currency
        contract.exchange = "SMART"
        contract.primaryExchange = "NASDAQ"
        # Define the order to place
        order = Order()
        if (amount > 0):
            order.action = "BUY"
        else:
            order.action = "SELL"
        order.orderType = "MKT"
        order.totalQuantity = abs(amount)
        self.place_order(contract, order)

    def place_order(self, contract, order):
        request_id = self.get_valid_order_id()
        log.debugv("%s (%s) %d %s (Order ID=%s)" % (
            order.action, order.orderType, order.totalQuantity, contract.symbol, request_id))
        self.placeOrder(request_id, contract, order)

    def get_open_orders(self):
        log.debugv("Fetching any open orders.")
        global open_orders_end
        open_orders_end = False
        self.reqAllOpenOrders()
        while not open_orders_end:
            sleep(0.1)
        global open_orders
        return open_orders

    def get_next_request_id(self):
        sleep(2)
        request_id = int(time())
        log.debugv("Generated the next request_id=%d." % (request_id))
        return request_id

    def get_valid_order_id(self):
        log.debugv("Getting valid order id from the server.")
        global next_valid_order_id
        next_valid_order_id = None
        self.reqIds(numIds=1)
        while next_valid_order_id is None:
            sleep(0.1)
        return next_valid_order_id


class Strategy(Wrapper, Client):
    def __init__(self, ipaddress, portid, clientid):
        Wrapper.__init__(self)
        Client.__init__(self, wrapper=self)
        self.connect(ipaddress, portid, clientid)
        """
        API programs always have at least two threads of execution.
        One thread is used for sending messages to TWS, and another thread is used for reading returned messages.
        The second thread uses the API EReader class to read from the socket and add messages to a queue.
        Everytime a new message is added to the message queue, a notification flag is triggered to let other threads
        now that there is a message waiting to be processed.
        The thread responsible for the message queue will decode messages and invoke the appropriate functions in EWrapper.
        In the two-thread design of an API program, the message queue is also processed by the first thread.
        Once the client is connected, a reader thread will be automatically created to handle incoming messages and put the messages
        into a message queue for further process. User is required to trigger Client::run() below, where the message queue is processed
        in an infinite loop and the EWrapper call-back functions are automatically triggered.
        https://api.lynx.academy/Connectivity
        """
        thread = Thread(target=self.run)
        thread.start()
        setattr(self, "_thread", thread)


def last_quarter():
    now = datetime.now()
    quarter_no = (now.month - 1) // 3 + 1
    return "Q{}".format(quarter_no-1)


def year():
    return datetime.now().year


def filter_stock_profiles(stock_profiles):
    """[summary]
    Keep only stock profiles whose industry sector is not 'Public Utilities' or 'Finance'.
    """
    log.debugv("Filtering stocks.")
    filtered_stock_profiles = list(filter(lambda stock:
                                          (stock.price_earnings is not None)
                                          and (not math.isnan(stock.price_earnings))
                                          and (stock.price_book is not None)
                                          and (not math.isnan(stock.price_book)), stock_profiles))
    filtered_stock_profiles = list(filter(lambda stock: (stock.industry is not None) and (
        stock.industry != 'Financials') and (stock.industry != 'Utilities'), filtered_stock_profiles))
    return filtered_stock_profiles


def rank_stocks(stock_profiles):
    """
    Ranks stocks by value.
    Arguments:
    stock_profiles -- A list of stock objects to rank.
    """
    return sorted(stock_profiles, key=lambda stock: 1.0 / (float(stock.price_earnings) + float(stock.price_book)), reverse=True)


def print_stocks(stock_profiles, title, limit):
    writer = io.StringIO()
    writer.write("\n%s\n" % (title))
    buf = ('%-4s %-6s %-35s %-25s %25s %25s %25s %25s\n' %
           ('#', 'Symbol', 'Name', 'Industry', 'PE', 'PB', 'Value', 'Weight'))
    writer.write(buf)
    writer.write('-' * len(buf))
    writer.write("\n")
    for i, stock in enumerate(stock_profiles):
        name = stock.name[: 30] if stock.name is not None else "N/A"
        industry = stock.industry[: 30] if stock.industry is not None else "N/A"
        price_earnings = stock.price_earnings if stock.price_earnings is not None else 0.0
        price_book = stock.price_book if stock.price_book is not None else 0.0
        value = 1.0 / (float(stock.price_earnings) + float(stock.price_book)) if (
            stock.price_earnings is not None) and (stock.price_book is not None) else 0.0
        weight = stock.portfolio_weight
        writer.write(
            '%-4d %s %-35s %-25s %25s %25.2f %25.3f %25s\n' %
            (i + 1, colored("%-6s" % stock.symbol, 'green'),
             name,
             industry,
             price_earnings,
             price_book,
             value,
             ("{:.0%}".format(weight))))
        limit -= 1
        if limit == 0:
            break
    return writer.getvalue()


def print_positions(positions):
    writer = io.StringIO()
    writer.write("\nCURRENT POSITIONS\n")
    buf = ('%-6s %-25s %-25s %25s\n' %
           ('Symbol', 'Position', 'Currency', 'Sec Type'))
    writer.write(buf)
    writer.write('-' * len(buf))
    writer.write("\n")
    for symbol in positions:
        position = positions[symbol]
        if position.position != 0:
            s = position.symbol if position.symbol is not None else "N/A"
            p = position.position
            c = position.currency if position.currency is not None else "N/A"
            t = position.sec_type if position.sec_type is not None else "N/A"
            writer.write('%-6s %-25s %-25s %25s\n' % (s, p, c, t))
    return writer.getvalue()


def print_order_book(order_book):
    writer = io.StringIO()
    writer.write("\nORDER BOOK\n")
    buf = ('%-25s %-25s %-25s %-65s\n' %
           ('Type', 'Symbol', 'Amount', 'Description'))
    writer.write(buf)
    writer.write('-' * len(buf))
    writer.write("\n")
    for order in order_book:
        t = order.type if order.type is not None else "N/A"
        s = order.symbol if order.symbol is not None else "N/A"
        a = order.amount
        d = order.description if order.description is not None else "N/A"
        color = 'green'
        if t == 'EXIT':
            color = 'red'
        elif t == 'ADJUSTMENT':
            color = 'yellow'
        writer.write('%-25s %-25s %-25s %-65s\n' %
                     ((colored("%-25s" % t, color), s, a, d)))
    return writer.getvalue()


def print_open_orders(orders):
    writer = io.StringIO()
    writer.write("\nOPEN_ORDERS\n")
    buf = ('%-25s %-25s %-25s %-25s\n' % ('ID', 'Action', 'Quantity', 'Type'))
    writer.write(buf)
    writer.write('-' * len(buf))
    writer.write("\n")
    for order_id in orders:
        order = orders[order_id]
        a = order.action
        q = order.totalQuantity
        t = order.orderType
        writer.write('%-25s %-25s %-25s %-25s\n' % (order_id, a, q, t))
    return writer.getvalue()


def main():
    setLoggerClass(MyLogger)

    global log
    log = logging.getLogger(__name__)
    log.setLevel(DEBUG_LEVELV_NUM)

    ch = logging.StreamHandler()
    fh = logging.handlers.RotatingFileHandler(
        'strategy.log', maxBytes=20000, backupCount=5)
    formatter = logging.Formatter('[%(asctime)s]: %(message)s')
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)
    log.addHandler(fh)
    log.addHandler(ch)
    log.propagate = False

    app = Strategy(ipaddress="localhost", portid=1234, clientid=0)

    log.debugv("Server version=%s Connection time=%s." %
               (app.serverVersion(), app.twsConnectionTime()))

    if not app.twsConnectionTime():
        log.debugv("Could not connect to TWS. Exiting. Bye!")
        exit()

    fundamentals = Fundamentals('database.ini')
    y = year()
    q = last_quarter()

    log.debugv("Fetching stocks from database for %s/%s." % (y, q))

    stocks = fundamentals.get_stock_profiles(y, q)

    log.debugv("Fetched %s stocks from database." % (len(stocks)))

    for stock in stocks:
        stock.price = app.get_close_price(stock.symbol)

    log.debugv("Fetched prices for all symbols.")

    stocks = filter_stock_profiles(stocks)
    if (len(stocks) >= TARGET_NUMBER_OF_POSITIONS):
        stocks = rank_stocks(stocks)
        longs = stocks[:MAX_LONG_POSITION_SIZE]
        shorts = stocks[-MAX_SHORT_POSITION_SIZE:]

        # Rebalance
        portfolio_value = app.get_portfolio_value()

        # don't overweight position sizes if the pickings are slim
        amount = portfolio_value // TARGET_NUMBER_OF_POSITIONS
        my_positions = app.get_positions()
        amount_available_for_long = (portfolio_value // 2)
        number_of_long_units = (MAX_LONG_POSITION_SIZE * (MAX_LONG_POSITION_SIZE + 1)) / 2
        amount_per_long_unit = amount_available_for_long // number_of_long_units

        log.debugv("Portfolio value=%s, amount_available_for_long=%s, number_of_long_units=%s, amount_per_long_unit=%s." %
                   (portfolio_value, amount_available_for_long, number_of_long_units, amount_per_long_unit))

        for rank, stock in enumerate(longs):
            # To diminish market impact, I invest less in stocks with very low liquidity.
            # if data.can_trade(stock):
            dollar_amount = (MAX_LONG_POSITION_SIZE - rank) * \
                amount_per_long_unit
            amount = dollar_amount // stock.price
            stock.portfolio_weight = (amount * stock.price) / portfolio_value
            stock.order_amount = amount

        amount_available_for_short = (portfolio_value // 2)
        number_of_short_units = (MAX_SHORT_POSITION_SIZE * (MAX_SHORT_POSITION_SIZE + 1)) / 2
        amount_per_short_unit = amount_available_for_short // number_of_short_units

        log.debugv("Portfolio value=%s, amount_available_for_short=%s, number_of_short_units=%s, amount_per_short_unit=%s." %
                   (portfolio_value, amount_available_for_short, number_of_short_units, amount_per_short_unit))

        for rank, stock in enumerate(shorts):
            # To diminish market impact, I invest less in stocks with very low liquidity.
            # if data.can_trade(stock):
            dollar_amount = (rank + 1) * amount_per_short_unit
            amount = dollar_amount // stock.price
            stock.portfolio_weight = (amount * stock.price) / portfolio_value
            stock.order_amount = -1 * amount

        log.debugv(print_stocks(longs, "NEW PORTFOLIO - LONGS", MAX_LONG_POSITION_SIZE))
        log.debugv(print_stocks(shorts, "NEW PORTFOLIO - SHORTS", MAX_SHORT_POSITION_SIZE))

        if any(stock.order_amount == 0 for stock in longs):
            log.error("Error: some long stocks have zero order amount. Exiting.")
            app.disconnect()
            exit()

        if any(stock.order_amount == 0 for stock in shorts):
            log.error("Error: some short stocks have zero order amount. Exiting.")
            app.disconnect()
            exit()

        log.debugv(print_positions(my_positions))

        order_book = []
        # Exit positions that are not ranking any more
        for symbol in my_positions:
            position = my_positions[symbol]
            if position.position > 0 and not any(stock.symbol == position.symbol for stock in longs):
                order = MyOrder(position.symbol)
                order.description = "Closing long position."
                order.type = "EXIT"
                order.amount = -1 * position.position
                order.currency = position.currency
                order_book.append(order)

        for symbol in my_positions:
            position = my_positions[symbol]
            if position.position < 0 and not any(stock.symbol == position.symbol for stock in shorts):
                order = MyOrder(position.symbol)
                order.description = "Closing short position."
                order.type = "EXIT"
                order.amount = -1 * position.position
                order.currency = position.currency
                order_book.append(order)

        log.debugv(print_order_book(order_book))

        if len(order_book) > 0:
            confirm = input("Do you want to place the above exit orders (yes/no)? ")
            if confirm == 'yes':
                open_orders = app.get_open_orders()
                if open_orders:
                    log.debugv("There are pending orders in the system.")
                    log.debugv(print_open_orders(open_orders))
                    log.debugv("Not executing. Bye!")
                    app.disconnect()
                    exit()
                else:
                    for order in order_book:
                        if order.currency is not None:
                            app.order_target_value(
                                symbol=order.symbol, amount=order.amount, currency=order.currency)
                        else:
                            app.order_target_value(
                                symbol=order.symbol, amount=order.amount)
            else:
                log.debugv("Decline execution accepted. Not executing. Bye!")
                app.disconnect()
                exit()
        else:
            log.debugv("Portfolio seems up-to-date. Not executing. Bye!")
            app.disconnect()
            exit()

        open_orders = app.get_open_orders()

        log.debugv("Waiting for any pending orders to be filled.")

        while open_orders:
            sleep(0.1)

        order_book = []

        log.debugv("No remaining pending orders.")

        # Buy/Sell/Adjust
        for stock in longs:
            if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0):
                order = MyOrder(stock.symbol)
                order.description = "New long position {0} x {1}".format(stock.order_amount, stock.symbol)
                order.type = "NEW"
                order.amount = stock.order_amount
                order_book.append(order)
            else:
                position = my_positions[stock.symbol]
                target_position = stock.order_amount
                if position.position < 0:
                    stock.order_amount = target_position + \
                        abs(position.position)
                elif position.position > 0:
                    if position.position != target_position:
                        stock.order_amount = target_position - position.position
                    else:
                        stock.order_amount = 0

                if stock.order_amount != 0:
                    order = MyOrder(stock.symbol)
                    order.description = "Adjusting existing long position for {0} from {1} -> {2}".format(
                        stock.symbol, my_positions[stock.symbol].position, target_position)
                    order.type = "ADJUSTMENT"
                    order.amount = stock.order_amount
                    order_book.append(order)

        for stock in shorts:
            if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0):
                order = MyOrder(stock.symbol)
                order.description = "New short position {0} x {1}".format(stock.order_amount, stock.symbol)
                order.type = "NEW"
                order.amount = stock.order_amount
                order_book.append(order)
            else:
                position = my_positions[stock.symbol]
                target_position = stock.order_amount
                if position.position > 0:
                    stock.order_amount = target_position - position.position
                elif position.position < 0:
                    if position.position != target_position:
                        stock.order_amount = target_position - position.position
                    else:
                        stock.order_amount = 0

                if stock.order_amount != 0:
                    order = MyOrder(stock.symbol)
                    order.description = "Adjusting existing short position for {0} from {1} -> {2}".format(stock.symbol, my_positions[stock.symbol].position, target_position)
                    order.type = "ADJUSTMENT"
                    order.amount = stock.order_amount
                    order_book.append(order)

        log.debugv(print_order_book(order_book))

        if len(order_book) > 0:
            confirm = input("Do you want to place the above orders (yes/no)? ")
            if confirm == 'yes':
                open_orders = app.get_open_orders()
                if open_orders:
                    log.debugv("There are pending orders in the system.")
                    log.debugv(print_open_orders(open_orders))
                    log.debugv("Not executing. Bye!")
                    app.disconnect()
                    exit()
                else:
                    for order in order_book:
                        if order.currency is not None:
                            app.order_target_value(
                                symbol=order.symbol, amount=order.amount, currency=order.currency)
                        else:
                            app.order_target_value(
                                symbol=order.symbol, amount=order.amount)
                    open_orders = app.get_open_orders()

                    log.debugv("Waiting for any pending orders to be filled.")

                    while open_orders:
                        sleep(0.1)

                    log.debugv("No remaining pending orders. Exiting. Bye!")

                    app.disconnect()
                    exit()
            else:
                log.debugv("Decline execution accepted. Not executing. Bye!")
                app.disconnect()
                exit()
        else:
            log.debugv("Portfolio seems up-to-date. Not executing. Bye!")
            app.disconnect()
            exit()
    else:
        log.debugv("Filter resulted in less than %d stocks." % (TARGET_NUMBER_OF_POSITIONS))


if __name__ == "__main__":
    main()
Written on August 19, 2021