Data Servers

It’s tedious to specify a filename to load data so we usually rely on a BarServer. By using a BarServer, we only have to specify the directory once and then we can load BarSeries by symbol. The backtester has quite a few bar servers built in which can be found in the Balsam.DataServers namespace. A few of the most commonly used include:

Ascii BarServer

One the simplest is AsciiBarServer, which as the name implies, loads ASCII or plain text files. It can be used on its own or it can be extended to provide additional functionality like the CsiBarServer we cover next. Rather than calling BarSeries.Load() and passing in a filename, typically you would do something like this:

 var server = new AsciiBarServer(@"c:\data\temp");
 var bcom = server.LoadSymbol("BCOM Index");
 var data = server.LoadAll();

• In line 1, we create a new AsciiBarServer for data found in the C:\data\temp directory. Unless directly specified, the file format will be automatically inferred for common formats (e.g. DOHLC as used here). Additional overloads to the constructor allow you to specify the file extension, delimiter (comma, tab, etc) as well as the file layout.
• In line 2, LoadSymbol() returns a BarSeries of all available data for the symbol ‘BCOM Index’.
• In line 3, the call to LoadAll() returns all the data in the specified directory as a BarSeriesCollection. As its name implies, a BarSeriesCollection is a collection of one or more BarSeries. It can be indexed by ordinal position within the collection (e.g. data[0]) or by symbol (e.g. data["BCOM Index"]). BarSeriesCollection is useful for portfolio testing to run a backtest across multiple symbols simultaneously.

CSI BarServer

Futures traders often perform research on backadjusted contracts. One commonly used end-of-day data provider is CSI. Their Unfair Advantage software can output files in a variety of formats, but ASCII is unbeatable for its simplicity and interoperability. The backtester has a specialized AsciiBarServer, CsiBarServer designed to read text files output by Unfair Advantage. By convention, backadjusted files should be output using the following CSI format: “DOHLCVIUN”, short for:

• Date
• Open
• High
• Low
• Close (settlement)
• Volume
• Open interest
• Unadjusted close
• Delivery code

The CsiBarServer will load these bars as a ContinuousBar, a subclass of Bar which includes an additional property DeliveryDate. CsiBarServer also handles some symbol translations automatically, adjusting prices to exchange convention in some cases (HG, SI, JY) and using a more industry standard tickers in others (EC instead of CU for the Euro).

If no instrument file is found in the data directory, the CsiBarServer will automatically check the directory where the program is running to see if it can load a global instrument file.

Binary BarServer

The BinaryBarServer reads data stored in a custom binary format. Binary tends to be much quicker to load at the expense of not being able to view or edit the data with a text editor or Excel. But when working with large amounts of data, the time savings from using a binary format become significant. In the simple benchmark below, we load approximately 13 years of one minute intraday data containing ~4.8 million bars. On a modern machine with a NVMe SSD drive, the ASCII data loads in about 7.5 seconds. That same data stored in a binary file loads in just over a second. Binary files also store instrument metadata eliminating the need for a separate instrument file unless desired.

internal class Program
{
    static void Main(string[] args)
    {
        IBarServer server = new AsciiBarServer(@"c:\data\intraday\continuous");
        var sw = Stopwatch.StartNew();
        var data = server.LoadSymbol("@ES");
        sw.Stop();
        Console.WriteLine($"{data.Count:N0} ascii bars loaded in {sw.Elapsed}.");

        data.Save(@"c:\data\temp\@ES.bin", PersistenceFormat.Binary);
        server = new BinaryBarServer(@"c:\data\temp");
        sw = Stopwatch.StartNew();
        data = server.LoadSymbol("@ES");
        sw.Stop();
        Console.WriteLine($"{data.Count:N0} binary bars loaded in {sw.Elapsed}.");
    }
}

4,769,984 ascii bars loaded in 00:00:07.6500165.
4,769,984 binary bars loaded in 00:00:01.1827305.

Other BarServers

• TiingoServer accesses the Tiingo API, an excellent source for free or inexpensive equity data. Visit Tiingo to register and get a personal API key.
• TradeStationBarServer is a specialized AsciiBarServer for reading data exported from a TradeStation Data Window.
• FREDServer allows retrieval of many economic series, including point-in-time data, from the St. Louis Federal Reserve’s excellent FRED service. Register for a free API key.
• BloombergBarServer interfaces directly with Bloomberg to retrieve data via their Desktop API. Licensed as a separate entitlement from the core backtester.
• Or build your own bar server by overriding a few methods of the abstract BarServer base class.