High Performance Limit Order Book Construction: an Order Book Snapshot of Financial Markets With Nanosecond-Resolution Time Stamps

Mao Ye, University of Illinois at Urbana-Champaign

Usage Details

Our research is about high-frequency trading and the current plan includes: (1) Continue our progress of constructing an order-by-order level snapshot of financial markets with nanosecond-resolution time stamps; and (2) Experiment wavelet analysis on historical trading data.

With many improvements yet to achieve in our data-parsing algorithm for the snapshot construction, we started expanding our effort into the data analysis side. With our current 60+ terabytes of raw trading records and 150+ terabytes of processed snapshot data, most, if not all, traditional data analysis methods would perform poorly in terms of efficiency and scalability. We are facing real Big Data challenges, as our project is probably one of the largest, if not the largest, computing effort ever in social, behavioral, and economic sciences.

As part of our normal routine, our researchers actively come up with immature ideas and want to analyze the data in a timely manner. For example, we currently intend to run wavelet analysis on 11-years of historical data, but our developed R code performs poorly on the large-scale wavelet analysis. Thus, we tentatively plan to integrate some matured and well-maintained open-source big data platforms/frameworks into our development process, with which we could experiment on large-scale data. We are considering some most popular framework such as the combo of Pig + Apache Hadoop YARN or Apache Spark or alternatives that are available on Blue Water.