Why a Hedge Fund Built Its Own Database
16 Sep 2024 (2 days ago)
Man Group and ArcticDB
- James Monroe works at Man Group, an alternative asset management firm.
- Man Group manages over $160 billion in assets and trades $6 trillion or more annually.
ArcticDB Development and Use Cases
- Arctic DB was initially developed around 2011 as a Python-based database solution utilizing MongoDB as its backend.
- M Group rewrote the core of their system in C++ and removed the separate database layer to improve performance and scale. This system is now used across M Group and by banks, asset managers, and data providers.
Justification for Building a Database
- The speaker argues that building a database, while seemingly counterintuitive, can be justified by the need for specialized tools, particularly when dealing with high-frequency data.
- Research productivity is crucial for systematic quant hedge funds like M Group, as it drives the generation of alpha and portfolio optimization.
Challenges in Quantitative Finance
- Research productivity is a key challenge in quantitative finance due to increasing data, competition, and market efficiency.
- Asset managers utilize high-frequency, low-latency market data, including alternative data such as weather, consumer, and environmental data.
Man Group's Data Science Transition
- In 2011, Man Group transitioned to Python for data science, despite the popularity of R, MATLAB, and C++ at the time, and supported the growth of the Python data science community.
- Eagle Alpha indexed thousands of datasets.
M Group's Data Science Operations
- M Group employs hundreds of quants who work on diverse problems using a variety of statistical methods, deep learning, and tools like ChatGPT.
- Systematic traders at M Group utilize a Lambda architecture, incorporating streaming and batch data pipelines for high-frequency and fundamental/alternative data analysis, respectively.
Bond Data Organization and Analysis
- Bonds represent a larger market than equities, being three times larger globally and even larger in the US.
- A normalized way of organizing bond data often involves using Python Pandas data frames with dates, bond IDs (like CUSIPs and ISINs), prices, and fundamental data points like duration.
- A more practical approach for analysis involves pivoting this data to have bond IDs as columns, time series data for a single measure (like price) as rows, which allows for efficient time series and cross-sectional analysis.
Database Design Considerations
- Columnar storage is beneficial for large tables, especially when users primarily need to read specific columns, which can involve billions or trillions of rows for data like tick data or decades of daily data.
- It is important to design databases that can handle both large tables and a high volume of columns, potentially hundreds of thousands or even millions, to support cross-sectional analysis.
- In database design, prioritizing agility and performance may involve sacrificing some degree of isolation, which helps users coordinate transactions from multiple sources.
Serverless Database Architecture
- A 2008 paper proposed building a database on Amazon S3, highlighting the potential of running databases without servers.
- The proposed system aimed to achieve atomicity, consistency, and delegate durability to the storage system, utilizing an immutable data structure where new versions are added instead of modifying existing ones.
- Serverless databases, while potentially sacrificing some transactional capabilities, offer advantages such as a heavier-weight client handling indexing and query execution, and reliance on robust storage systems like S3 for security, capacity, and resiliency.
ArcticDB Features and Benefits
- ArcticDB scales alongside its users, meaning those conducting the most work benefit from the most database resources.
- ArcticDB offers a serverless architecture, allowing users to focus on data storage without server management.
- ArcticDB provides database semantics for reading, writing, updating, and deleting data from data frames.
- ArcticDB allows users to update data in the middle of a data frame, automatically updating indexing and versions.
- ArcticDB is capable of handling large datasets, with one example showing a data set with 100,000 columns and 100,000 rows.