BloombergGPT: Domain-specific LLM

In 2018, Bidirectional Encoder Representations from Transformers (BERT) managed to excite many of us working in the AI field. Machine Learning Models were learning to read, process, regurgitate and even speak. Today, Large Language Models (LLMs), are rapidly advancing, demonstrating dexterity in a multitude of applications and by extension, industries, and sectors.

LLMs are being exploited to advance drug discovery led by among others the Rostlab at the Technical University of Munich, as well as work by a team from Harvard, Yale, and New York University. In independent efforts, LLMs have been applied to interpret the strings of amino acids that form the basis of proteins, which has advanced our understanding of these biological building blocks and given us tremendous insights into mother nature’s ‘kitchen.’

Furthermore, LLMs are making inroads into sectors such as healthcare, robotics, manufacturing, and numerous other fields. It was only a matter of time before we saw this disruption permeate into the FinTech sphere.

Until now, no domain specific LLMs have been publicly announced in the financial sector. With the announcement of ‘BloombergGPT,’ we can only expect this to become the beginning of an endemic trend in sector specific LLM development and adoption.

BloombergGPT has been presented as a fifty-billion parameter language model that is trained on a broad range of financial data. It is constructed on a 363 billion token dataset provided by Bloomberg's extensive data resources. This is perhaps the largest domain-specific dataset thus far, further augmented with 345 billion tokens from general-purpose datasets.

The LLM’s creators have exploited a mixed dataset to train the model which they advocate enables it to surpass existing models in engaging financial tasks by substantial margins without sacrificing performance against the general LLM benchmarks.

LLMs represent an advancement in Transformer AI based on transformer models which are artificial neural networks that learn context and thus meaning by tracking relationships in sequential data like the words in this article.

Transformer AI can be used to detect trends and identify anomalies to prevent fraud, detect money laundering and assess credit decisions. In addition, it can be used to streamline manufacturing, make online recommendations, or improve financial decision-making. It is a dynamic and versatile technology which may be exploited in numerous contexts.

People use ‘transformers’ every time they search using Google or Microsoft Bing.

Since the release of OpenAI’s LLMs – ChatGTP and GPT-4, LLMs have entered the mainstream.

LLMs are higher-order systems built upon the foundations of Transformer AI. In the FinTech sphere applications which exploit LLMs may be broad and diverse which include areas such sentiment analysis, named entity recognition, and potentially even algorithmic trading. Other applications may include ‘Call Centre’ Natural Language Processing (NLP), Credit Scoring, and Robo-Advisory services.

I discussed many of these dimensions in my book, Bad Money.

In an extreme use case the technology may be used to attempt to identify self-organised criticality in financial systems by attempting to map complex behaviours in multibody systems such as financial markets.

What applications do you envision this technology may be used for?

Kindly comment and share your thoughts below.