ChatGPT’s latest test: Accounting

When OpenAI’s ChatGPT was released for public use, it prompted people to do two things:

1. Test its ability to generate information around a topic of interest or start a conversation to experience its natural language capabilities
2. See how it stacks up against the knowledge of humans

Over the past few months, we have seen stories of these comparisons, including ChatGPT passing a Wharton Business School MBA exam [1] and the Uniform Bar Examination [2]. The latest news of ChatGPT taking on an exam designed for humans hit the accounting world when a global team led out of Brigham Young University administered a test on various accounting topics. Overall, ChatGPT was outperformed by its collegiate counterparts – 47.4% to 76.7%, respectively [3].

It is worth noting some performance results from the study, which featured over 27,000 questions input to ChatGPT, highlighted the strengths of ChatGPT and where we will continue to see improvement. ChatGPT did well on questions regarding accounting information systems and auditing, as well as questions in true/false and multiple-choice formats. Conversely, ChatGPT struggled with tax and financial topics and was not as successful when responding to short-answer questions.

What does this mean for the accounting profession?

The short answer is, for now, we really don’t know. This experiment was conducted with ChatGPT, OpenAI’s GPT-3.5. The company has since released GPT-4, which has already demonstrated significantly more capabilities than its predecessor. We have not yet seen GPT-4 applied in a similar fashion to demonstrate accounting acumen. However, based on this academic experiment, GPT-4 will need to improve responses for certain topics and in particular formats. Considering the rate of development within the AI space, it likely won’t be long before a similar experiment is conducted in the accounting industry and we see how the latest GPT fares.

It is clear companies within the industry need to continue to monitor the advancements of AI and find ways to integrate AI technologies into their accounting services. Uses for increased efficiency and automation feel inevitable, and Baker Tilly remains committed to being at the forefront of AI in the accounting and professional services space to understand how we can better perform our services and effectively bring value to our clients.