Can ChatGPT Summarise a Legal Case Accurately? Legal Island Conducts an Experiment to Find Out

Since the moment OpenAI launched its chatbot ChatGPT, powered by artificial intelligence, users from around the world have been testing its capabilities and its limits. Legal Island is one of those inquisitive users. We recently conducted an experiment to test the chatbot's ability to write an employment law case review.

The Process

Legal Island inputted three NI decisions into ChatGPT and asked it to summarise each case using five bullet points.

The Results

According to OpenAI, ChatGPT has a limit of 3000 words while ChatGPT-4 has a limit of up to 25,000 words at a time (available to those who have subscribed to ChatGPT Plus).It was a challenging task to find a decision from the Employment Appeals Tribunal (EAT) or Industrial Tribunal (IT) that did not surpass the 3,000 words.

In our view, ChatGPT provided more of narrative rather than an accurate legal assessment of each of the decisions. In terms of accuracy, ChatGPT didn’t “make anything up” per se but instead it omitted vital information such as the outcome of the case (pretty important!) There is also the risk that in being so good at summarising, ChatGPT forgets or isn’t aware of the importance of obiter dictum comments made by judges – often these comments can be the most insightful part of a decision. Finally, the case reviews don’t provide any practical guidance for employers. This is distinct to Legal Island’s case reviews prepared by barristers who guide employers on the practical aspects of an EAT or IT decision. While ChatGPT’s speed of summary writing is very impressive indeed, in our opinion, it is safer for now to rely on the legal experts for an accurate assessment of a case! 

The chatbot summaries are set out below.

Mioduszewska v Western Urgent Care Limited [2023]

ChatGPT summarised the Northern Ireland Industrial Tribunal (NIIT) case as follows:

1. The claimant, a medical doctor, worked as a General Practitioner for the respondent from April 2018 until her dismissal on 26 July 2021.
2.  She lodged a complaint with the Office of the Industrial Tribunals and Fair Employment Tribunal (OITFET) on 26 October 2021, claiming unfair dismissal, sex discrimination and harassment, and detriment due to whistleblowing.
3.  The complaint was rejected because it lacked the required conciliation certificate from the Labour Relations Agency (LRA).
4. The time limits for unfair dismissal and whistleblowing claims were not met, and the claimant failed to provide sufficient grounds for extending the time limit.
5. The discrimination and harassment claims were also outside the time limit, and the claimant could not justify extending the time. As a result, all of the claimant's complaints were dismissed in their entirety.

To view and compare this with Legal Island’s full case review prepared by a real human(!) click here:
https://www.legal-island.com/articles/uk/case-law/2023/june/mioduszewska-v-western-urgent-care-limited-2023 

Rankin v Precept IT Limited [2023]

ChatGPT summarised the NIIT case as follows:

1. The respondent is a small business providing managed IT services, hardware, software, and IT support in Northern Ireland with 13 employees.
2.  The claimant worked as a Business Development Manager for the respondent from 1 February 2018 to 31 July 2022, responsible for securing new customers.
3. The claimant alleged underpayment of commission amounting to £14,026.18 from January 2020 to July 2021, and claimed breach of contract and unauthorised deduction from wages based on this alleged underpayment.
4. The respondent contested the claim and argued that all commission owed to the claimant had been paid.
5. The tribunal found that the claimant's calculations were unreliable and inaccurate, and the respondent's calculations were accurate. As a result, the claimant's claims for unauthorised deduction from wages and breach of contract were dismissed.

To view and compare this with Legal Island’s full case review prepared by a real human(!) click here:

https://www.legal-island.com/articles/uk/case-law/2023/may/rankin-v-precept-it-limited-2023/

Halpin v Stewart t/a Corporation Square Limited, Ben Stewart t/a Alexanders Holywood Limited [2023]

ChatGPT summarised the NIIT case as follows:

1. The respondents failed to pay the appropriate rate of National Minimum Wage to the claimant, resulting in an unlawful deduction of wages. They are ordered to pay the claimant £600.
2. The respondents unlawfully deducted wages by not paying the claimant's wages for one week in April 2022. They must pay the claimant £330.
3. The respondents made an unlawful deduction of wages by withholding wages for eight weeks in June and July 2022. They are ordered to pay the claimant £2,583.
4. The respondents unlawfully deducted wages by not providing the claimant with the agreed increase in wages. They are to pay the claimant £195.
5. The respondents made an unlawful deduction of wages by failing to pay the appropriate overtime rate and bank holiday pay to the claimant. They must pay the claimant £442.80.

To view and compare this with Legal Island’s full case review prepared by a real human(!) click here:

https://www.legal-island.com/articles/uk/case-law/2023/may/halpin-v-stewart-ta-corporation-square-limited-2023/