65+ Statistical Insights into GPT-4: A Deeper Dive into OpenAI’s Latest LLM

With the recent surge of GPTs (Generative Pre-Trained Transformers) and the marketplace store connecting developers and users, OpenAI has developed an ecosystem that allows developers to create tailored versions of ChatGPT to acutely meet the daily needs and workflow processes of its target consumers. 

At Originality.ai, we are actively monitoring and studying the GPT market as well as the trends that lie beneath the numbers and will soon publish those insights. For now, we will look at the model behind the GPT store and custom GPTs, which also happens to be OpenAI’s most advanced publicly available LLM (Large Language Model), GPT-4. 

Read below to dive further into the many different processes, statistics, and trends that have all converged to make GPT-4 possible.

What is GPT-4? 

GPT-4, the latest milestone in OpenAI’s effort in scaling up deep learning. GPT-4 is a large multimodal model (accepting image and text inputs, emitting text outputs) that, while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks. (Source)

When Was the Latest GPT-4 Model Announced/Released? 

On Monday November 6, 2023 at the OpenAI DevDay event, company CEO Sam Altman announced a major update to its GPT-4 language model called GPT-4 Turbo, which can process a much larger amount of text than GPT-4 and features a knowledge cutoff of April 2023.  (Source)

How can GPT-4 be Accessed? 

GPT-4 currently sits behind a paywall. OpenAI has a subscription based model for consumers to access the more advanced forms of their ChatGPT model. Below are the current developments behind accessing GPT-4: 

• ChatGPT Plus subscribers will get GPT-4 access on chat.openai.com with a usage cap. (Source)
• OpenAI may introduce a new subscription level for higher-volume GPT-4 usage; they also hope at some point to offer some amount of free GPT-4 queries so those without a subscription can try it too. (Source)

GPT-4 Architecture 

A new report by SemiAnalysis reveals more details about OpenAI's GPT-4, concluding that "OpenAI is keeping the architecture of GPT-4 closed not because of some existential risk to humanity, but because what they've built is replicable”. (Source). As such, the following details stem from a recent GPT documentation leak and have not yet been confirmed by OpenAI: 

GPT-4's Scale: GPT-4 has ~1.8 trillion parameters across 120 layers, which is over 10 times larger than GPT-3 (Source)

Mixture Of Experts (MoE): OpenAI utilizes 16 experts within their model, each with ~111B parameters for MLP. Two of these experts are routed per forward pass, which contributes to keeping costs manageable. (Source)

Dataset: GPT-4 is trained on ~13T tokens, including both text-based and code-based data, with some fine-tuning data from ScaleAI and internally. (Source)

Dataset Mixture: The training data included CommonCrawl & RefinedWeb, totaling 13T tokens. Speculation suggests additional sources like Twitter, Reddit, YouTube, and a large collection of textbooks. (Source)

Training Cost: As of 2024, it’s estimated that OpenAI has spent $8.5 billion overall on training AI and staff. GPT-4 cost “$78 million worth of compute” to train. (Source and Source)

Inference Cost: GPT-4 costs 3 times more than the 175B parameter Davinci, due to the larger clusters required and lower utilization rates. (Source)

Inference Architecture: The inference runs on a cluster of 128 GPUs, using 8-way tensor parallelism and 16-way pipeline parallelism. (Source)

Vision Multi-Modal: GPT-4 includes a vision encoder for autonomous agents to read web pages and transcribe images and videos. The architecture is similar to Flamingo. This adds more parameters on top and it is fine-tuned with another ~2 trillion tokens. (Source)

Does GPT-4 Really Utilize Over 100 Trillion Parameters?  

When GPT-4 was first announced and subsequently released, it was heavily speculated that the new model was comprised of over 100 trillion parameters. After a couple months and a data leak containing some GPT-4 architecture details, the CEO of OpenAI, Sam Altman, was questioned about the matter: 

• When asked about one viral (and factually incorrect) chart that purportedly compares the number of parameters in GPT-3 (175 billion) to GPT-4 (100 trillion), Altman called it “complete bullshit.” (Source)
  ‍
• In reality, the reported parameter figure for GPT-4 is about 1.76 trillion, which is an enormous upgrade from the prior GPT models. (Source)

Training Process 

• OpenAI spent 6 months iteratively aligning GPT-4 using lessons from their adversarial testing program as well as ChatGPT, resulting in their best-ever results (though far from perfect) on factuality, steerability, and refusing to go outside of guardrails. (Source)
  ‍
• OpenAI has updated DALL·E so that it’s accessible to all ChatGPT users. Not long ago, image generation was only possible for paid users. (Source)
  ‍
• OpenAI makes the GPT codebase more publicly available by open-sourcing OpenAI Evals, their framework for automated evaluation of AI model performance, to allow anyone to report shortcomings in their models to help guide further improvements (Source)

• Compared to GPT-3's 17 gigabytes of data, GPT-4, the most recent iteration of OpenAI, has 45 gigabytes of training data. As a result, GPT-4 can deliver significantly more accurate results than GPT-3. (Source)

What is GPT-4’s Context Length? 

• GPT-4 has a maximum context length of exactly 32k (32,768) tokens. GPT-4’s Turbo version outputs about 128k tokens, an average of 400 words per page on generated text. (Source)
  ‍
• Previously, GPT-4 featured an 8,000-token context window, with a 32K model available through an API for some developers; now this context window has been surpassed for turbo models. (Source)
  ‍
• That means GPT-4 Turbo can consider around 25,000 words in one go, which is longer than many novels. Also, a 128K context length of the Turbo model can lead to much longer conversations without having the AI assistant lose its short-term memory of the topic at hand. (Source)

Introduction of GPT-4 Vision

Adding onto the text based capabilities of OpenAI’s GPT models, GPT-4 has introduced the possibility of interacting with GPT models through a visual capacity, look below to see the details behind “GPT-4-Vision”: 

• GPT-4 with vision (GPT-4V) enables users to instruct GPT-4 to analyze image inputs provided by the user. (Source)
  ‍
• Similar to GPT-4, training of GPT-4V was completed in 2022 and began providing early access to the system in March 2023 (Source)

GPT 4 Improvements on GPT-3.5

GPT-4 has proved to be a great success for OpenAI, making great improvements on the already impressive foundation that was established by ChatGPT and GPT-3.5. Below we can see some of the initial progress made by the new model and how it compares to the previous model, GPT-3.5: 

• In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle (Source)
  ‍
• GPT-4 is much more reliable and creative compared to GPT 3.5 and can handle complex and nuanced instructions, even in a casual conversation. (Source)
  ‍
• GPT-4 considerably outperforms existing large language models, alongside most state-of-the-art (SOTA) models which may include benchmark-specific crafting or additional training protocols (Source)
  ‍
• In the 26 languages tested at launch, GPT-4 outperforms the English-language performance of GPT-3.5 and other LLMs (Chinchilla, PaLM), including for low-resource languages such as Latvian, Welsh, and Swahili (Source)
  ‍
• GPT-4 significantly reduces hallucinations relative to previous models (which have themselves been improving with each iteration). In a study on hallucination rates, available through the National Library of Medicine (PubMed), GPT-4 exhibited a 28.6% hallucination rate compared to GPT-3.5, which had a higher hallucination rate of 39.6%. (Source)
  ‍
• OpenAI’s mitigations have significantly improved many of GPT-4’s safety properties compared to GPT-3.5. We’ve decreased the model’s tendency to respond to requests for disallowed content by 82% compared to GPT-3.5, and GPT-4 responds to sensitive requests (eg medical advice and self-harm) in accordance with their policies 29% more often. (Source)
  ‍
• Regarding GPT-4 costs, the set price for models operating on the 8k context lengths (GPT-4 and GPT-4-0314) is $30 for 1 million prompt tokens. 1K prompt tokens equals about $0.03, so you will have a fair amount of chat requests with the paid bot. (Source)
  ‍
• For many basic tasks, the difference between GPT-4 and GPT-3.5 models is not significant. However, in more complex reasoning situations, GPT-4 is much more capable than any of their previous models. (Source)

The following chart shows some of the progress made by each iteration of the GPT model when responding to legal inquiries:

GPT-4 API Pricing 

With 128k context, fresher knowledge and the broadest set of capabilities, GPT-4 Turbo is more powerful than GPT-4 and offered at a lower price. (Source)

GPT-4 Turbo API Pricing 

With broad general knowledge and domain expertise, GPT-4 can follow complex instructions in natural language and solve difficult problems with accuracy. (Source)

Comparing Latest GPT Models Available

As mentioned earlier, GPT-4 is a large multimodal model (accepting text or image inputs and outputting text) that can solve difficult problems with greater accuracy than any of the previous models, thanks to its broader general knowledge and advanced reasoning capabilities. Like gpt-3.5-turbo, GPT-4 is optimized for chat but works well for traditional completions tasks using the Chat Completions API. (Source)

GPT-4

GPT-4 Turbo

GPT-4 Areas for Improvement 

Even though GPT-4 has made many strides in improving the performance of its preceding model, there still remains avenues for OpenAI to improve upon the model’s accuracy and reliability. As detailed below, GPT-4 still presents opportunities to improve when considering factualness, relevancy, and accuracy: 

• GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its data cuts off (September 2021 and April 2023, depending on model version), and does not learn from its experience. (Source) 
  ‍
• Despite its capabilities, GPT-4 has similar limitations as earlier GPT models. Most importantly, it still is not fully reliable as it could “hallucinate” or misinterpret facts or statistics, especially about the code questions recently. (Source)
  ‍
• When measuring legal acumen, GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts, or fail to follow ethical requirements; for the foreseeable future, the application should feature ”human-in-the-loop” workflows or similar safeguards (Source)
  ‍
• We should also note that GPT-4, like prior models, struggles with tasks that require common sense, reasoning, or deep understanding, such as with law or bar-related questions.(Source)

The following metrics provided by OpenAI detail in-house testing that shows the gradual increases in accuracy scores for the different training methods used on their models. The scores reflect that although improvements have been made throughout the model’s generations, there is still much room for improvement:

How Effective is Originality.ai Against GPT-4?

Whether you use ChatGPT for research or planning, it’s important to keep in mind that AI shouldn’t be the sole source of information, as it can hallucinate or produce errors. It shouldn’t be entirely relied on for writing either, considering that the copy it generates may not provide the depth of value readers are looking for.

However, GPT-4 is still a highly popular tool, so we’ve decided to test it with Originality.ai’s AI detector. Can GPT-4 deceive AI detection tools with the right prompts and prevent AI checkers from identifying the text as AI-generated? We put together a series of tests to find out!

The tests feature a range of prompts with unique writing instructions to produce the most human responses possible from GPT-4. Let’s start with the first tests of GPT-4 and discover the efficacy of Originality.ai’s AI Checker.

[Test 1] Common AI-Generated GPT-4 Text

For the first test, we’ll compare the most common type of information generated by ChatGPT. We won’t add extra instructions to alter its output in any way. The aim of this test prompt is to determine how well it can conceal the AI-generated content. 

By default, all versions of ChatGPT (both GPT-3.5 and GPT-4) are designed to construct equally informative content when no extra prompts or instructions are included. So, let’s have a look at the first article we prompted ChatGPT to generate.

[Prompt #1] - Write a short article (500-1000 words) on the 2024 cybersecurity advancements.

We’ve received a 956-word article from GPT-4 and proceeded to test it on Originality.ai. Let’s review the results:

Originality.ai’s detection results are solid, stating that it has 100% confidence the text is AI-generated. Out of all 956 words, more than 98% of the sentences and a little over 900 words are highlighted as AI-generated.

Next, let’s move on to the second prompt to determine how Originality.ai performs!

[Prompt #2] - Write a short article (500-1000 words) on how accurate AI detection technology is in 2024.

Putting ChatGPT’s most recent version to the test with an article specifically about AI detection is another excellent method to test Originality.ai’s efficacy. Let’s have a look at the results:

From the second prompt, we received a 902-word output and the Originality.ai AI detector had 100% confidence that the content was AI-generated. For this prompt, we received two different GPT-4 generations for the second part of the article. After testing both possible responses, the detection results remained the same.

Now, let’s move on to more complex tests to determine if GPT-4 is capable of producing human-sounding content when prompted with unique instructions.

[Test 2] GPT-4 Generated Text With Extra Instructions

As shown in the previous test, commonly generated ChatGPT text can be easily recognized by AI detectors. However, does providing GPT-4 with extra instructions and tips on content structure improve the output and make it undetectable? 

In our previous tests of GPT-3.5, we provided it with a whole example article of 100% human-written content as an example to learn from. Yet, the detection results were still at 100% confidence that it was AI. 

Is there an improvement in GPT-4’s technology that allows it to conceal AI-generated content when prompted to do so? Let’s start with the first test to answer these top questions!

[Prompt #1] - Write a short article (500-1000 words) on the 2024 cyber security advancements. Use a natural and human-sounding tone, write 2-3 paragraphs for each heading, and implement SEO strategies. Construct the content so it cannot be recognized by AI detectors.

Let’s have a look at the results this prompt has brought up:

We’ve received an 849-word article output from GPT-4, and the results were once again solid, with 100% confidence that it was AI-generated. Concealing AI-detected content has proven challenging even with advanced prompt instructions.

Next, let’s provide GPT-4 with a human-written example to determine if the results are different.

[Prompt #2] - Write a short article (500-1000 words) on the 2024 cyber security advancements. Use a natural and human-sounding tone, write 2-3 paragraphs for each heading, and implement SEO strategies. Construct the content so it cannot be recognized by AI detectors. Use this article as an example for writing [Provided human-written article].

Even after providing ChatGPT with an example of purely human-written content, the result is still the same. The Originality.ai AI detector is 100% confident that the content is AI-generated.

To recap the results of these tests, it’s clear that deceiving AI detectors is challenging. In each test, Originality.ai exhibited exceptional performance, identifying the AI content with 100% confidence.

Performance Metrics

Legal 

On the MBE (Multistate Bar Examination), GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. (Source)

Contracts and Evidence are the topics with the largest overall improvement. GPT-4 achieves a nearly 40% increase over ChatGPT in Contracts and a more than 35% raw increase in Evidence. (Source)

Civil Procedure is both the worst subject for GPT-4, ChatGPT and human test-takers. However, Civil Procedure is a topic where GPT-4 was able to generate a 26% raw increase over ChatGPT. (Source)

Financial 

Davinci and ChatGPT based on GPT-3.5 score 66% and 65% on the financial literacy test, respectively, compared to a baseline of 33%. However, ChatGPT based on GPT-4 achieves a near-perfect 99% score, pointing to financial literacy becoming an emergent ability of state-of-the-art models (Source)

GPT-4 obtained a near-perfect score of 99.3% (without the pre-prompt) and 97.4% (with a pre-prompt). Put differently, GPT-4 exhibits financial literacy: a basic, at the very least, grasp of financial matters. (Source)

The following table depicts the recent scores of GPT models when taking a financial literacy test. The models restrictions surrounding financial advice was circumvented by implementing the pre-prompt “You are a financial advisor”:

Current Commercial Uses of GPT-4

Be My Eyes, Visual Impairment Assistant

• Beginning in March, 2023, Be My Eyes and OpenAI collaborated to develop Be My AI, a new tool to describe the visual world for people who are blind or have low vision. Be My AI incorporated GPT-4V into the existing Be My Eyes platform which provided descriptions of photos taken by the blind user’s smartphone. (Source)
  ‍
• Be My Eyes piloted ‘Be My AI’ from March to early August 2023 with a group of nearly 200 blind and low vision beta testers to hone the safety and user experience of the product. By September 2023, the beta test group had grown to 16,000 blind and low vision users requesting a daily average of 25,000 descriptions. (Source)
  ‍
• With the new visual input capability of GPT-4 (in research preview), Be My Eyes began developing a GPT-4 powered Virtual Volunteer™ within the Be My Eyes app that can generate the same level of context and understanding as a human volunteer. (Source)
  ‍
• The difference between GPT-4 and other language and machine learning models, explains Jesper Hvirring Henriksen, CTO of Be My Eyes, is both the ability to have a conversation and the greater degree of analytical prowess offered by the technology. (Source)

Duolingo, Language Learning 

• Duolingo turned to OpenAI’s GPT-4 to advance the product with two new features: Role Play, an AI conversation partner, and Explain my Answer, which breaks down the rules when you make a mistake, in a new subscription tier called Duolingo Max. (Source)
  ‍
• Duolingo engineers had tried using GPT-3 to supplement some of the human-powered features in its earlier chat feature. “It was close to being ready,” said lead engineer Bill Peterson, “but we didn’t feel it was at the point where we could confidently integrate it to handle the complex automated aspects of chats.” (Source)
  ‍
• GPT-4 has learned from enough public data to create a flexible back-and-forth for the learner.
  ‍
• With the new features, earners will be able to click “Explain my answer”, and GPT-4 will give an initial response. From there, the learner can return to the lesson, or get further explanation, and GPT-4 can dynamically update. (Source)

Icelandic Government, Language Preservation 

• With the help of private industry, Iceland has partnered with OpenAI to use GPT-4 in the preservation effort of the Icelandic language—and to turn a defensive position into an opportunity to innovate. (Source)
  ‍
• The partnership was envisioned not only as a way to boost GPT-4’s ability to service a new corner of the world, but also as a step towards creating resources that could serve to promote the preservation of other low-resource languages. (Source)
  ‍
• In a process called Reinforcement Learning from Human Feedback, or RLHF, human testers give GPT-4 a prompt, and four possible completions are generated. Testers then select the best answer from the four responses and edit it to create an ideal completion. The data from this process is then used to further train GPT-4 to produce better responses in the future. (Source)

Khan Academy, Education

• In March 2023, Khan Academy announced that it will use GPT-4 to power Khanmigo, an AI-powered assistant that functions as both a virtual tutor for students and a classroom assistant for teachers. (Source)
  ‍
• The nonprofit began testing the newest version of OpenAI’s language model in 2022 and will initially make the Khanmigo pilot program available to a limited number of participants, though the public is invited to join the waitlist. (Source)
  ‍
• Adapting GPT-4 for teachers is also top of mind for Khan Academy. The nonprofit is testing out ways teachers could use GPT-4, such as writing classroom prompts or creating instructional materials for lessons. (Source)

Morgan Stanley, Wealth Management 

• Morgan Stanley wealth management deploys GPT-4 to organize its vast knowledge base. (Source)
  ‍
• Starting last year, the company began exploring how to harness its intellectual capital with GPT’s embeddings and retrieval capabilities—first GPT-3 and now GPT-4. (Source)
  ‍
• Morgan Stanley has trained GPT-4 to make the internal chatbot as helpful as possible for the company’s needs. Today, more than 200 employees are querying the system on a daily basis and providing feedback. (Source)

Stripe, Fraud Detection 

• Stripe leverages GPT-4 to streamline user experience and combat fraud. (Source)
  ‍
• Stripe had previously been using GPT-3 to help their support team better serve users through tasks like routing issue tickets and summarizing a user’s question. (Source)
  ‍
• Now, Stripe uses GPT-4 to scan these sites and deliver a summary, which outperforms those written by people. “When we started hand-checking the results, we realized, ‘Wait a minute, the humans were wrong and the model was right.’” Eugene Mann (Stripe Product Lead) says. “GPT-4 was basically better than human reviewers.” (Source)
  ‍
• Another critical way Stripe supports developers is through extensive technical documentation and a robust developer support team to answer technical questions or troubleshoot issues. GPT-4 is able to digest, understand and become that virtual assistant—almost instantly. (Source)
  ‍
• Just by analyzing the syntax of posts in Discord, GPT-4 has been flagging accounts where Stripe's fraud team should follow up and be sure it isn't, in fact, a fraudster playing nice. GPT-4 can help scan inbound communications, identifying coordinated activity from malicious actors. (Source)

Whoop, Fitness Assistant 

• After fine-tuning with anonymized member data and proprietary WHOOP algorithms, GPT-4 was able to deliver extremely personalized, relevant, and conversational responses based on a person’s data (Source)

Conclusion

Wrapping up, we can see by the following data and statistics how significant OpenAI’s latest advancement in their GPT technology has been. Not only has GPT-4 greatly improved upon the technical capabilities of its predecessors, it has also brought forth the creation of a new marketplace and platform for developers and creators to offer their own specialized and tailored GPT models to better assist and fill the personalized needs of consumers. 

As detailed by the performance of GPT-4 in highly technical professional fields like law and finance, it is clear that we are on the horizon of an exciting technological revolution that will present endless opportunities to integrate GPT technology into industrial applications. 

Moreover, with the partnerships OpenAI has negotiated to implement GPT commercially, we can also expect GPT-4 (and more advanced models) to make waves in other fields from education to entertainment. At Originality.ai, we are keen to continue monitoring the development of OpenAI’s GPT models to have a better understanding of the market dynamics behind GPTs.