You can stumble upon AI just about anywhere in the online space today. It is discussed from the perspective of its use, in memes, and even its risks. However, I have seldom found an explanation of how it works in a form that is understandable even to a layperson. In this article, we'll look at how it is possible that ChatGPT or Bard can generate perfect text outputs including programming code, while others like DALL-E or Midjourney create amazing images, all while operating on the same basic principles.
What is GPT (not ChatGPT)?
GPT is an abbreviation for Generative Pretrained Transformer. Let's break down each word in this acronym.
Are you interested in the IT field and looking for job positions and opportunities in the IT industry? Whether you are a programmer, developer, tester, analyst, or software architect, contact us and we will find an IT project tailored to your needs from our IT job offers. Take a look at the current available job positions in the IT field. We will help you find new job challenges and opportunities. We look forward to collaborating with you!
Generative
Generative means that AI can generate unique output that follows the input text (i.e., the prompt). This output does not have to be created just by copying and pasting learned parts of texts, but is created based on the analysis of individual dependencies between words, which AI forms during its learning.
For example, if the input sentence is: "On the table is", the AI from the learned data knows that the word "book" most often appeared in this context, to which it assigns a probability of, for example, 90%, then it inserts other words ("glass", "plate", etc.), to which it also assigns a certain probability. The subsequent output always depends on which word has the highest probability. So if you generate the output for the command "On the table is" 100 times, in 90 cases the AI will write "cat" in the remaining 10 cases some other word. This is why AI sometimes has a different result for the same task, because chance plays a role here.
Note:
If you are trying out ChatGPT more in detail, you may come across the Temperature" parameter. This parameter determines how much AI will respect the order of probability of words. If you set the temperature to zero, it will always select only the most likely word, and therefore the result will be the same every time. In our example, the AI will always write "On the table is a book". If, on the other hand, you set the temperature to a value of 1, you may get quite diverse results, as AI will choose less likely words. So for example "On the table is a cat" or "On the table is a miniature model of an Antarctic station".
Temperature is therefore a great way to keep AI on a leash (writing technical texts), or on the contrary, to allow full creative freedom (poems, jokes, etc.).
Pretrained
Pretrained means that this type of AI must be trained on data before use. The more complex and complicated the task is, the better and higher quality data we must provide to AI for learning.
What does AI training look like?
Let's say we want to teach AI to recognize whether a given text is written in Czech or English. So we have to teach AI what such Czech or English looks like. We will have lists of words or entire paragraphs and for each we provide information whether it is written in Czech or English. After learning, the correctness of the teaching is verified on completely new data, where we check whether AI correctly recognized the language. If AI does not work according to our expectations, it is unfortunately not possible to look into the source code or follow step by step how AI reached the result. The calculations here are extremely complex and contain up to billions of steps. The only way to improve AI results is to provide more data, increase their quality, or change learning principles.
In the case of ChatGPT, the task was to understand and recognize all languages and word meanings. Therefore, it was necessary to provide a really large amount of texts. The training of the ChatGPT-3 model took place on more than 1,000 graphics processors for 34 consecutive days. The volume of data was also enormous, around 500 GB.
Transformer
Transformer is the name of a neural network architecture introduced by Google in 2017. You might be asking now, what is a neural network? It's an electronic imitation of the functions of our brain. So similarly, it consists of "electronic" neurons that are interconnected. And according to how we interconnect and layer these neurons, we achieve a certain structure, which we call architecture. Transformer is one of the most common types of neural network architecture, which has proven to work well with text, as it can understand relationships and dependencies between words, regardless of their distance from each other in the text.
Very simplistically - at the input of the neural network, there are numbers that arise from converting input words (or images in the form of pixels). Then a process occurs, wonderfully described by Lubomír Lipský in the fairy tale The Emperor's Baker: “Patláma, patláma paprťála, žbrluch!”
These numbers simply "bubble" through the neurons, and other numbers appear at the output. Then these numbers are decoded into an understandable form and compared with expected results. And if they are not correct, a complicated process of adjusting the parameters of the neural network occurs, so that for the next input, the result is closer to the expected output. This process is repeated until the result is satisfactory.
In short, the neural network and its learning is a very complex process. As a result, even as a programmer, you have no chance of adjusting anything within the neural network, as it is impossible to understand the individual parameters inside the neural network and their dependencies. In ChatGPT-3, there are over 175 billion parameters. For the first time in history, man has created a machine / program that he does not fully understand, and which externally appears as a so-called black box.
Nowadays, fields called "Interpretability of AI" are emerging, which means, "why on earth does this AI behave the way it behaves."
In short: If you are communicating with ChatGPT or creating images in MidJourney or DALL-E, know that there is not an advanced intelligent form of life behind it, but "just" encoding into a numeric series, bubbling through a learned neural network, which gives the most likely result.
GPT and Security
There is also a lot of talk about AI security these days. However, from what I wrote above, it is clear that GPT models are termed as artificial intelligence, but in fact, they are a very complicated and huge statistical model. They are basically a colossal mathematical function of billions of parameters. It is therefore unlikely that consciousness could arise in these models.
It is also unlikely that someone at home in their garage could create their own harmful AI that would destroy the world. These huge models like GPT-3 require millions of dollars for training, not to mention the need to have GB or even TB of data for learning.
However, it should be noted that the field of AI is moving forward at a tremendous pace and new models, new learning approaches may emerge, which means that the possibility of creating so-called general artificial intelligence (AGI) cannot be completely ruled out. It is therefore appropriate to be cautious, especially when building new generations of AI. Still, I am convinced that the benefits today far outweigh the disadvantages. AI is becoming a universal educational tool, a partner in creating any online content, and also a scientific tool for solving problems that a human-created program would not be able to solve.
Therefore, I perceive our future positively. AI is becoming a universal educational tool, a partner in creating any online content or also a scientific tool for solving problems that a program created by a human would not be able to solve. I perceive our future positively.
Author of the article: Martin Smětala
🟡 Are you looking for an interesting project? Check out how we do things here and see which colleagues we're currently looking for.
🟡 Do you have a colleague or friend who is looking for a new project? Join our Referral program and get a financial reward for your recommendation.
🟡 Would you like to start working in IT? Download our ebook START WORKING IN IT: From First Steps to Dream Job, in which we guide you step by step with information, courses, and practical experience that are so essential not only for those who want to switch fields, but also for those who want to advance their careers and further their education.
Or share this article, which may also be useful to your acquaintances.
