Robert Polding's Home

Large Language Models (LLMs) have emerged as significant tools in artificial intelligence, and they are having a huge impact on education. This report examines the historical development of LLMs, tracing their evolution from their early origins to their current advanced state. Subsequently, it analyzes the implications of LLMs in educational contexts, including data security concerns and best practices for integrating LLMs into educational assessments. While incorporating LLMs into education offers excellent opportunities, it also raises ethical considerations. Thus, the responsible utilization of these models while upholding the integrity of educational processes, stands as the primary imperative for educational institutions.

A History of Large Language Models

The development of large language models has its roots in the study of the nervous system. Santiago Ramón y Cajal, a pioneering neuroscientist, laid the foundation by positing that the nervous system consists of discrete individual cells. This concept has evolved over time, leading to the creation of artificial neural networks (ANNs) and deep learning algorithms that mimic the structure and function of the human brain.

In 1958, psychologist Frank Rosenblatt invented the perceptron, a rudimentary form of artificial neural network. This invention marked the beginning of the development of “artificial brains.” ANNs consist of neurons connected in layers, each performing specific mathematical functions. The first convolutional neural network was introduced in 1988, paving the way for more complex architectures.

Deep learning emerged as a specialized subfield of machine learning, focusing on algorithms inspired by the brain’s structure and function. These algorithms automatically learn to represent data through multiple layers of interconnected nodes, or “neurons.” Deep learning has proven to be particularly effective in handling large and complex data sets, leading to breakthroughs in image recognition, natural language processing, and autonomous vehicles.

One of the most notable applications of deep learning was developed by DeepMind, a London-based startup now part of Google. They created models that could learn to play video games in a manner similar to humans.

The Transformer

The modern transformer model was proposed in the 2017 paper titled ‘Attention Is All You Need’ by Ashish Vaswani and colleagues from the Google Brain team. This architecture revolutionized the field by introducing a simplified yet powerful structure that excels in various applications.

Language models like ChatGPT rely on input and output embeddings to interpret and generate text. These embeddings act as a mathematical dictionary, positioning semantically similar words close to each other in a mathematical space. Traditional machine learning models struggled with understanding word order, a critical aspect of sentence meaning. The introduction of positional encoding, combined with input embeddings, allows transformer models to understand sentence structure better and generate meaningful output.

ChatGPT models are trained on vast text data, including sources like Common Crawl, BooksCorpus, and Wikipedia. ChatGPT benefits immensely from this extensive training data. Efforts are also underway to improve the model’s capabilities in non-English languages.
The journey of large language models from their origins in neuroscience to their current state-of-the-art architectures is a testament to the rapid advancements in the field. These models have revolutionized natural language processing and shown promise in various other domains, making them a cornerstone of modern artificial intelligence.

Current State

Large language models (LLMs) have evolved to serve as communication layers within AI systems. They can interpret and explain results from more specialized and accurate AI models, such as search engines. However, it’s important to note that LLMs alone cannot be relied upon to produce accurate information.

The rise of LLMs has significantly impacted traditional information sources, including search engines like Google and Bing [https://www.wired.com/story/fast-forward-chatbot-hallucinations-are-poisoning-web-search]. One of the challenges arising from this development is the generation of AI-created fake responses, which calls into question the trustworthiness of information.

Post-Training

To address these issues, fine-tuning LLMs has become imperative. Fine-tuning allows for higher-quality results and the ability to train on a broader set of examples. The process involves preparing and uploading training data, training a new fine-tuned model, and then deploying that model for specific tasks.

Furthermore, the use of embeddings in LLMs has expanded their capabilities. Embeddings enable a range of functionalities, including search, clustering, recommendations, anomaly detection, and classification. These advancements contribute to making LLMs more versatile and effective, but they also underscore the need for establishing trust and reliability in AI-generated information.

Different Models

OpenAI is not the sole entity in the large language models (LLMs) market. Specialized models like Apple Intelligence are already integrated into iPhones for text prediction, and run locally without the need for cloud computing, and which is truly personalized thanks to the large amount of personal data on iPhones. Many users are interacting with these technologies without even realizing it.

In addition to OpenAI, Google has its own LLM called Gemini, which is integrated with the world’s most reliable search engine and supports coding tasks. Google has been in the AI development space since the launch of its search engine in 1997.

Meta, formerly known as Facebook, has also entered the market with a freely downloadable LLM named Llama. Although it is not open-source as claimed, it allows users to easily download and create a local LLM without requiring an API. This has led to the development of specialized variants, including coding-focused LLMs.

Finally, Anthropic have entered the market with the well renowned Claude. This was a project created from OpenAI exiles and shows incredible promise to be the most capable model.

Education

The educational sector is undergoing significant changes with the introduction of Large Language Models (LLMs). These tools offer immense potential for aiding technical researchers and professionals. However, there’s a downside: they could be used for cheating in assessments, potentially causing students to miss out on acquiring fundamental skills.

In technology-related classes, the aim is not to ban LLMs, as they are considered vital for the future of work. Banning them would create an educational environment that doesn’t reflect real-world work settings. Nevertheless, there will be instances, such as closed-book exams, where the use of LLMs will be restricted to test fundamental knowledge.

It’s important to note that these are suggested best practices, and the application may vary depending on the class content or departmental policies.

LLMs in Education

For Students

Large Language Models (LLMs) benefit programming students, offering functionalities like code generation, testing, and documentation. However, they have limitations such as producing partly random, sometimes plagiarized, insecure, or outdated code. Therefore, human intervention and expertise in programming are still required.

Beyond coding, LLMs have a wide range of applications for learners. They can assist in brainstorming by generating lists of related words and concepts, help in project planning by outlining objectives and requirements, and contribute to character development and dialogue writing. They can also be instrumental in research, acting as advanced search engines by generating lists of relevant articles, books, and websites (when integrated with a search engine). Furthermore, they have potential in educational settings, offering practical exercises and challenges for revision and possibly tutoring in the future.

LLMs are still under development but have the potential to revolutionize various professional fields by automating repetitive tasks, improving code quality, and facilitating communication. They are a valuable tool for enhancing productivity and are essential for a student’s future career.

For Teachers

Large Language Models (LLMs) are becoming increasingly useful in the educational sector, particularly for preparing classes. They can assist professors in various ways, such as querying documents in PDF form through specific websites and plugins like http://www.chatpdf.com. LLMs can also generate challenges and practice exercises for students, which can be beneficial for exams and homework, although verifying that these exercises are sensible and relevant is crucial.

Regarding lesson planning, LLMs can help generate comprehensive lesson plans and class activities. However, it’s essential to refine the prompts used with LLMs to ensure they align with the real-world context, as initial attempts often require improvement.

Risks

Data security is a critical concern, especially when dealing with institutional and student data. One solution to ensure data privacy is using a service such as Bing Enterprise (included as part of Office365), which guarantees that the data will not be used for training the LLM. This should be applied even when mentioning any seemingly insignificant data about a university, as it can accumulate and pose a risk.

The ultimate solution for data security lies in local Large Language Models. These local models will enable the use of sensitive and legally restricted data, such as health and legal information while ensuring full compliance with GDPR regulations.

Assessments

In educational assessments, the use of Large Language Models (LLMs) should vary depending on the skills being tested. For exams focusing on fundamental skills, such as closed-book exams, LLMs should not be used as aids. On the other hand, in assessments like group projects that aim to test the ability to achieve larger results using all available tools, including AI, LLMs should be considered.
Specific university exams where LLMs are not allowed should include closed-book exams, where no external materials are permitted, and invigilated exams, which are monitored to ensure that no unauthorized materials are used.

In cases where LLMs are used, it’s important to reference them in the work to maintain academic integrity.

Accidental Cheating

Students may unknowingly use Large Language Models (LLMs) through tools like Grammarly and Quillbot. While using these tools for correcting writing is acceptable, using them to generate text is considered the same as using any other LLM and could be categorized as academic dishonesty.

Therefore, we must educate students to ensure they know and use these tools correctly. It is essential that they can still take advantage of the writing improvement without accidentally using the generative AI built into these applications.

Consequences

Misusing Large Language Models (LLMs) in academic settings can lead to serious consequences. These include academic dishonesty, where using LLMs to generate essays or research papers can result in disciplinary actions such as a failing grade, suspension, or expulsion. Another consequence is plagiarism, which can severely damage a student’s academic reputation and lead to disciplinary action.

The severity of the consequences can vary, often escalating with repeated offenses. Initial instances may result in warnings, but repeated misuse can lead to more serious actions, including reporting the student to the university’s ethics committee, which could ultimately result in expulsion from an institution.

Conclusions

The journey of Large Language Models from their roots in neuroscience to sophisticated AI architectures is remarkable. They serve as vital communication layers in various AI systems but pose challenges, such as generating untrustworthy information. Fine-tuning and embeddings have expanded their capabilities, but there is a need for trust and reliability. Multiple companies offer specialized LLMs, making the market diverse. In education, LLMs can be both a boon and a bane, aiding in teaching and research but also posing risks like cheating and plagiarism. Data security, especially in educational settings, remains a critical concern. The use of LLMs in assessments must be carefully managed to maintain academic integrity. As LLMs become more integrated into daily life, responsible use and governance are increasingly important.