Generative AI and Large Language Models#
Generative AI technologies have become some of the most powerful capabilities for information technology. Probably, the most common application now is text generation with the ability to produce human-like text based on a given prompt and some context. People use this for writing assistance and content generation. Generative AI can also create images, videos, and music. Generative AI is also remarkably adept at programming tasks, especially when the task is well-defined and broken down into appropriate sizes. As such, it can be tempting to copy the assignment text into a tool such as ChatGPT and then submit the response to the autograder, hoping that it will pass the test cases without any further work. We refer to this homework approach as “submit and pray”. However, does someone learn anything? Programming, as with most other activities, takes practice to become proficient. We’ve designed these notebooks and course assignments to help guide students and provide the appropriate practice. We don’t expect these notebooks to be the only source of information that you will use. Many other high-quality books, websites(Stack Overflow, Geeks for Geeks, LeetCode, Real Python, …), videos, and other resources will be helpful. We also expect you to work with your fellow students, teaching assistants, and instructors. Finally, we do not expect you to avoid Generative AI tools, but rather to use these tools to help assist you in your journey to learn how to program.
Ultimately, learning takes place when we apply knowledge in the context of a specific problem. We need to understand why something works as well as to understand when something does not work. Use Generative AI to increase your knowledge, not as a shortcut to completing homework.
At the end of each notebook, we include several prompts that may be helpful for you to use as samples and a starting point for you to develop your own prompts. Within the Software Engineering course, we will discuss many ways to utilize these tools throughout the software development process.
General Pitfalls of Large Language Models#
While large language models (LLMs) can generate content that appears correct with a seemingly confident tone, you do need to analyze the responses to verify that they are in fact, correct. You may also wish to utilize a service such as Perplexity that provides sources from which the responses were derived.
As LLMs generate content based on their training data, existing biases within those datasets may appear in the results. Search “bias in word embeddings” as a starting point to see how such biases can arise.
Additionally, LLMs often struggle with common problems such as mathematics and logic. Here, you need to make sure you are using the appropriate tool such as Wolfram Alpha for computational logic.
General Advice for LLMs#
Prompt engineering is the process of crafting effective prompts or instructions for large language models to produce desired outputs. It involves careful framing of queries, providing relevant context, and iteratively refining prompts based on the model’s responses. An excellent prompt engineering guide is available at https://www.promptingguide.ai/.
In terms of immediate advice, providing relevant context is essential to using LLMs correctly. Context can -
improve accuracy: Providing the appropriate context to an LLM can significantly improve the accuracy of its outputs. On the flip side, if an LLM is given an entire document when only a specific section is relevant to the user’s query, it can become confused and provide less accurate responses
prevent hallucination: Providing the right context can help mitigate this issue, as the LLM will have a better understanding of the task at hand and the information it should be drawing from.
provide in-context learning: LLMs have the remarkable ability to learn new concepts and skills within the context of a given prompt, without requiring changes to the model’s underlying weights. This “in-context learning” allows LLMs to adapt to specific use cases and tasks, but it relies heavily on the quality and relevance of the provided context. For one such technique, explore retrieval augmented generation.
Specify precise instructions for what you ask the LLM to produce. This includes an overall instruction such as write, summarize, extract, etc. These instructions should also include specific numbers if you are asking for a list of items. You can specific tags or specific formats (e.g., JSON) for the output. You can also ask the LLM to justify its answer. Tell the LLM what to do, not what not to do - this can lead to confusion.
Sample LLM Prompts to Help You Learn to Code#
The following are some possibilities on how you can start to use an LLM to learn how to program:
Provide 5 pieces of advice to give to a novice on how to learn to program
I’m trying to write a program to read stock data from a CSV file and compute the daily average returns, but I’m not sure how to get started. Please give me some hints and advice without telling me the solution.
Explain the following code block: add code
You can also build upon the explanation approach to ask the LLM to use an analogy for a given circumstance: Explain the following code block to me using an analogy as if I were a high school student: add code
Provide five use cases in the financial domain for using Python lists.
Why does the following code block not work when trying to do x? add code
While we have provided answers to the review questions, you can also put these questions into an LLM.