With all the hype around AI, particularly large language models (LLMs), you may be one of those people who had their mind blown by what questions ChatGPT (or something similar) was able to answer. And being the forward thinker you are, you thought “Surely this thing could answer some Acumatica questions.” You gave it a shot, and somehow you ended up here, reading about why (according to me) the shortcut you discovered might not be as cool as you thought. 

There are a couple fundamental problems with using LLMs in this context: 

1. They are “smart” or rather, they sound smart. To someone who isn’t familiar with a topic, they sound like they really know what they’re talking about – like some people I know. Even if they really have no idea, they’ll take their best guess, but rather than tell you, “My best guess is ___,” they’ll just pretend they are completely sure. I don’t know about you, but if I’m getting a “My best guess is ___” answer, I’d like to be told that. 

2. They are trained on publicly available data. LLMs get their knowledge from publicly available sources. If the topic in question has no publicly documented answers, you won’t get an accurate answer, and even if there are publicly documented answers, that doesn’t mean the LLM was trained on them. Still, you’ll get a very certain answer (problem One). 

3. LLM answers tend to be both generic and thorough. Since answering a technical question requires a clear understanding of the problem (which requires very specific prompting), the LLM will usually give a very generic (and often long) answer. This isn’t necessarily bad, but this is a situation where an experienced human has the advantage of understanding at least the basics of the situation without needing to spell out every detail. 

The above content was written by me

The below content was generated by Gemini (Google’s LLM)

Prompt:

I answer forum questions for an ERP called Acumatica. I am an expert with years of experience using the software. I am frustrated with contributors attempting to answer forum questions by pasting the questions into an LLM. The answers are very generic and sound technical even though they often end up being rather unhelpful. Please explain what advantages an expert has over an LLM. Please also include how an LLM could be useful to an expert. Then give examples of things an LLM would excel at. 

Response:

Advantages of a Human Expert Over an LLM for Acumatica Forum Questions 

1. Deep-seated Knowledge and Experience: 

• Contextual Understanding: Experts have a deep understanding of Acumatica's nuances, including its history, industry-specific applications, and potential pitfalls. This allows them to provide tailored solutions based on specific scenarios. 

• Real-world Application: Years of hands-on experience enable experts to offer practical advice, considering factors like system configurations, data integrity, and best practices. 

2. Problem-solving and Critical Thinking: 

• Customization: Experts can adapt their answers to the unique needs of the questioner, considering factors like their role, company size, and industry. 

• Troubleshooting: They can diagnose complex issues by asking targeted questions to gather more information and provide effective solutions. 

3. Human Connection and Empathy: 

• Relatability: Experts can empathize with the user's frustrations and provide support in a compassionate manner. 

• Clarity: They can explain complex concepts in a way that is easy to understand, avoiding technical jargon. 

How an LLM Can Assist an Expert 

• Research and Information Gathering: LLMs can efficiently search through vast amounts of data to find relevant information, saving experts time and effort. 

• Code Generation and Scripting: For technical questions, LLMs can generate code snippets or scripts based on specific requirements. 

• Idea Generation: LLMs can provide alternative perspectives or suggest potential solutions that the expert might not have considered. 

Areas Where an LLM Excels 

• Data Analysis and Pattern Recognition: LLMs can analyze large datasets to identify trends, anomalies, or correlations. 

• Natural Language Processing: They can understand and respond to human language, making them useful for tasks like customer service chatbots or language translation. 

• Content Creation: LLMs can generate creative content, such as articles, poems, or scripts. 

In conclusion, while LLMs can be valuable tools for information retrieval and basic task automation, they cannot replace the expertise, problem-solving skills, and human connection that a seasoned Acumatica professional brings to the table. By combining the strengths of both humans and AI, we can provide more comprehensive and effective support to users. 

As you can see, Gemini quickly answered my prompt with a nicely formatted, multi-faceted answer. Granted, my prompt was very thorough, but I get why people think LLMs are all that. They can be a useful tool to an expert, but they are not a replacement for the expert. It's genuine human responses from experts that maintain the community and integrity of the forums.