Should you choose an Open Source, Open Weight or Proprietary LLM? Let’s face it: it’s a rhetorical question, and you won’t find the answer in this text.
We’d like you to gather useful information and a first-level analysis on this complex subject to help form your own opinion.
LLM is not (just) software
There’s a tendency to think of an LLM as conventional software. But in reality, it’s much more: a language model, as the name suggests.
An LLM is a combination of many ingredients:
- A database containing a large number of texts (e.g. much of the Internet);
- Mechanisms for cleaning up, discarding, etc. some of these texts (“the Internet is a dustbin” we sometimes read);
- One or more deep neural networks, specialized in the analysis of the statistical distribution of words (or “tokens”, to be more precise);
- One or more training algorithms that calculate the values of the neural network’s several (hundred) billion parameters, thus transforming certain properties of the text base into numbers that can later be processed by computers;
- Vectorization functions, projecting (sequences of) words into a high-dimensional space, and enabling the calculation of proximities between “chunks”;
- Devices designed to limit potential vulnerabilities and misuse of the solution;
- Modules for efficient prompting, possibly specialized for certain tasks (code generation, mathematical calculations, etc.);
- Modules for language processing (NLP), dialog management, etc.
Eventually, orchestration and routing functions to drive the different parts of the LLM;
As with a recipe, varying just one of these ingredients will affect the quality of the product. Too much salt and it’s ruined; too little and it’s bland.
Let's start here: what's Open Source
“Open source” refers to software or intellectual work licensed per Open Source Initiative criteria, such as free redistribution and access to the source code.(Wikipedia).
According to Github, Open Source Software (OSS) is software with publicly accessible source code that users can consult, modify, adopt, and share freely.
10 Golden Rules of OSI (Open Source Initiative)
defines the ten principles of open source:
- Free redistribution: The license does not prevent any party from selling or transferring the software.
- Source code: The program must include the source code, and must allow distribution of the source code and its compiled form.
- Derivative works: The license must permit modifications and derivative works, and their distribution under the same conditions as the license of the original software.
- Integrity of the author’s source code: The license must explicitly authorize the distribution of software built from modified source code.
- No discrimination against individuals or groups.
- No discrimination in areas of expertise.
- Distribution of the license: The rights attached to the program must apply to all those to whom the program is redistributed, without the need for them to enter into an additional license.
- The license must not be product-specific.
- The license must not restrict other software.
- The license must be technology-neutral.
Types of license
Open source code is usually stored in a public repository that anyone can access and contribute to. Contributors upload new versions of the code to the project, building on and improving existing work to provide new features and updates.
The code is accompanied by its license, which defines what a user may or may not do. Some licenses are permissive, allowing use and distribution of the code for any purpose, while others require explicit sharing of all modifications. Still others stipulate that copies of the source code must be free and available for public use.
Among the most popular licenses are :
- The MIT license has few restrictions on what can be done, making it the most permissive and widely used free license. All the license requires is that future versions of the code include the original copyright notice and a copy of the license itself.
- The Apache 2.0 license is also a permissive software license, allowing users to do whatever they like with the code, provided they log any major changes they have made to it.
- The GPLv2 license requires that the source code be made available to the public. GPLv2 is also a copyleft license, which means that any version of the source code must also be released under the same license, GPLv2.
- GPLv3 eis compatible with the Apache 2.0 license and does not require the source code to be made available to the public.
Open Source LLM versus Open Weight
The traditional definition of Open Source code cannot be applied so simply to artificial intelligence technologies. This is because, unlike conventional Open Source code, which consists mainly of programming instructions, LLM relies on :
- (very) large quantities of training data, likely to include protected work or private data, which can raise legal issues if shared ;
- numerical parameters that determine how the input data is processed and how the model “understands” the language.
While Open Weight LLMs may disclose the model weights, they do not necessarily share the data sources used to create the LLM.
In contrast, an Open Source LLM (in theory) shares every step and every data source under a permissive license that allows the model to be used, modified and distributed.
Advantages and limitations of Open Weight LLMs
Advantages:
Flexible model customization: model weights can be adjusted to suit the LLM’s mission, without starting from scratch;
- By providing a ready-to-use base, these models facilitate implementation for those looking to take advantage of AI’s advanced capabilities.
Limitations
- Dependence on proprietary software. The effectiveness and adaptability of open weight models depend on the tools and platforms used by the model’s creators;
- Limited transparency and interpretability: without access to the underlying data and algorithms, you won’t be able to understand how decisions are made.
Aruna Kolluru, Chief Technologist, AI at Dell Technologies, proposes the following comparison:
See also What is an open source LLM? by Red Hat.
Case study#1 : Meta Llama 3
Meta CEO Mark Zuckerberg is a recent convert to open source. His letter ‘’Open Source AI Is the Path Forward’’ is a case in point.
In his view, Open Source AI is the right solution for developers:
- They need to be able to train, refine and distill their own models.
- They need to control their destiny and not be locked in by a closed supplier.
- They need to protect their data.
- They need an efficient and affordable model.
- They need to invest in the ecosystem that will be the long-term norm.
This is the significant position of a major player who positions its business model on “the next move”, unlike Open AI (which is no longer open…) for example.
It should be noted that Microsoft is in between the two, as the Phi range of compact LLMs is available as Open Source (MIT license).
In his opinion, Open Source AI is the right solution for developers:
- They need to be able to train, refine and distill their own models.
- They need to control their destiny and not be locked in by a closed supplier.
- They need to protect their data.
- They need an efficient and affordable model.
- They need to invest in the ecosystem that will be the long-term norm.
This is the significant position of a major player who positions its business model on “the next move”, unlike Open AI (which is no longer open…) for example.
It should be noted that Microsoft is in between the two, as the Phi range of compact LLMs is available as Open Source (MIT license).
Llama 3 License
Llama 3 license comes close to an Open Source license, with a few additional limitations.
What you can do:
- Use and modify: you have the freedom to use, adapt and develop the Llama 3 model, while retaining ownership of your new creations.
- Redistribute: you can distribute original or modified versions of your Llama 3 model, provided you include a copy of the license agreement and a note stating “Built with Meta Llama 3”.
- Intellectual property: innovations created from your Llama 3 model belong to you.
What you can’t do:
- If your services exceed 700 million monthly users, you will need to obtain an additional license from Meta.
- The license prohibits the use of your Llama 3 model to enhance competing Meta models.
- The license restricts the use of Meta trademarks, unless specifically authorized.
Ethical considerations
Meta seems to have taken the ethical and security aspects seriously, multiplying safeguards and checkpoints. Only time will tell how effective this will be, but it seems a laudable intention.
Case study #2 : Mistral
At the time of writing, most of Mistral’s general-purpose models benefit from an Apache v2 license, making them a good candidate for integration into software suites.
However, economic considerations have led Mistral to define the MNPL license (Mistral AI non-production license).
This allows free use of the software concerned in non-commercial situations (e.g. for research, testing, etc.): “You shall only use the Mistral Models and Derivatives (whether or not created by Mistral AI) for testing, research, personal, or evaluation purposes in Non-Production Environments”.
Among Mistral’s most recent products
- Mistral Nemo, developed in collaboration with NVIDIA, is available under the Apache 2.0 license.
- Mistral Large and Codestral (specialized in code generation) are affiliated to the MNPL license, which can be supplemented by a commercial license to be negotiated with Mistral for the transition to production.
Comparing performance: a perilous exercise
Just as you can’t compare Simone Biles and Teddy Riner at the Olympic Games, you can’t answer the question “Which is the best LLM?
There are many categories, and within each of them we need to refine equally according to the number of parameters (and therefore memory size required), contextual window length, tendency to hallucinate, reaction speed and so on.
Table of medals
Some brave souls have set out to establish a ranking of LLMs based on the Elo algorithm used for chess. The idea is to pit the candidates against each other and compile the results. A relative judgment, not an absolute index.
Without placing too much importance on the figures themselves, we can nevertheless deduce two basic trends:
- Continuous improvement in performance over time (as expected),
- A narrowing of the gap between the best proprietary and open solutions
Some other interesting sources:
- Hugging Face updates its ranking
- Lmsys does the same in a crowdsourcing approach.
Example of qualitative analysis
Lsmsys offers an interesting example of qualitative analysis, breaking down performance into eight criteria: writing, role-playing, reasoning, mathematics, coding, knowledge extraction, STEM (Science, Technology, Engineering and Mathematics) and humanities.
The following graph shows that the competition brings together llama-13b (13B parameters) and GPT4 (150B parameters). It’s not surprising, therefore, that the results are very different.
Note the correlations between criteria: the performance of the group [reasoning, mathematics and coding] seems similar, as does that of the triplet [writing, human sciences and STEM].
GDPR, AI Act and sovereignty
The legal aspects are more or less acute, depending on the nature of the exchanges and requests with the LLM.
For example, a healthcare application will require “HDS” hosting.
High-risk applications (see our article about the AI Act)
and those handling personal data will require precautions regarding hosting and a high level of transparency regarding the constitution of the LLM.
Applications in the field of defense or sensitive industries will require hosting on European (or even French) soil, and in some cases with hosting providers governed by European law (to avoid concerns about the extraterritoriality of the laws of certain countries). Or, in extreme cases, completely dedicated servers.
All these criteria need to be determined early on in the LLM selection process, as they have a limiting impact on the choice of LLM and its hosting mode.
Costs comparison
As with performance, there are no ready-made answers (sorry…), and each project will need to be studied in its own specific context.
The total cost of an LLM depends as much on its technical characteristics and deployment model as on its licensing model.
- For example, the use of GPT4 on Azure includes the cost of using the API, the Open AI model, hosting and so on.
- For those who opt for a hosted Open Source LLM, it’s essentially the hosting costs (server rental) that will determine the price.
- And finally, if you choose an Open Source LLM not hosted by the publisher or one of its partners, you’ll need to provide one (or more) servers with a level of memory and performance suited to the intended use and volume.
Some criteria to consider when assessing the TCO (total cost of ownership) of an LLM:
- Use cases, user values, expected UX quality ;
- Adaptation or direct use of the language model ;
- Hosted locally by a partner ;
- Integration and maintenance ;
- Volumetry, scaling, availability ;
- Availability of competent staff in your organization;
- Etc.
Some have ventured to propose a performance/cost hierarchy, in particular to promote a hybrid approach, with optimized routing according to the nature of the requests.
And tomorrow?
What is certain is that nothing is certain. Positions are not yet established, technologies have not yet stabilized, and uses have yet to be explored. So we have to accept that today’s choices will be called into question tomorrow.
There seems to be a tendency to offer (relatively) small LLMs, facilitating their dissemination in the digital world. Sufficiently specialized, they offer performance comparable to their big brothers, most of whose capabilities are under-utilized, and perhaps even better speed.
Architectures with several LLMs (e.g. Mistral 8×7) have appeared, and orchestrator functions and routing to specialized agents are also being studied.
Back to the future
Let’s use the history of operating systems (OS) to guess at the future of LLMs:
- 50 BAI (Before AI): the mainframe era: operating systems were big, closed and structuring: the reign of IBM ;
- 40 BAI: arrival of microcomputing and multiplication of offerings, with the miniaturization of hardware making it available to all businesses (DEC, Novell…);
- 30 BAI: Open Source battle around UNIX, finally won by Linux, with many collateral victims (SUN, HP…) ;
- 20 BAI: widespread use of personal computing, giving proprietary OSes (IOS, Windows) back the upper hand;
- 10 BAI: relatively limited, stable offering, mixing proprietary and open source solutions: IOS, Windows, Linux and Android.
It’s now up to you to write the history of LLMs and generative AI…
_____________
Are you a software publisher? We wanted to share with you this reflection on the environmental impact of digital technology.
Agora Software is the publisher of conversational AI solutions dedicated to software publishers. We rapidly deploy conversational application interfaces that enhance the user experience of applications and platforms. By integrating Agora Software into your application, your users benefit from rich, multilingual and omnichannel interactions.
Would you like to integrate conversational AI into your applications?
Let’s talk: contact@agora.software
If you liked this article, you might like to read Hallucinations, LLM and conversational AI.
And join us on our Linkedin page to keep up with all the latest news!
Want to understand how our conversational AI platform
optimizes your users’ productivity and engagement by effectively complementing your business applications?