A team wants a private assistant and assumes it must train from nothing. How to create a large language model depends on the goal: pretrain a transformer on general text and fine-tune it, or fine-tune an existing model.
The wider category is covered in our overview of AI and automation tools for software teams, which helps place a custom model among other automation options.
Build, pretrain or fine-tune: where the confusion starts
Most confusion comes from treating “build” as a single act. Stanford University IT describes large language models as systems that often contain billions or even trillions of parameters. They are trained on text from books, websites and articles. The build path runs from pretraining on a general corpus to fine-tuning for specific tasks, which is the sequence Manning’s build-from-scratch book follows. Fine-tuning is the cheaper end of that path, because a pretrained model can be adapted with relatively smaller task-specific datasets.
| Common belief | What the evidence supports | Planning use |
|---|---|---|
| Building a model means training a giant one. | A learning-scale build is realistic. The book plans and codes an LLM comparable to GPT-2, and the model can be developed on an ordinary laptop, although a GPU is highly recommended for training. | Use a from-scratch build to learn the mechanics. Models with billions or trillions of parameters are a different scale of project. |
| Fine-tuning needs as much data as pretraining. | Pretrained models can be fine-tuned for specific applications with relatively smaller datasets. | If you need one defined task, start from a pretrained model and gather task data. |
| Training cost is easy to estimate loosely. | Training requires substantial computational resources and time, which can limit access to well-funded organizations. | Treat compute and time as the main cost drivers and check provider pricing before committing. |
The conditions above come from Stanford University IT’s explainer on LLMs and Manning’s book page.
The practical reading is simple. A from-scratch build teaches you how the pieces fit together, while a fine-tune delivers a task-specific result sooner. Decide which outcome you need before you provision anything, because the two paths demand very different resources. The next two sections cover what the build itself involves.
Design and data choices
Architecture: the transformer is the default
A frequent misreading is that a custom project needs a novel architecture. The evidence points the other way. Stanford University IT notes that the transformer architecture is particularly popular for LLMs because it can process large amounts of data efficiently, and its self-attention mechanism handles long-range dependencies in text. For a first project, choose a standard transformer rather than inventing a design.
Implementation is well trodden. Sebastian Raschka’s LLMs-from-scratch material walks through text embedding, attention, GPT-style model components, pretraining and finetuning in Python and PyTorch. PyTorch is therefore a sensible framework if you want to follow a documented path instead of assembling one from scattered tutorials.
Data and tokenization: quality is a safety decision
More text is not automatically better. LLMs can inherit and amplify biases present in the data they are trained on, so choosing sources shapes behaviour as well as capability. Review what goes in before scaling up the volume, and keep a record of where each source came from.
Preprocessing then converts text into units the model can consume. Tokenization splits the training text into smaller units, such as words or subwords. Treat the tokenizer as part of the model: whichever scheme you choose should be applied the same way in training and in use.
Training, compute and cost
Hardware: what a GPU changes
Hardware is where the biggest misconception sits: that a laptop can train anything you want. The Manning book says a model can be developed on an ordinary laptop, but it also states that using a GPU is highly recommended for training. GPUs significantly speed up training because they handle parallel computation efficiently. A laptop suits small experiments and learning builds. A GPU belongs in the plan as soon as you run real training jobs.
The evidence supplies no single dollar figure for training. It does describe the shape of the cost: substantial computational resources and time, which can restrict this work to well-funded organizations. To budget, take the GPU rates on your cloud provider’s current pricing page and set them against the compute time your run needs, then keep headroom for failed or repeated runs.
A staged pipeline to control cost
Researchers have argued that single-stage pipelines are the expensive way to deploy. A paper on cost-efficient LLM deployment reports that relying on an API or a single fine-tuning step suffers from substantial costs and high inference latency. Its alternative is a three-stage pipeline of prototyping, knowledge transfer and model compression.
The size reductions are the useful part for developers:
- Knowledge transfer: the authors distil a large model’s behaviour into a 0.5B-parameter student model, aiming for effective performance at minimal cost.
- Compression: quantization and pruning then shrink the models further to 0.4B parameters.
- Reported result: the pipeline realizes 180 times compression with nearly consistent performance compared with the original LLMs.
Read these as reported results for that study’s tasks, not as a guarantee for yours. They do show why prototyping with a large model and then distilling is a credible way to lower serving cost, so test a compressed model on your own validation set before you commit to it.
Choosing a path: build, fine-tune or call an API
The right choice depends on what you need to learn or control. These distinctions help:
- Build from scratch if understanding is the goal. Raschka’s position is that if you want to understand LLMs beyond API calls, the best way is to build one from the ground up. This suits developers and teams who want to know how attention, pretraining and finetuning work.
- Fine-tune if the task is narrow. A pretrained model needs relatively smaller task-specific datasets, so this path suits teams with a defined use case, their own data and a limited compute budget.
- Use a hosted API such as the OpenAI API to prototype. It lets you test whether a task is feasible without training anything. The deployment paper cited above warns that API-only or single-stage fine-tune pipelines can carry substantial costs and high inference latency, so move to distillation if usage grows.
- Wait on a full pretraining run. If you lack the compute and time that training demands, a from-scratch model at scale is not the right starting point.
If your real need is automating rule-based work rather than generating language, a custom model may be the wrong tool. Our guides to RPA In Healthcare: How To Automate Workflows, Cut Costs, And Improve Patient Care In 2026 – Interestingdevelopment and How Insurance Teams Are Saving Time With RPA: 7 Practical Use Cases And Where To Start In 2026 – Interestingdevelopment show where robotic process automation fits in two regulated industries.
Deploying safely
A trained model is not a finished product. Unsafe output and hidden bias are the two risks that the evidence calls out most clearly, and both need controls before release.
- Filter and monitor outputs. Stanford University IT warns that LLMs might generate inappropriate or unsafe content, especially if not properly filtered or monitored. Put filtering in front of users and keep monitoring running after launch.
- Test for bias. Because models can inherit and amplify biases in their data, check results across the groups and topics your users represent, not only on average scores.
- Evaluate before use. Raschka’s material presents benchmarks, verifiers, leaderboards and LLM judges as practical tools for checking model behavior. Use more than one, since each measures something different.
- Compress deliberately. Pruning, quantization or distillation can make a model more efficient to serve. Re-run your evaluation after each step so that efficiency gains do not hide a quality loss.
- Cover the wider challenges. The Manning book lists biases, safety and ethical use, transparency and explainability, and privacy among the key challenges. Assign an owner to each before launch.
Treat these as release conditions, not follow-up tasks. A model that cannot pass your safety and evaluation checks is not ready, however well it scores on general benchmarks.
Frequently asked questions
| Question | Answer |
|---|---|
| Who introduced the transformer architecture? | It was introduced in the paper “Attention Is All You Need” by Vaswani et al. (2017), as summarised in a Pluralsight guide to building an LLM. |
| Do I need an existing LLM library to build a model? | No. The Manning book builds its model without relying on any existing LLM libraries, and Raschka’s walkthrough uses Python and PyTorch. |
| What can fine-tuning be used for? | The Manning book covers fine-tuning LLMs for text classification and with your own data, after pretraining on a general corpus. |
| Does a large language model need a GPU to run at all? | Development can happen on an ordinary laptop. A GPU is the recommended option once you move to training, because it speeds up parallel computation. |
