codellama README

The project's README from github.com/meta-llama/codellama. As one markdown file: README.md.

Introducing Code Llama

Code Llama is a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama was developed by fine-tuning Llama 2 using a higher sampling of code. As with Llama 2, we applied considerable safety mitigations to the fine-tuned versions of the model. For detailed information on model training, architecture and parameters, evaluations, responsible AI and safety refer to our research paper. Output generated by code generation features of the Llama Materials, including Code Llama, may be subject to third party licenses, including, without limitation, open source licenses.

We are unlocking the power of large language models and our latest version of Code Llama is now accessible to individuals, creators, researchers and businesses of all sizes so that they can experiment, innovate and scale their ideas responsibly. This release includes model weights and starting code for pretrained and fine-tuned Llama language models — ranging from 7B to 34B parameters.

This repository is intended as a minimal example to load Code Llama models and run inference.

Download

In order to download the model weights and tokenizers, please visit the Meta website and accept our License.

Once your request is approved, you will receive a signed URL over email. Then run the download.sh script, passing the URL provided when prompted to start the download. Make sure that you copy the URL text itself, do not use the 'Copy link address' option when you right click the URL. If the copied URL text starts with: https://download.llamameta.net, you copied it correctly. If the copied URL text starts with: https://l.facebook.com, you copied it the wrong way.

Pre-requisites: make sure you have wget and md5sum installed. Then to run the script: bash download.sh.

Keep in mind that the links expire after 24 hours and a certain amount of downloads. If you start seeing errors such as 403: Forbidden, you can always re-request a link.

Model sizes

Model Size
7B ~12.55GB
13B 24GB
34B 63GB
70B 131GB

Setup

In a conda environment with PyTorch / CUDA available, clone the repo and run in the top-level directory:

pip install -e .

Inference

Different models require different model-parallel (MP) values:

Model MP
7B 1
13B 2
34B 4
70B 8

All models, except the 70B python and instruct versions, support sequence lengths up to 100,000 tokens, but we pre-allocate the cache according to max_seq_len and max_batch_size values. So set those according to your hardware and use-case.

Pretrained Code Models

The Code Llama and Code Llama - Python models are not fine-tuned to follow instructions. They should be prompted so that the expected answer is the natural continuation of the prompt.

See example_completion.py for some examples. To illustrate, see command below to run it with the CodeLlama-7b model (nproc_per_node needs to be set to the MP value):

torchrun --nproc_per_node 1 example_completion.py \
    --ckpt_dir CodeLlama-7b/ \
    --tokenizer_path CodeLlama-7b/tokenizer.model \
    --max_seq_len 128 --max_batch_size 4

Pretrained code models are: the Code Llama models CodeLlama-7b, CodeLlama-13b, CodeLlama-34b, CodeLlama-70b and the Code Llama - Python models CodeLlama-7b-Python, CodeLlama-13b-Python, CodeLlama-34b-Python, CodeLlama-70b-Python.

Code Infilling

Code Llama and Code Llama - Instruct 7B and 13B models are capable of filling in code given the surrounding context.

See example_infilling.py for some examples. The CodeLlama-7b model can be run for infilling with the command below (nproc_per_node needs to be set to the MP value):

torchrun --nproc_per_node 1 example_infilling.py \
    --ckpt_dir CodeLlama-7b/ \
    --tokenizer_path CodeLlama-7b/tokenizer.model \
    --max_seq_len 192 --max_batch_size 4

Pretrained infilling models are: the Code Llama models CodeLlama-7b and CodeLlama-13b and the Code Llama - Instruct models CodeLlama-7b-Instruct, CodeLlama-13b-Instruct.

Fine-tuned Instruction Models

Code Llama - Instruct models are fine-tuned to follow instructions. To get the expected features and performance for the 7B, 13B and 34B variants, a specific formatting defined in chat_completion() needs to be followed, including the INST and <<SYS>> tags, BOS and EOS tokens, and the whitespaces and linebreaks in between (we recommend calling strip() on inputs to avoid double-spaces). CodeLlama-70b-Instruct requires a separate turn-based prompt format defined in dialog_prompt_tokens(). You can use chat_completion() directly to generate answers with all instruct models; it will automatically perform the required formatting.

You can also deploy additional classifiers for filtering out inputs and outputs that are deemed unsafe. See the llama-recipes repo for an example of how to add a safety checker to the inputs and outputs of your inference code.

Examples using CodeLlama-7b-Instruct:

torchrun --nproc_per_node 1 example_instructions.py \
    --ckpt_dir CodeLlama-7b-Instruct/ \
    --tokenizer_path CodeLlama-7b-Instruct/tokenizer.model \
    --max_seq_len 512 --max_batch_size 4

Fine-tuned instruction-following models are: the Code Llama - Instruct models CodeLlama-7b-Instruct, CodeLlama-13b-Instruct, CodeLlama-34b-Instruct, CodeLlama-70b-Instruct.

Code Llama is a new technology that carries potential risks with use. Testing conducted to date has not — and could not — cover all scenarios. In order to help developers address these risks, we have created the Responsible Use Guide. More details can be found in our research papers as well.

Issues

Please report any software “bug”, or other problems with the models through one of the following means:

Model Card

See MODEL_CARD.md for the model card of Code Llama.

License

Our model and weights are licensed for both researchers and commercial entities, upholding the principles of openness. Our mission is to empower individuals, and industry through this opportunity, while fostering an environment of discovery and ethical AI advancements.

See the LICENSE file, as well as our accompanying Acceptable Use Policy

References

  1. Code Llama Research Paper
  2. Code Llama Blog Post

codellama (meta-llama/codellama)

Searching Database…

16.2K ·GitHub
Codellama
archived
这是干什么的?这是 Meta 官方的 Code Llama 推理代码。Code Llama 是基于 Llama 2 构建的一系列面向代码的大语言模型。这个仓库本身就是一个最小示例,用来加载模型并运行文本生成。
模型权重在这个仓库里,还是只有运行它们的代码?只有代码——权重需要单独下载。你需要先在 Meta 的网站上接受许可协议,通过邮件收到一个签名 URL,然后用仓库自带的 download.sh 脚本下载各个模型检查点。
我怎么把它跑起来?先在 Meta 网站上申请访问权限,运行 bash download.sh 并填入邮件里收到的签名 URL,然后用 PyTorch 和 CUDA 配置好 conda 环境,接着运行示例脚本,比如 example_completion.py 或 example_infilling.py。
开始之前我需要装好哪些东西?需要一个装好 PyTorch 的 conda 环境,另外下载脚本还用到 wget 和 md5sum——仓库没有声明其他依赖。
下载权重有什么麻烦的地方吗?有:下载链接会在 24 小时后失效,而且下载次数也有限,所以如果遇到 403 错误,就重新申请一个 URL。另外模型检查点非常大——从约 12.55GB(7B)到 63GB(34B)和 131GB(70B)不等。
Meta 把这个东西开放给谁用?个人、创作者、研究人员以及各种规模的企业——Meta 把 Code Llama 定位给任何想试验代码大语言模型的人,从业余项目到商业用途都可以。
我能在商业项目里使用 Code Llama 吗?可以——该模型和权重同时授权给研究人员和商业实体使用。它采用 Meta 的自定义许可证(GitHub 将其列为非标准许可证),附带一份可接受使用政策,并不是经典的开源许可证。
它实际上需要什么样的硬件才能运行?需要 NVIDIA GPU 环境:推理要用支持 CUDA 的 PyTorch。更大的模型需要多块 GPU——代码使用了模型并行,7B 需要 1 块,13B 需要 2 块,34B 需要 4 块,70B 需要 8 块。
有哪些模型版本可以选?三种:基础版 Code Llama、专注 Python 的 Code Llama - Python,以及遵循指令的 Code Llama - Instruct,每种都有 7B、13B、34B 和 70B 几种规模。7B 和 13B 的基础版和 Instruct 版还能在文件中间补全代码。
它能在文件中间补全代码,而不是只能从末尾接着写吗?可以——7B 和 13B 的 Code Llama 与 Code Llama - Instruct 版本支持这种中间补全(infilling):它们会根据上下文来补全代码,示例见 example_infilling.py。
输入的上下文最长能有多长?模型按 16k token 的序列训练,对最长 100k token 的输入也有提升,但 70B 的 Python 和 Instruct 版本除外。KV cache 是预先分配的,所以要把 max_seq_len 和 max_batch_size 设置成与你的硬件匹配的值。
我该怎么给纯预训练模型写提示词?它们没有针对遵循指令做微调——所以要把提示词写成:你期望的答案正好是你文字的自然延续,示例见 example_completion.py。如果要让它遵循指令,就用 Instruct 版本,通过 chat_completion() 调用,它会处理所需的格式。
这个仓库还在维护吗?不——meta-llama/codellama 在 GitHub 上已经归档,内容原样冻结。这些代码仍然是下载和运行原始 Code Llama 权重的参考,但别指望这里会有更新。
What are similar projects?llama3, petals, airllm, ai-torture-chamber, StartLux-Decision, Qwen, Model-Optimizer, EvoVLM
Which topics is it in?meta, open-weights, llama, code-generation, llm-inference, large-language-models, inference, cuda, pytorch, deep-learning, gpu, machine-learning
Creator
meta-llama
Size
<0.1 MB
Stars
16.2K
Updated
CommitMay 21
Age
3y
Commits · 90 Days