h2ogpt README

The project's README from github.com/h2oai/h2ogpt. As one markdown file: README.md.

h2oGPT

Turn ★ into ⭐ (top-right corner) if you like the project!

Query and summarize your documents or just chat with local private GPT LLMs using h2oGPT, an Apache V2 open-source project.

Check out a long CoT Open-o1 open 🍓strawberry🍓 project: https://github.com/pseudotensor/open-strawberry

Try Enterprise Version for Free

Enterprise h2oGPTe

Video Demo

https://github.com/h2oai/h2ogpt/assets/2249614/2f805035-2c85-42fb-807f-fd0bca79abc6

YouTube 4K Video

Features

Private offline database of any documents (PDFs, Excel, Word, Images, Video Frames, YouTube, Audio, Code, Text, MarkDown, etc.)
- Persistent database (Chroma, Weaviate, or in-memory FAISS) using accurate embeddings (instructor-large, all-MiniLM-L6-v2, etc.)
- Efficient use of context using instruct-tuned LLMs (no need for LangChain's few-shot approach)
- Parallel summarization and extraction, reaching an output of 80 tokens per second with the 13B LLaMa2 model
- HYDE (Hypothetical Document Embeddings) for enhanced retrieval based upon LLM responses
- Semantic Chunking for better document splitting (requires GPU)
Variety of models supported (LLaMa2, Mistral, Falcon, Vicuna, WizardLM. With AutoGPTQ, 4-bit/8-bit, LORA, etc.)
- GPU support from HF and LLaMa.cpp GGML models, and CPU support using HF, LLaMa.cpp, and GPT4ALL models
- Attention Sinks for arbitrarily long generation (LLaMa-2, Mistral, MPT, Pythia, Falcon, etc.)
Gradio UI or CLI with streaming of all models
- Upload and View documents through the UI (control multiple collaborative or personal collections)
- Vision Models LLaVa, Claude-3, Gemini-Pro-Vision, GPT-4-Vision
- Image Generation Stable Diffusion (sdxl-turbo, sdxl, SD3), PlaygroundAI (playv2), and Flux
- Voice STT using Whisper with streaming audio conversion
- Voice TTS using MIT-Licensed Microsoft Speech T5 with multiple voices and Streaming audio conversion
- Voice TTS using MPL2-Licensed TTS including Voice Cloning and Streaming audio conversion
- AI Assistant Voice Control Mode for hands-free control of h2oGPT chat
- Bake-off UI mode against many models at the same time
- Easy Download of model artifacts and control over models like LLaMa.cpp through the UI
- Authentication in the UI by user/password via Native or Google OAuth
- State Preservation in the UI by user/password
Open Web UI with h2oGPT as backend via OpenAI Proxy
- See Start-up Docs.
- Chat completion with streaming
- Document Q/A using h2oGPT ingestion with advanced OCR from DocTR
- Vision models
- Audio Transcription (STT)
- Audio Generation (TTS)
- Image generation
- Authentication
- State preservation
Linux, Docker, macOS, and Windows support
Inference Servers support for oLLaMa, HF TGI server, vLLM, Gradio, ExLLaMa, Replicate, Together.ai, OpenAI, Azure OpenAI, Anthropic, MistralAI, Google, and Groq
OpenAI compliant
- Server Proxy API (h2oGPT acts as drop-in-replacement to OpenAI server)
- Chat and Text Completions (streaming and non-streaming)
- Audio Transcription (STT)
- Audio Generation (TTS)
- Image Generation
- Embedding
- Function tool calling w/auto tool selection
- AutoGen Code Execution Agent
JSON Mode
- Strict schema control for vLLM via its use of outlines
- Strict schema control for OpenAI, Anthropic, Google Gemini, MistralAI models
- JSON mode for some older OpenAI or Gemini models with schema control if model is smart enough (e.g. gemini 1.5 flash)
- Any model via code block extraction
Web-Search integration with Chat and Document Q/A
Agents for Search, Document Q/A, Python Code, CSV frames
- High quality Agents via OpenAI proxy server on separate port
- Code-first agent that generates plots, researches, evaluates images via vision model, etc. (client code openai_server/openai_client.py).
- No UI for this, just API
Evaluate performance using reward models
Quality maintained with over 1000 unit and integration tests taking over 24 GPU-hours

Get Started

Install h2oGPT

Docker is recommended for Linux, Windows, and MAC for full capabilities. Linux Script also has full capability, while Windows and MAC scripts have less capabilities than using Docker.

Collab Demos

Resources

Docs Guide

Development

To create a development environment for training and generation, follow the installation instructions.
To fine-tune any LLM models on your data, follow the fine-tuning instructions.

To run h2oGPT tests:

pip install requirements-parser pytest-instafail pytest-random-order playsound==1.3.0
conda install -c conda-forge gst-python -y
sudo apt-get install gstreamer-1.0
pip install pygame
GPT_H2O_AI=0 CONCURRENCY_COUNT=1 pytest --instafail -s -v tests
# for openai server test on already-running local server
pytest -s -v -n 4 openai_server/test_openai_server.py::test_openai_client

or tweak/run tests/test4gpus.sh to run tests in parallel.

Acknowledgements

Some training code was based upon March 24 version of Alpaca-LoRA.
Used high-quality created data by OpenAssistant.
Used base models by EleutherAI.
Used OIG data created by LAION.

Why H2O.ai?

Our Makers at H2O.ai have built several world-class Machine Learning, Deep Learning and AI platforms:

#1 open-source machine learning platform for the enterprise H2O-3
The world's best AutoML (Automatic Machine Learning) with H2O Driverless AI
No-Code Deep Learning with H2O Hydrogen Torch
Document Processing with Deep Learning in Document AI

We also built platforms for deployment and monitoring, and for data wrangling and governance:

H2O MLOps to deploy and monitor models at scale
H2O Feature Store in collaboration with AT&T
Open-source Low-Code AI App Development Frameworks Wave and Nitro
Open-source Python datatable (the engine for H2O Driverless AI feature engineering)

Many of our customers are creating models and deploying them enterprise-wide and at scale in the H2O AI Cloud:

Multi-Cloud or on Premises
Managed Cloud (SaaS)
Hybrid Cloud
AI Appstore

We are proud to have over 25 (of the world's 280) Kaggle Grandmasters call H2O home, including three Kaggle Grandmasters who have made it to world #1.

Disclaimer

Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.

Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.

By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.

Star History

11.9K ·GitHub

H2ogpt

archived

这是什么？

h2oGPT 是一款开源的私有聊天应用，可以查询和总结你自己的文档，也可以直接与本地大语言模型聊天，数据不会离开你的电脑。

这是一个库，还是一个完整的应用？

这是一个完整的应用：包含基于 Gradio 的聊天界面和命令行工具（CLI），并内置兼容 OpenAI 的 API 服务器，而不只是一个需要导入的库。

我该怎么让它运行起来？

在 Linux、Windows 和 macOS 上，推荐使用 Docker 获得完整功能；此外，也提供各操作系统的原生安装脚本，但功能比 Docker 版本少。

要真正运行大语言模型，它需要哪些后端和模型？

它使用 Python 编写，支持多种后端：oLLaMa、HF TGI、vLLM、llama.cpp/GGML、GPT4ALL、Gradio、ExLLaMa、Replicate，也支持 OpenAI、Azure、Anthropic、MistralAI、Google 和 Groq 等托管 API。

这个项目还在积极维护吗？

GitHub 已将该仓库标记为已归档，因此虽然功能丰富，但已不再积极开发；另外请注意项目声明：大语言模型的输出可能存在偏差或错误，使用风险请自行承担。

它能在哪些操作系统上运行？

它官方支持 Linux、macOS 和 Windows，其中通过 Docker 安装可在这三个系统上获得完全一致的功能。

我的文档数据会留在本地吗？

会的。h2oGPT 会为你的文档（PDF、Word、图片、视频帧、音频、代码等）建立一个私有的离线数据库，内容在本地处理和查询。

h2oGPT 能导入哪些文档和媒体类型？

它可以解析 PDF、Excel、Word 文档、图片、视频帧、YouTube 视频、音频、代码、纯文本和 Markdown，用于文档问答。

h2oGPT 能替代 OpenAI 的 API 服务器吗？

可以。它运行一个兼容 OpenAI 的代理服务器，现有的 OpenAI 客户端代码可以指向 h2oGPT，用于聊天/补全、嵌入、TTS、STT 和图像生成。

它支持语音输入和输出吗？

语音转文字使用 Whisper，文字转语音使用 Microsoft SpeechT5，或一个采用 MPL2 许可、支持语音克隆的引擎，两者都支持流式音频。

内置了哪些图像生成模型？

支持的图像生成模型包括 Stable Diffusion 系列版本（SDXL-turbo、SDXL、SD3）、PlaygroundAI（playv2）和 Flux。

What are similar projects?

unsloth, LocalAI, llama-gpt, DB-GPT, turnstone, runanywhere-sdks, LibreChat, ODS

Which topics is it in?

generative, llama2, llamacpp, stable-diffusion, gradio, whisper, speech-to-text, vector-database, tts, vllm, ocr, embeddings

Creator: h2oai
Size: 53.4 MB
Stars: 11.9K
Updated: CommitOct 09 Release0.2.1Jun 04
Age: 3y
Commits · 90 Days