serve README

The project's README from github.com/jina-ai/serve. As one markdown file: README.md.

Jina-Serve

PyPI PyPI - Downloads from official pypistats Github CD status

Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets. Scale your services from local development to production while focusing on your core logic.

Key Features

Comparison with FastAPI

Key advantages over FastAPI:

  • DocArray-based data handling with native gRPC support
  • Built-in containerization and service orchestration
  • Seamless scaling of microservices
  • One-command cloud deployment

Install

pip install jina

See guides for Apple Silicon and Windows.

Core Concepts

Three main layers:

Build AI Services

Let's create a gRPC-based AI service using StableLM:

from jina import Executor, requests
from docarray import DocList, BaseDoc
from transformers import pipeline


class Prompt(BaseDoc):
    text: str


class Generation(BaseDoc):
    prompt: str
    text: str


class StableLM(Executor):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.generator = pipeline(
            'text-generation', model='stabilityai/stablelm-base-alpha-3b'
        )

    @requests
    def generate(self, docs: DocList[Prompt], **kwargs) -> DocList[Generation]:
        generations = DocList[Generation]()
        prompts = docs.text
        llm_outputs = self.generator(prompts)
        for prompt, output in zip(prompts, llm_outputs):
            generations.append(Generation(prompt=prompt, text=output))
        return generations

Deploy with Python or YAML:

from jina import Deployment
from executor import StableLM

dep = Deployment(uses=StableLM, timeout_ready=-1, port=12345)

with dep:
    dep.block()

jtype: Deployment
with:
 uses: StableLM
 py_modules:
   - executor.py
 timeout_ready: -1
 port: 12345

Use the client:

from jina import Client
from docarray import DocList
from executor import Prompt, Generation

prompt = Prompt(text='suggest an interesting image generation prompt')
client = Client(port=12345)
response = client.post('/', inputs=[prompt], return_type=DocList[Generation])

Build Pipelines

Chain services into a Flow:

from jina import Flow

flow = Flow(port=12345).add(uses=StableLM).add(uses=TextToImage)

with flow:
    flow.block()

Scaling and Deployment

Local Scaling

Boost throughput with built-in features:

Example scaling a Stable Diffusion deployment:

jtype: Deployment
with:
 uses: TextToImage
 timeout_ready: -1
 py_modules:
   - text_to_image.py
 env:
  CUDA_VISIBLE_DEVICES: RR
 replicas: 2
 uses_dynamic_batching:
   /default:
     preferred_batch_size: 10
     timeout: 200

Cloud Deployment

Containerize Services

  1. Structure your Executor:
TextToImage/
├── executor.py
├── config.yml
├── requirements.txt
  1. Configure:
# config.yml
jtype: TextToImage
py_modules:
 - executor.py
metas:
 name: TextToImage
 description: Text to Image generation Executor
  1. Push to Hub:
jina hub push TextToImage

Deploy to Kubernetes

jina export kubernetes flow.yml ./my-k8s
kubectl apply -R -f my-k8s

Use Docker Compose

jina export docker-compose flow.yml docker-compose.yml
docker-compose up

JCloud Deployment

Deploy with a single command:

jina cloud deploy jcloud-flow.yml

LLM Streaming

Enable token-by-token streaming for responsive LLM applications:

  1. Define schemas:
from docarray import BaseDoc


class PromptDocument(BaseDoc):
    prompt: str
    max_tokens: int


class ModelOutputDocument(BaseDoc):
    token_id: int
    generated_text: str
  1. Initialize service:
from transformers import GPT2Tokenizer, GPT2LMHeadModel


class TokenStreamingExecutor(Executor):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.model = GPT2LMHeadModel.from_pretrained('gpt2')
  1. Implement streaming:
@requests(on='/stream')
async def task(self, doc: PromptDocument, **kwargs) -> ModelOutputDocument:
    input = tokenizer(doc.prompt, return_tensors='pt')
    input_len = input['input_ids'].shape[1]
    for _ in range(doc.max_tokens):
        output = self.model.generate(**input, max_new_tokens=1)
        if output[0][-1] == tokenizer.eos_token_id:
            break
        yield ModelOutputDocument(
            token_id=output[0][-1],
            generated_text=tokenizer.decode(
                output[0][input_len:], skip_special_tokens=True
            ),
        )
        input = {
            'input_ids': output,
            'attention_mask': torch.ones(1, len(output[0])),
        }
  1. Serve and use:
# Server
with Deployment(uses=TokenStreamingExecutor, port=12345, protocol='grpc') as dep:
    dep.block()


# Client
async def main():
    client = Client(port=12345, protocol='grpc', asyncio=True)
    async for doc in client.stream_doc(
        on='/stream',
        inputs=PromptDocument(prompt='what is the capital of France ?', max_tokens=10),
        return_type=ModelOutputDocument,
    ):
        print(doc.generated_text)

Support

Jina-serve is backed by Jina AI and licensed under Apache-2.0.

serve (jina-ai/serve)

Searching Database…

21.9K ·GitHub
Serve
What is this?Jina is an open-source Python framework for building multimodal AI applications on a cloud-native stack — you wrap models into gRPC microservices, chain them into flows, and scale from local to cloud.
Is it a framework or a library — and how is it different from just using FastAPI?A framework. Unlike plain FastAPI it gives you DocArray-based data handling with native gRPC, built-in containerization and service orchestration, seamless microservice scaling, and one-command cloud deployment.
How do I get it running?Install the Python package — the README points to install guides, with dedicated ones for Apple Silicon and Windows — then define and run services locally in Python or YAML.
Do I need Kubernetes or Docker set up before anything will run?No — Python is enough to run services locally. Kubernetes, Docker Compose, and the Jina Hub only enter the picture when you containerize and deploy to the cloud.
Do I have to deploy through Jina's own cloud (JCloud)?No. Services scale locally out of the box, and you can containerize and deploy them yourself to Kubernetes or with Docker Compose; JCloud's single-command deploy is just the shortcut option.
Can I use it in my own commercial product?Yes — jina-serve is Apache-2.0 licensed and backed by Jina AI, so it's free to use, including commercially.
What are the main building blocks of a Jina project?Three layers: Data — BaseDoc and DocList for input and output; Serving — Executors process Documents while a Gateway connects services; Orchestration — Deployments serve Executors and Flows create pipelines.
What does a minimal AI service look like?You create a gRPC-based service around a model — the README demos StableLM — deploy it with Python or YAML, and call it via the client. Chain several services into a Flow and you have a pipeline.
How does it handle heavy load?Built-in scaling: replicas for parallel processing, shards for data partitioning, and dynamic batching for efficient model inference — the README shows this on a Stable Diffusion deployment.
Does it support streaming LLM responses?Yes — token-by-token streaming is built in, so LLM applications can render answers as they're generated instead of waiting for the full response.
What are similar projects?metaflow, gofr, signoz, jaeger, openobserve, bifrost, OpenLLM, skypilot
Which topics is it in?neural-search, ai-framework, microservice, pipeline, tracing, llm-serving, opentelemetry, llmops, distributed, cncf, model-serving, mlops
Creator
jina-ai
Size
1.6 GB
Stars
21.9K
Updated
CommitMar 24Releasev3.28.0Nov 12
Age
6y
Commits · 90 Days