djev-run README

The project's README from github.com/taeold/djev-run. As one markdown file: README.md.

djev-run

Open In Colab

Serve DiffusionGemma-Jev (djev) on a TypeSafe AI compatible API on Cloud Run with an NVIDIA RTX PRO 6000 Blackwell or NVIDIA L4 GPU, or run directly inside Google Colab Pro (A100 40GB/80GB or L4 24GB) via colab.ipynb, built on mmastrac/djev.

Built-in demo apps:

djev snake djev tetris

Deploy on Google Cloud Run

Follows Cloud Run GPU best practices.

Step 1: Upload Model to GCS

export BUCKET="your-gcs-bucket"
export REGION="us-central1" # Supported RTX PRO 6000 regions: us-central1, europe-west4, asia-southeast1, asia-south2

gcloud storage buckets create "gs://${BUCKET}" --location="${REGION}"
hf download nvidia/diffusiongemma-26B-A4B-it-NVFP4 --local-dir /tmp/dgemma
gcloud storage cp -r /tmp/dgemma/* "gs://${BUCKET}/dgemma/"

Step 2: Deploy to Cloud Run

gcloud beta run deploy djev-dgemma \
  --region="${REGION}" \
  --image=ghcr.io/taeold/djev-run:latest \
  --gpu=1 \
  --gpu-type=nvidia-rtx-pro-6000 \
  --no-gpu-zonal-redundancy \
  --cpu=20 \
  --memory=80Gi \
  --no-cpu-throttling \
  --concurrency=32 \
  --min-instances=0 \
  --max-instances=1 \
  --port=8080 \
  --network=default \
  --subnet=default \
  --vpc-egress=all-traffic \
  --add-volume=name=weights,type=cloud-storage,bucket="${BUCKET}",readonly=false,mount-options=enable-buffered-read=true \
  --add-volume-mount=volume=weights,mount-path=/mnt/gcs \
  --startup-probe=httpGet.path=/health,httpGet.port=8080,initialDelaySeconds=5,periodSeconds=2,timeoutSeconds=2,failureThreshold=120 \
  --set-env-vars="MODEL=/mnt/gcs/dgemma,CANVAS=128,MAX_SEQS=32,MAX_MODEL_LEN=4096,GPU_UTIL=0.40,KV_CACHE_GB=2,ATTN=TRITON_ATTN,TEST_PAGE=1,COPY_TO_SHM=1,ENFORCE_EAGER=1,DISABLE_MM=1,TORCH_COMPILE_DISABLE=1,VLLM_WORKER_MULTIPROC_METHOD=fork,VLLM_UF_EAGER_ALL=1,VLLM_FLASHINFER_MOE_BACKEND=masked_gemm,CUDA_MODULE_LOADING=LAZY"

# --image=ghcr.io/taeold/djev-run:latest: prebuilt from github.com/mmastrac/djev (upstream does not publish a registry image)
# --no-gpu-zonal-redundancy: required for standard regional RTX PRO 6000 quota
# --no-cpu-throttling: keeps all 20 vCPUs active during weight loading and vLLM scheduling
# --network=default --subnet=default --vpc-egress=all-traffic: streams weights from GCS over Google internal networking (~1.05 GiB/s)
# mount-options=enable-buffered-read=true: prefetches 18 GB safetensors shards sequentially from GCS
# TEST_PAGE=1: enables the built-in /snake, /dino, /tetris, and /playground web UIs
# COPY_TO_SHM=1: stages the 17.5 GB model into /dev/shm RAM in the background while Python imports torch/vllm
# VLLM_WORKER_MULTIPROC_METHOD=fork: forks EngineCore from APIServer without re-importing Python
# ENFORCE_EAGER=1 & TORCH_COMPILE_DISABLE=1: skips torch.compile, CUDA graph capture, and redundant startup profiling
# DISABLE_MM=1: skips SigLIP vision/video encoder profiling for text-only evaluation

Step 3: Open the Built-in Demos

All four demos are standalone HTML files with zero external dependencies that call POST /v1/systemone directly from the browser via fetch():

Use with Vercel AI SDK (Optional)

Because djev-run implements the /v1/systemone endpoint contract, you can also point @ai-sdk/typesafe-ai at your Cloud Run URL from Node.js or TypeScript (index.ts):

import { createTypeSafeAi } from '@ai-sdk/typesafe-ai';
import { experimental_evaluate, type Experimental_EvaluationModel } from 'ai';

const typeSafeAi = createTypeSafeAi({
  baseURL: 'https://<your-cloud-run-url>/v1',
});

async function triage(model: Experimental_EvaluationModel, message: string) {
  return experimental_evaluate({
    model,
    state: { message },
    questions: {
      department: {
        type: 'choice',
        instructions: 'Which team should handle this?',
        criteria: {
          billing: 'Payments and refunds',
          support: 'Other requests',
        },
      },
      severity: {
        type: 'score',
        instructions: 'How severe is the issue?',
        criteria: ['Cosmetic', 'Workaround exists', 'Blocking; no workaround'],
      },
      requestsRefund: {
        type: 'boolean',
        instructions: 'Is the customer requesting money back?',
      },
    },
  });
}

const result = await triage(
  typeSafeAi.evaluationModel('jev-latest'),
  'I was charged twice and my account is locked',
);
console.log(result);
// {
//   department: {
//     type: 'choice',
//     choice: 'billing',
//     probabilities: { billing: 0.9988, support: 0.0012 }
//   },
//   severity: {
//     type: 'score',
//     score: 1.9995,
//     probabilities: { '0': 0.0002, '1': 0.0001, '2': 0.9997 }
//   },
//   requestsRefund: {
//     type: 'boolean',
//     probability: 0.9928
//   }
// }

npm install
CLOUD_RUN_URL="https://<your-cloud-run-url>" TYPESAFE_AI_API_KEY="$(gcloud auth print-identity-token)" npm start

Performance

JevBench v1.3.0 (N = 231 public suite):

Configuration Overall Accuracy Composite Score Median Response Time
djev-run (steps=1, samples=1) 81.4% 73.4 117 ms
djev-run (steps=1, samples="auto") 81.8% 73.5 121 ms
api.djev.dev 81.8% 73.0 237 ms

To reach a 47.5-second cold start on Cloud Run, the container streams the 17.5 GB safetensors weights from GCS into /dev/shm RAM in the background (1.05 GiB/s) while Python imports torch and vllm, forks EngineCore from APIServer (VLLM_WORKER_MULTIPROC_METHOD=fork) so modules are not imported twice, disables unused SigLIP vision profiling (DISABLE_MM=1), and skips torch.compile, CUDA graph capture, and redundant memory-profiling passes (ENFORCE_EAGER=1, TORCH_COMPILE_DISABLE=1, --kv-cache-memory).

Empirical benchmarks on Cloud Run with the RTX PRO 6000 confirmed:

Pricing

1 NVIDIA RTX PRO 6000 GPU (20 vCPU, 80 GiB RAM) costs $3.19 per hour while active and scales to $0 when idle with --min-instances=0.

djev-run (taeold/djev-run)

Searching Database…

585 ·GitHub
Djev-run
What is this?djev-run serves DiffusionGemma-Jev (djev) behind a TypeSafe AI compatible API on Google Cloud Run with an NVIDIA RTX PRO 6000 Blackwell or L4 GPU, or runs it directly inside Google Colab Pro via colab.ipynb; it is built on mmastrac/djev.
Is this a hosted service I sign up for, or something I deploy myself?You deploy it yourself: to your own Google Cloud Run project backed by an NVIDIA GPU, or by running colab.ipynb in Colab Pro. The deployment follows Cloud Run GPU best practices.
How do I get it running?On Cloud Run: upload the model to GCS, deploy to Cloud Run, then open the built-in demos. Alternatively run colab.ipynb inside Google Colab Pro on an A100 40GB/80GB or L4 24GB.
Do I need special hardware to run it?Yes, an NVIDIA GPU: an RTX PRO 6000 Blackwell or L4 on Cloud Run, or A100 40GB/80GB or L4 24GB in Colab Pro. The model ships as 17.5 GB of safetensors weights.
How does it hit a 47.5-second cold start?The container streams the 17.5 GB weights from GCS into /dev/shm RAM at ~1.05 GiB/s while torch and vllm import, forks EngineCore from APIServer so modules are not imported twice, and skips torch.compile and CUDA graph capture. Streaming beats baking weights into the image (80-115 s).
What does the Vision demo show?One-step 6x6 spatial semantic segmentation — all 36 cells in about 73 ms — plus a 16-sensor System-1 visual radar built on gemma4vision with 280 soft tokens.
Why deploy my own instance instead of using api.djev.dev?Accuracy is nearly identical (81.4% vs 81.8%, composite 73.4 vs 73.0 on JevBench v1.3.0), but a self-deployed Cloud Run instance answers about twice as fast: 117 ms vs 237 ms median end-to-end.
What are similar projects?solo_jev, qwen3.8-Flash-DGX, Qwen3.8-Flash-Next-Dual-DGX-Sparks, tokenspeed, petals, vllm-omni, GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark, v100-skinny
Which topics is it in?colab, vision, gcp, segmentation, model-serving, llm-inference, vllm, jupyter notebook, inference, nvidia, computer-vision, benchmark
Creator
taeold
Size
1.8 MB
Stars
585
Updated
CommitOct 14
Age
30d
Commits · 90 Days