What is this?
It's a collection of Python scripts that probe OpenAI-API-compatible model endpoints to check whether a model is 'masked' (mislabeled as a different architecture than it actually is). Each script fetches the model list, runs an identity/logic probe, and prints a summary.
It's a collection of Python scripts that probe OpenAI-API-compatible model endpoints to check whether a model is 'masked' (mislabeled as a different architecture than it actually is). Each script fetches the model list, runs an identity/logic probe, and prints a summary.Is this a single tool or a set of separate scripts?It's three standalone Python scripts, not one unified CLI tool: each is run separately and targets a different testing scenario (single-model, InferHub, or CFRouter).
It's three standalone Python scripts, not one unified CLI tool: each is run separately and targets a different testing scenario (single-model, InferHub, or CFRouter).How do I set it up?Copy .env.example to .env and set the variables manually. On Windows, PowerShell does not auto-load .env files, so you need to set the environment variables yourself before running a script.
Copy .env.example to .env and set the variables manually. On Windows, PowerShell does not auto-load .env files, so you need to set the environment variables yourself before running a script.Do I need an API key to run this?Yes, you need an OpenAI-compatible API key and endpoint URL, read from an environment variable rather than hardcoded in the script.
Yes, you need an OpenAI-compatible API key and endpoint URL, read from an environment variable rather than hardcoded in the script.Does running these probes cost me anything?Yes, each probe sends a real inference request to the target endpoint, which can consume your provider quota or billing.
Yes, each probe sends a real inference request to the target endpoint, which can consume your provider quota or billing.Is it safe to point this at any endpoint?No, you should only test endpoints and API keys that you own or have explicit permission to test; never commit your real .env or key.
No, you should only test endpoints and API keys that you own or have explicit permission to test; never commit your real .env or key.What does 'masking' mean in this context?It refers to a model being mislabeled or rebranded - for example a provider advertising a model as DeepSeek while it actually answers like GPT. The scripts flag this by comparing the model family in the label against what the model says about itself in an identity probe.
It refers to a model being mislabeled or rebranded - for example a provider advertising a model as DeepSeek while it actually answers like GPT. The scripts flag this by comparing the model family in the label against what the model says about itself in an identity probe.What's the difference between the three scripts?cekmodelfast.py lets you pick a single model and runs an identity/logic probe with a mismatch warning; cekmodelwwaiting.py tests all InferHub models with retries; cekmodelwaiting3s.py tests all CFRouter models with a delay between requests.
cekmodelfast.py lets you pick a single model and runs an identity/logic probe with a mismatch warning; cekmodelwwaiting.py tests all InferHub models with retries; cekmodelwaiting3s.py tests all CFRouter models with a delay between requests.How does it detect a mismatch?cekmodelfast.py prints an 'ARCHITECTURE MISMATCH / SUSPECT' warning when the model family implied by its label (e.g. DeepSeek) differs from the family the model claims to be in its identity probe answer (e.g. GPT).
cekmodelfast.py prints an 'ARCHITECTURE MISMATCH / SUSPECT' warning when the model family implied by its label (e.g. DeepSeek) differs from the family the model claims to be in its identity probe answer (e.g. GPT).What are similar projects?iFixAi, jevlint, pyinstaller, aichat, strix, dingtalk-workspace-cli, crush, promptfoo
iFixAi, jevlint, pyinstaller, aichat, strix, dingtalk-workspace-cli, crush, promptfooWhich topics is it in?llm-security, openai-api, script, llms, python3, cli-tool, inference, testing, openai, command-line-tool, security, api
llm-security, openai-api, script, llms, python3, cli-tool, inference, testing, openai, command-line-tool, security, api