evaluating-code-models
Warn
Audited by Gen Agent Trust Hub on Sep 17, 2026
Risk Level: MEDIUMREMOTE_CODE_EXECUTIONCOMMAND_EXECUTIONDYNAMIC_EXECUTIONPRIVILEGE_ESCALATIONINDIRECT_PROMPT_INJECTION
Full Analysis
- [REMOTE_CODE_EXECUTION]: Fetches the evaluation harness from the BigCode Project's official GitHub repository and downloads Docker images from the GitHub Container Registry (ghcr.io).
- [COMMAND_EXECUTION]: Instructs users to enable the
--allow_code_executionflag, which permits the execution of arbitrary, untrusted code generated by AI models during the benchmarking process. - [DYNAMIC_EXECUTION]: Recommends the use of the
--trust_remote_codeflag when loading specific models from HuggingFace, which allows the execution of arbitrary Python code provided by the model's repository. - [PRIVILEGE_ESCALATION]: Includes instructions for building Docker containers using
sudoin the troubleshooting documentation (sudo make DOCKERFILE=Dockerfile all). - [INDIRECT_PROMPT_INJECTION]: The skill operates as a pipeline for processing and executing external model outputs, creating an attack surface for malicious code injection.
- Ingestion points: Model-generated code is loaded from JSON files (
generations.json) or directly from model inferences for evaluation. - Boundary markers: No explicit boundary markers are used to separate instructions from the code being executed; the system relies on user-provided flags for execution control.
- Capability inventory: The evaluation script (
main.py) has the capability to execute shell commands and arbitrary code through subprocesses and the Python interpreter. - Sanitization: The documentation provides mitigation strategies, specifically recommending the use of Docker containers to isolate the host system from the code execution environment.
Audit Metadata