langsmith-evaluator

External Downloads

• Description: The skill provides a setup command to install the LangSmith CLI via a remote shell script.
• Concern: Downloading and piping a script to a shell (curl | sh) is a pattern used for remote code execution. In this instance, the script is hosted on cli.langsmith.com, which is an official domain of the trusted vendor, LangChain. This is a standard installation procedure for the tool.
• Evidence:
• SKILL.md: curl -fsSL https://cli.langsmith.com/install.sh | sh

Indirect Prompt Injection

• Description: The skill defines methods for analyzing and grading the outputs and execution paths (trajectories) of AI agents.
• Concern: By ingesting data generated by agents, the skill establishes a surface for indirect prompt injection. Malicious instructions contained within agent responses could attempt to manipulate evaluation results or the behavior of evaluators using the 'LLM-as-Judge' pattern. This represents a known risk associated with the automated analysis of LLM outputs.
• Evidence:
• SKILL.md: The <run_functions> and <evaluator_format> sections detail how agent outputs are passed to evaluator functions for processing.