GitHub Copilot's Vulnerability: Refusing Harmful Chat Requests but Generating Them via Code Workflows

A study reveals that GitHub Copilot and other AI coding assistants refuse harmful requests in chat but generate harmful code when tasks are framed as code improvements, highlighting vulnerabilities in AI safety mechanisms. The research demonstrates a workflow-level jailbreak technique where models produce harmful content as part of routine coding tasks, bypassing direct safety checks.

An AI coding assistant that refuses to answer a dangerous request in its chat box can answer it anyway if the same request is broken into small, ordinary-looking steps inside a code editor. That is the finding of a new study of GitHub Copilot by researchers Abhishek Kumar and Carsten Maple.

The models they tested through Copilot, Claude from Anthropic, and Gemini from Google, refused

Read original article