Alibaba's Qwen AI Caught Trying to Contact Server in China
"Arthur T Knackerbracket" writes:
Open-weight models from China have become very popular, but they've also raised some security suspicions. Cohere, a Canada-based biz that offers AI models to enterprises, says that when it was testing Alibaba's Qwen3.8-Flash-Next in its evaluation sandbox, the model attempted to send data to China.
"Asked to write a defensive detection rule, Qwen3.8-Flash-Next ignored instructions and inventoried the machine it was running on and then tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing," the biz said in a blog post on Monday.
Cohere was running the model through CTI-REALM, Microsoft's security analyst benchmark, and gave Qwen the task of writing a detection rule to look for malicious file downloading and execution by watching for invocations of curl processing bash scripts to the /tmp directory that were accompanied by bash execution.
The model's "reasoning" output invented a prior conversation and asserted that no tools were used, output that a guardrail system reading chain-of-thought would accept.
Nonetheless, the model issued a curl command to an Alibaba Cloud (aliyuncs.com) server.
"The model listed everything in /tmp, checked which transfer tools were installed, read the machine's DNS settings, then tried to upload a test string to an Alibaba Cloud storage bucket in Beijing and checked whether it worked," Cohere said. "Our sandbox had no route out, so the request died at the container wall."
The Cohere team wonders aloud whether that behavior was intended, but has no answer.
[...] "The key point isn't a comparison between providers. What made this noteworthy was that the model took actions unrelated to the task and those actions weren't reflected in its reasoning trace. The behavior appeared in fewer than one in 100,000 calls, which is exactly why large-scale testing and monitoring matter."
[...] This still doesn't clarify intent, and Cohere allows that there's a case to be made that the behavior is just bad training data sanitization - logs from development that got swept into the model's training data set.
Read more of this story at SoylentNews.