Article 6KTP2 AI Hallucinates Software Packages and Devs Download Them

AI Hallucinates Software Packages and Devs Download Them

by
janrinok
from SoylentNews on (#6KTP2)

hubie writes:

Simply look out for libraries imagined by ML and make them real, with actual malicious code. No wait, don't do that:

Several big businesses have published source code that incorporates a software package previously hallucinated by generative AI.

Not only that but someone, having spotted this reoccurring hallucination, had turned that made-up dependency into a real one, which was subsequently downloaded and installed thousands of times by developers as a result of the AI's bad advice, we've learned. If the package was laced with actual malware, rather than being a benign test, the results could have been disastrous.

According to Bar Lanyado, security researcher at Lasso Security, one of the businesses fooled by AI into incorporating the package is Alibaba, which at the time of writing still includes a pip command to download the Python package huggingface-cli in its GraphTranslator installation instructions.

There is a legit huggingface-cli, installed using pip install -U "huggingface_hub[cli]".

But the huggingface-cli distributed via the Python Package Index (PyPI) and required by Alibaba's GraphTranslator - installed using pip install huggingface-cli - is fake, imagined by AI and turned real by Lanyado as an experiment.

He created huggingface-cli in December after seeing it repeatedly hallucinated by generative AI; by February this year, Alibaba was referring to it in GraphTranslator's README instructions rather than the real Hugging Face CLI tool.

[...] The willingness of AI models to confidently cite non-existent court cases is now well known and has caused no small amount of embarrassment among attorneys unaware of this tendency. And as it turns out, generative AI models will do the same for software packages.

[...] So far at least, this technique hasn't been used in an actual attack that Lanyado is aware of.

"Besides our hallucinated package (our package is not malicious it is just an example of how easy and dangerous it could be to leverage this technique), I have yet to identify an exploit of this attack technique by malicious actors," he said. "It is important to note that it's complicated to identify such an attack, as it doesn't leave a lot of footsteps."

Original Submission

Read more of this story at SoylentNews.

External Content
Source RSS or Atom Feed
Feed Location https://soylentnews.org/index.rss
Feed Title SoylentNews
Feed Link https://soylentnews.org/
Feed Copyright Copyright 2014, SoylentNews
Reply 0 comments