One million public Bluesky posts scraped for AI training

Bluesky is already facing its first major AI scrape, despite the stance of its owners that it will never train generative AI on user data.

Reported by 404Media on Nov. 26, one million public Bluesky posts — complete with identifying user information — were crawled and then uploaded to AI company Hugging Face. The dataset was created by machine learning librarian Daniel van Strien, intended to be used in the development of language models and natural language processing, as well as general analysis of social media trends, content moderation, and posting patterns. It contains users' decentralized identifiers (DIDs) and even has a search function to find content from specific users.

According to the dataset's description, the set "contains 1 million public posts collected from Bluesky Social's firehose API (Application Programming Interface), intended for machine learning research and experimentation with social media data. Each post contains text content, metadata, and information about media attachments and reply relationships."

Bluesky users didn't opt-in to such uses of their content, but neither is it expressly prohibited by Bluesky. The platform's firehose API is an "aggregated, chronological stream of all the public data updates as they happen in the network, including posts, likes, follows, handle changes, and more." Bluesky's API — coupled with the public and decentralized Authenticated Transfer (AT) Protocol the site is built on — means Bluesky content is open and available to the third party developers the platform is trying to court, 404Media explains.

This could be a major warning sign to many of the site's millions of new users, many of whom left competitor X in the wake of an alarming new AI training policy. A Bluesky representative responded to 404Media's requests for comment: "Bluesky is an open and public social network, much like websites on the Internet itself. Just as robots.txt files don't always prevent outside companies from crawling those sites, the same applies here. We'd like to find a way for Bluesky users to communicate to outside orgs/developers whether they consent to this and that outside orgs respect user consent, and we're actively discussing how to achieve this."

Shortly after the article's publication, the dataset was removed from Hugging Face. "I've removed the Bluesky data from the repo. While I wanted to support tool development for the platform, I recognize this approach violated principles of transparency and consent in data collection. I apologize for this mistake," van Strien wrote in a follow-up Bluesky post.