Join the Github discussion here.

Metamorphosis is the process where a caterpillar forms a chrysalis, liquifies its own body, and emerges as an imago, or butterfly. It is a pretty nifty trick. Anyways, trees are budding, Lexicons are blossoming, spring is happening, and it is time for an update to the AT Protocol roadmap.

We recently summarized progress on the protocol in 2024. This blog post will be forward looking, covering our protocol goals for the next 6-7 months. As a high-level summary:

• Updates to the relay, firehose, and public repo sync semantics (Sync v1.1) are starting to roll out
• Design work on Auth Scopes has started, which will improve atproto OAuth
• PDS will get a web interface for generic account management and signup
• Shared data (eg, group privacy) will likely be the next major protocol component, with E2EE DMs following that

We also have a quick section on deprecated developer patterns; please give those a look!

Sync v1.1​

We are iterating on the core public data synchronization components of the protocol. Relays will become much cheaper to operate, and we’re clarifying the process for fully validating the firehose. The full proposal gets into all the details, but to summarize:

• Efficient mechanism for validating MST operations in individual repository commits ("inductive firehose")
• Adding a new #sync message type, and removing the tooBig flag on commits
• New desynchronized and throttled account statuses, to communicate temporary failures
• New com.atproto.sync.listReposByCollection endpoint to help with backfill

Auth Scopes​

We are updating OAuth for AT Protocol with a way to request and grant granular permissions. For example, it should be possible to give a client permission to read and write posts on Bluesky, but not insert arbitrary block records or access DMs. This is obviously important for user control, privacy, and account security. The system will allow application designers to declare their own auth scopes, as part of the Lexicon system. PDS implementations will be able to enforce these permissions in an interoperable way, at runtime. We will share more details soon.

In addition to completing OAuth for existing apps, Auth Scopes will be necessary for upcoming protocol features, like group-private data and on-protocol DMs.

PDS Account Management​

More and more folks are building independent apps on atproto. While they can use OAuth to authenticate users from any PDS instance, account signup is more complicated. In theory it is possible to implement account creation using the com.atproto.* Lexicons, but in practice this is difficult (or impossible) to implement in independent apps, because of anti-bot measures. This results in developers directing new users to sign up with Bluesky, which is a bad user experience, and conflates having an account on the AT Protocol with having a Bluesky account.

To improve this situation, we are implementing a web interface in the PDS reference distribution which will give users a less-branded account sign-up experience. The PDS technically already has a web interface, used for the OAuth authorization flow, and this simply extends that. Over time, we expect the web interface to provide generic account management capabilities, such as password recovery flows, additional 2FA mechanisms, management of active auth sessions, account deactivation, etc.

The details of the web interface will be implementation-specific. Other PDS implementations might provide different functionality, or make different design choices.

Privately Shared Data and E2EE DMs​

We believe that robust support for group-private data will be necessary for the long-term success of the protocol (and for apps built on the protocol). Similarly, the ability to share private content with a specific group or audience continues to be a top feature request for both the AT Protocol and the Bluesky app. Just as we’re currently doing with public conversation on the Bluesky app and the AT Protocol, we also want to co-design the protocol specification for private data in tandem with specific real-world product features: this results in better outcomes for both. Designing for privacy is pretty different from designing for global broadcast, and we think the data architecture will probably look pretty different from the MST + firehose system.

Shared data will depend on Auth Scopes, and we don't expect to start design work until that is complete.

Looking forward, we continue to have plans to implement on-protocol DMs and E2EE group chat. However, we don’t expect to start work on this until after shared data is implemented. Meanwhile, there has been exciting progress in the broader tech world around the Messaging Layer Security (MLS) standard, and we are optimistic that we will be able to build on reusable components and design patterns when the time comes. It is also possible (and exciting!) that the atproto dev community will experiment and build E2EE chat apps off-protocol before there is an official specification.

Deprecations​

There are a few protocol features and API endpoints which were supported in early days of atproto development. They have been deprecated for some time, but have continued to function. As the protocol stabilizes, we want to ensure developers are building against the current protocol, and will start to remove this functionality more aggressively.

A simple deprecation are the #tombstone, and #handle, and #migrate events on the firehose. These were replaced with #identity and #account early last year (2024), and have been deprecated since then. We will remove them from the atproto Lexicons entirely soon.

Client apps should resolve user login identifiers (handles or DIDs) to PDS instances, and should not hardcode the bsky.social domain for API requests. In the early days, all API requests could be made to this server, and we have continued to proxy requests to avoid breakage. Most clients and SDKs have been updated, and we may stop proxying in the near future.

When making proxied requests to a PDS, clients can specify a remote service to forward to via the atproto-proxy header. To date, the reference PDS implementation has automatically forwarded app.bsky.* endpoints to the Bluesky API server (api.bsky.app). No other services or Lexicon namespaces in the network have this sort of default forwarding. To keep the network more provider-neutral, clients should not rely on this default, and should always specify a service in the proxy header. The service DID reference for the Bluesky AppView is did:web:api.bsky.app#bsky_appview; you can see more example service DIDs in the API Hosts and Auth docs.

Keep up with Ecosystem​

The AT Protocol developer ecosystem continues to grow at a fast pace, with more developers launching new projects and organizations by the week. Here are some ways to stay updated or get involved:

• Follow the @atproto.com Bluesky account
• Contribute to or read discussions on Github
• Subscribe to the The Fediverse Report, an independent newsletter that includes weekly AT Protocol coverage
• Attend atprotocol.dev’s independently-run Tech Talks or the ATmosphere Conference this month in Seattle

Today we are excited to announce the availability of version 0.14 of our TypeScript SDK on npm.

This release is a big step forward, significantly improving the type safety of our @atproto/api package. Let’s take a look at the highlights:

• Lexicon derived interfaces now have an explicitly defined $type property, allowing proper discrimination of unions.
• Lexicon derived is* utility methods no longer unsafely type cast their input.
• Lexicon derived validate* utility methods now return a more precise type.

Context​

Atproto is an "open protocol" which means a lot of things. One of these things is that the data structures handled through the protocol are extensible. Lexicons (which is the syntax used to define the schema of the data structures) can be used to describe placeholders where arbitrary data types (defined through third-party Lexicons) can be used.

An example of such a placeholder exists in the Lexicon definition of a Bluesky post (app.bsky.feed.post), which enables posts to have an embed property defined as follows:

  "embed": {
    "type": "union",
    "refs": [
      "app.bsky.embed.images",
      "app.bsky.embed.video",
      "app.bsky.embed.external",
      "app.bsky.embed.record",
      "app.bsky.embed.recordWithMedia",
    ]
  }

The type of the embed property is what is called an "open union". It means that the embed field can basically contain anything, although we usually expect it to be one of the known types defined in the refs array of the Lexicon schema (an image, a video, a link, or another post).

Systems consuming Bluesky posts need to be able to determine what type of embed they are dealing with. This is where the $type property comes in. This property allows systems to uniquely determine the Lexicon schema that must be used to interpret the data, and it must be provided everywhere a union is expected. For example, a post with a video would look like this:

{
  "text": "Hey, check this out!",
  "createdAt": "2021-09-01T12:34:56Z",
  "embed": {
    "$type": "app.bsky.embed.video",
    "video": {
      /* reference to the video file, omitted for brevity */
    }
  }
}

Since embed is an open union, it can be used to store anything. For example, a post with a calendar event embed could look like this:

{
  "text": "Hey, check this out!",
  "createdAt": "2021-09-01T12:34:56Z",
  "embed": {
    "$type": "com.example.calendar.event",
    "eventName": "Party at my house",
    "eventDate": "2021-09-01T12:34:56Z"
  }
}

Revamped TypeScript interfaces​

In order to facilitate working with the Bluesky API, we provide TypeScript interfaces generated from the lexicons (using a tool called lex-cli). These interfaces are made available through the @atproto/api package.

For historical reasons, these generated types were missing the $type property. The interface for the app.bsky.embed.video, for example, used to look like this:

export interface Main {
  video: BlobRef
  captions?: Caption[]
  alt?: string
  aspectRatio?: AppBskyEmbedDefs.AspectRatio
  [k: string]: unknown
}

Because the $type property is missing from that interface, developers could write invalid code, without getting an error from TypeScript:

import { AppBskyFeedPost } from '@atproto/api'

const myPost: AppBskyFeedPost.Main = {
  text: 'Hey, check this out!',
  createdAt: '2021-09-01T12:34:56Z',
  embed: {
    // Notice how we are missing the `$type` property
    // here. TypeScript did not complain about this.

    video: {
      /* reference to the video file, omitted for brevity */
    },
  },
}

Similarly, a Bluesky post’s embed property was previously typed like this:

export interface Record {
  // ...
  embed?:
    | AppBskyEmbedImages.Main
    | AppBskyEmbedVideo.Main
    | AppBskyEmbedExternal.Main
    | AppBskyEmbedRecord.Main
    | AppBskyEmbedRecordWithMedia.Main
    | { $type: string; [k: string]: unknown }
}

It was therefore possible to create a post with a completely invalid "video" embed, and still get no error from the type system:

import { AppBskyFeedPost } from '@atproto/api'

const myPost: AppBskyFeedPost.Main = {
  text: 'Hey, check this out!',
  createdAt: '2021-09-01T12:34:56Z',

  // This is an invalid embed, but TypeScript
  // does not complain.
  embed: {
    $type: 'app.bsky.embed.video',
    video: 43,
  },
}

We have fixed these issues by making the $type property in the generated interfaces explicit. The app.bsky.embed.video interface now looks like this:

export interface Main {
  $type?: 'app.bsky.embed.video'
  video: BlobRef
  captions?: Caption[]
  alt?: string
  aspectRatio?: AppBskyEmbedDefs.AspectRatio
}

Notice how the $type property is defined as optional (?:) here. This is due to the fact that the schema definitions are not always used from open unions. In some cases, a particular schema can be referenced from another schema (using a "type": "ref"). In those cases, there will be no ambiguity as to how the data should be interpreted.

For example, a "Bluesky Like" (app.bsky.feed.like) defines the following properties in its schema:

  "properties": {
    "createdAt": { "type": "string", "format": "datetime" },
    "subject": { "type": "ref", "ref": "com.atproto.repo.strongRef" }
  },

As can be seen, the subject property is defined as a reference to a com.atproto.repo.strongRef object. In this case, there is no ambiguity as to how the subject of a like should be interpreted, and the $type property is not needed.

const like: AppBskyFeedLike.Record = {
  $type: 'app.bsky.feed.like',
  createdAt: '2021-09-01T12:34:56Z',
  subject: {
    // No `$type` property needed here
    uri: 'at://did:plc:123/app.bsky.feed.post/456',
    cid: '[...]',
  },
}

Because the $type property of objects is required in some contexts while optional in others, we introduced a new type utility type to make it required when needed. The $Typed utility allows marking an interface’s $type property non-optional in contexts where it is required:

export type $Typed<V> = V & { $type: string }

The embed property of posts is now defined as follows:

export interface Record {
  // ...
  embed?:
    | $Typed<AppBskyEmbedImages.Main>
    | $Typed<AppBskyEmbedVideo.Main>
    | $Typed<AppBskyEmbedExternal.Main>
    | $Typed<AppBskyEmbedRecord.Main>
    | $Typed<AppBskyEmbedRecordWithMedia.Main>
    | { $type: string }
}

In addition to preventing the creation of invalid data as seen before, this change also allows properly discriminating types when accessing the data. For example, one can now do:

import { AppBskyFeedPost } from '@atproto/api'

// Say we got some random post somehow (typically
// via an API call)
declare const post: AppBskyFeedPost.Main

// And we want to know what kind of embed it contains
const { embed } = post

// We can now use the `$type` property to disambiguate
if (embed?.$type === 'app.bsky.embed.images') {
  // The `embed` variable is fully typed as
  // `$Typed<AppBskyEmbedImages.Main>` here!
}

$type property in record definitions​

While optional in interfaces generated from Lexicon object definitions, the $type property is required in interfaces generated from Lexicon record definitions.

is* utility methods​

The example above shows how data can be discriminated based on the $type property. The SDK provides utility methods to perform this kind of discrimination. These methods are named is* and are generated from the lexicons. For example, the app.bsky.embed.images Lexicon used to generate the following isMain utility method:

export interface Main {
  images: Image[]
  [x: string]: unknown
}

export function isMain(value: unknown): value is Main {
  return (
    value != null &&
    typeof value === 'object' &&
    '$type' in value &&
    (value.$type === 'app.bsky.embed.images' ||
      value.$type === 'app.bsky.embed.images#main')
  )
}

That implementation of the discriminator is invalid.

• First, because a $type is not allowed to end with #main (as per AT Protocol specification).
• Second, because the isMain function does not actually check the structure of the object, only its $type property.

This invalid behavior could yield runtime errors that could otherwise have been avoided during development:

import { AppBskyEmbedImages } from '@atproto/api'

// Get an invalid embed somehow
const invalidEmbed = {
  $type: 'app.bsky.embed.images',
  // notice how the `images` property is missing here
}

// This predicate function only checks the value of
// the `$type` property, making the condition "true" here
if (AppBskyEmbedImages.isMain(invalidEmbed)) {
  // However, the `images` property is missing here.
  // TypeScript does not complain about this, but the
  // following line will throw a runtime error:
  console.log('First image:', invalidEmbed.images[0])
}

The root of the issue here is that the is* utility methods perform type casting of objects solely based on the value of their $type property. There were basically two ways we could fix this behavior:

1. Alter the implementation to actually validate the object's structure. This would be a non-breaking change that has a negative impact on performance.
2. Alter the function signature to describe what the function actually does. This is a breaking change because TypeScript would start (rightfully) returning lots of errors in places where these functions are used.

Because this release introduces other breaking changes, and because adapting our own codebase to this change showed it made more sense, we decided to adopt the latter option.

For example, this is the case when working with data obtained from the API. Because an API is a "contract" between a server and a client, the data returned by Bluesky's server APIs is "guaranteed" to be valid. In these cases, the is* utility methods provide a convenient way to discriminate between valid values.

import { AppBskyEmbedImages } from '@atproto/api'

// Get a post from the API (the API's contract
// guarantees the validity of the data)
declare const post: AppBskyEmbedImages.Main

// The `is*` utilities are an efficient way to
// discriminate **valid** data based on their `$type`
if (isImages(post.embed)) {
  // `post.embed` is fully typed as
  // `$Typed<AppBskyEmbedImages.Main>` here!
}

validate* utility methods​

As part of this update, the signature of the validate* utility methods was updated to properly describe the type of the value in case of success:

import { AppBskyEmbedImages } from '@atproto/api'

// Aliased for clarity
const Images = AppBskyEmbedImages.Main
const validateImages = AppBskyEmbedImages.validateMain

// Get some data somehow
declare const data: unknown

// Validate the data against a particular schema (images here)
const result = validateImages(data)

if (result.success) {
  // The `value` property was previously typed as `unknown`
  // and is now properly typed as `Image`
  const images = result.value
}

These methods perform data validation, making them somewhat slower than the is* utility methods. They can, however, be used in place of the is* utilities when migrating to this new version of the SDK.

New asPredicate function​

The SDK exposes a new asPredicate function. This function allows converting a validate* function into a predicate function. This can be useful when working with libraries that expect a predicate function to be passed as an argument.

import { asPredicate, AppBskyEmbedImages } from '@atproto/api'

// Aliased for clarity
const Images = AppBskyEmbedImages.Main
const isValidImages = asPredicate(AppBskyEmbedImages.validateMain)

// Get an embed with unknown validity somehow
declare const embed: unknown

// The following condition will be true if, and only
// if, the value matches the `Image` interface.
if (isValidImages(embed)) {
  // `embed` is of type `Images` here
}

// Similarly, the type predicate can be used to
// infer the type of an array of unknown values:
declare const someArray: unknown[]

// This will be typed as `Images[]`
const images = someArray.filter(isValidImages)

Removal of the [x: string] index signature​

Another property of Atproto being an "open protocol" is the fact that objects are allowed to contain additional — unspecified — properties (although this should be done with caution to avoid incompatibility with properties that are added in the future). This used to be represented in the type system using a [k: string]: unknown index signature in generated interfaces. This is how the video embed used to be represented:

export interface Main {
  video: BlobRef
  captions?: Caption[]
  alt?: string
  aspectRatio?: AppBskyEmbedDefs.AspectRatio
  [k: string]: unknown
}

This signature allowed for undetectable mistakes to be performed:

import { AppBskyEmbedVideo } from '@atproto/api'

const embed: AppBskyEmbedVideo.Main = {
  $type: 'app.bsky.embed.video',
  video: {
    /* omitted */
  },

  // Notice the typo in `alt`, not resulting in a TypeScript error
  atl: 'My video',
}

We removed that signature, requiring any unspecified fields intentionally added to be now explicitly marked as such:

import { AppBskyEmbedVideo } from '@atproto/api'

const embed: AppBskyEmbedVideo.Main = {
  $type: 'app.bsky.embed.video',
  video: {
    /* omitted */
  },

  // Next line will result in the following
  //  TypeScript error: "Object literal may only
  // specify known properties, and 'atl' does not
  // exist in type 'Main'"
  atl: 'My video',

  // Unspecified fields must now be explicitly
  // marked as such:

  // @ts-expect-error - custom field
  comExampleCustomProp: 'custom value',
}

Other considerations​

When upgrading, please make sure that your project does not depend on multiple versions of the @atproto/* packages. Use resolutions or overrides in your package.json to pin the dependencies to the same version.

Recap​

We hope this release helps you build better codebases with improved type safety. During our own migration, we found and fixed a few small bugs, and we believe these changes will benefit the entire developer community.

Migration TL;DR:​

• Need to be absolutely sure of your data? Use asPredicate or validate* utilities.
• Using data from the Bluesky app view? You can use is* utilities.
• Building lex objects for writing? Make sure you use $Typed when building those.

Happy coding!

In May 2024, we published a 2024 Protocol Roadmap, and we want to give an end-of-year update. We will follow up soon with a forward-looking roadmap for 2025.

In the big picture, most of the public data aspects of the protocol have now been designed and implemented. The last missing pieces are nearing completion, and we do not foresee disruptive changes or additions which would impact interoperability. Now is a great time to start building on the protocol and assembling independent infrastructure.

A lot of progress was made in 2024! Some large protocol milestones include:

• Open PDS federation in the live network: At the time of roll-out, Bluesky initially required pre-registration and placed a limit on the number of hosted accounts. These limits have since been removed, and now any PDS can participate in the live network without prior coordination. We encourage the growth of independent PDS instances of any size. There are some rate-limits in place to prevent bot farms, but we will increase them when needed to accommodate PDS growth.
• Labeling: We launched the stackable moderation system with labels and reports.
• Account Migration: Users can migrate their account to alternate PDSes using command-line tooling.
• Generic Service Proxying: Client XRPC requests to the PDS can be proxied on to arbitrary service providers, using inter-service auth, via a client-controlled HTTP header.
• Flexible Record Schemas: Data records of any schema can be written to atproto repositories, with "eager" Lexicon validation controlled by query parameter.
• Account Deactivation: This was implemented at the protocol layer, along with an overhaul of the #identity and #account firehose events.
• Initial OAuth Support: This was launched in production, with extensive documentation.
• Jetstream: We shipped an alternative WebSocket API for the firehose, which uses simple JSON and record-level operations (as opposed to "commits"). This makes it easier for independent developers to process the Bluesky firehose.

Lexicon Resolution​

One of the more recently-completed components is Lexicon resolution. This is the mechanism for looking up the schema of new data by NSID. After feedback on a public proposal, we settled on a design that uses DNS TXT records that map to a DID, then schemas stored as records in an atproto repo. We have some early implementations which prove out the design.

We plan on writing this up as a formal specification, and would like to build tooling to support publishing, mirroring, discovery, and integration of Lexicons.

Auth and OAuth​

The OAuth announcement blog post gives a good overview of the progress made in 2024, and links out to developer resources. Server-side support has been implemented in the Bluesky PDS distribution, and several independent projects are using it for login in the live network. We iterated on our design to align with the Web Auth Working Group of the IETF, and will make small changes as needed to stay in alignment.

One of the missing pieces is Auth Scopes, which will allow more granular and flexible Authorization grants. We have made a fair amount of design progress on this mechanism but still have a few issues to resolve. Keep an eye out for a public summary of this work soon.

Based on the OAuth Roadmap, we are still in the first phase. Apart from Scopes, we have feedback from developers that token lifetimes may need tuning.

Apart from OAuth, we are increasingly interested in a more powerful and flexible auth token mechanism, but have not started any design or planning work yet.

Sync, Firehose, and Backfill​

As the overall network has scaled to over 26 million accounts, resource costs around Relays and the firehose mechanism have become more urgent.

The initial Relay design required a full mirror of all repository data (records and MST nodes) to fully verify each commit message. This meant that disk consumption increased both with the number of accounts, and the amount of data each account stored. This was considered relatively affordable, and further scaling could be achieved with infrastructure sharding.

However, we are currently exploring a "non-archival" relay design which is significantly cheaper to operate, even at full network scale. This will simplify the role of relays in the network, replace the tooBig mechanism for large commits, and clarify what a downstream service needs to do to synchronize data reliably.

We are also working on improved ergonomics for working with subsets of data in the network. In particular, new small applications (Lexicon schemas) should be able to start small. It is important to be able to backfill only the relevant data already in the network and then subscribe to just a subset of data downstream of the firehose.

Specifications and Ecosystem​

We refreshed the AT Protocol website, and the written specifications were expanded to cover the Firehose, Blobs, Account Hosting, and more.

Bluesky sent a representative to IETF 120 in Vancouver, Canada. We have started participating in the OAuth and DNSOP working groups, and have been discussing timelines and strategy for the standards process with members of the community.

DASL is an independent effort to define a coherent subset of the IPLD specifications which projects can build upon without the full complexity and large implementation surface of those systems. These align very tightly with the atproto data model! It should be possible for atproto software to build on top of DASL implementation libraries (which have fewer dependencies than full IPLD implementations), and there is potential for collaboration within formal standards bodies.

lexicon.community is an independent effort to develop reusable atproto Lexicon schemas in a collaborative manner. They have a defined governance and contribution model, and an initial schema declared for bookmarks.

What Else?​

A few important areas did not see much change in 2024. The PLC identity system has been operating reliably, but the plc.directory service needs to be decentralized through technical and governance improvements. Protocol features to support private content in the network continue to be a top external request, and are likewise a priority within the team. E2EE DMs are the planned successor to the initial DM system, but we have not begun work on them.

All the above are important and will take time to get right. We will share a forward-looking protocol roadmap in the near future, covering these projects, decentralization efforts, and more.

In the meanwhile, we encourage you to check in on both official and community channels for updates on AT Protocol. An increasing share of development is happening out in the ecosystem, with more projects and organizations getting started by the week:

• @atproto.com account on Bluesky
• Discussions on Github
• The Fediverse Report includes weekly AT Protocol coverage
• atprotocol.dev runs Tech Talks

Update January 2025: Many of the operational changes described here were not implemented. We will give an update on Relay infrastructure early this year.

Summary: We are making a couple of changes to the Bluesky relay servers this week. The wire protocol and semantics are not impacted, but the changes may impact firehose consumers:

• One way or another, the event stream sequence will change for all subscribers. Most consumers can probably just reset to the new cursor; see details below.
• The bsky.network hostname will be swapped to a new relay instance (with new sequence)

The atproto network is growing rapidly! This is exciting and positive, but it means we need to move up some of our network scaling plans. One particular resource bottleneck is the relay firehose. We are seeing sustained traffic of over 2,000 events per second, and we have hundreds of active consumers. Multiplied, the overall throughput is hard for any one server to keep up with.

We saw this problem coming from the beginning and we have several plans and options to mitigate it. Details for each of these are linked or in sections below.

First, we are simply upgrading the size of server that we run our primary relay instances on. We expect to do this in coming days. The main impact here will be a reset of the firehose sequence (including cursor values), which will impact all downstream consumers.

Second, we are introducing the concept of "firehose fan-out" services, and releasing our implementation Rainbow. These are servers which re-broadcast the event stream firehose to many clients, reducing the bandwidth load on relays themselves. Rainbow is now in production, as an implementation detail.

Third, we recently released Jetstream as a more informal and light-weight option for event stream consumption. We encourage developers to check out Jetstream, and switch over if it works for their use case.

We encourage folks to take advantage of Rainbow and Jetstream. You can run your own, or make use of our hosted instances. Some additional longer-term options are discussed at the bottom of this article.

We don't have a hard time or date estimate for the cursor sequence change, but we expect it this week, possibly tomorrow (Tuesday). We generally try to give more time and make changes less disruptive but we're not able to do so in this situation. Please follow the @atproto.com account, this blog, and in Github Discussions for announcements.

Relay Upgrade​

The current Bluesky relay instance is available at wss://bsky.network. A more specific hostname for this relay instance will soon be relay1.us-west.bsky.network (not yet configured).

Instead of upgrading the relay in-place, we are going to start a new relay from scratch. This means that event stream ordering will be different and sequence numbers will not align with the current relay. This new instance will soon run at relay2.us-west.bsky.network (not yet configured).

There will be at least a short window when both relays are running, so downstream consumers can update their services to "cut over". At some point soon after the new relay is deployed, we will switch the generic bsky.network hostname to point at it.

What firehose consumers need to do about the cut over depends on how careful they need to be about processing every single event on the firehose.

Services which always connect to the firehose from the current offset don’t need to do anything. This includes developer tools like goat, and real-time sampling like Firesky.

Best-effort consumers can likely be configured to consume from the current firehose offset, and simply restart, missing at most a few seconds of events. This probably makes sense for feed generators, analytics, automated labeling, etc.

Services which want to really minimize the number of missed events can take a different approach. They can be programmed to connect to the new relay with no cursor, detect the current sequence number, then reconnect with a cursor a couple of minutes back (at current rates this means a few hundred thousand events back). This results in a time overlap window between the two relays (with some events processed twice), which helps minimize the chance of missing any events.

Firehose Fan-out and Rainbow​

Firehose fan-out servers have a single upstream connection to a relay firehose, and re-broadcast that stream to multiple subscribing clients. They maintain a local backfill window (allowing re-connection with a cursor), but do not implement endpoints like getRecord or getRepo, only com.atproto.sync.subscribeRepos. They do not re-validate the event stream (eg, they don't verify signatures). They maintain the exact sequence numbers of the upstream relay, meaning that they are interchangeable with the relay and with sibling fan-out servers, and can be placed behind a load-balancer.

Today we are releasing Rainbow, a fan-out service implemented in Go. The source code is available in the indigo git repository, with the same MIT/Apache licensing as our other open source projects.

Rainbow is already running as part of our production relay deployment (wss://bsky.network), with our load balancer (haproxy) distributing WebSocket subscriptions to a separate server running Rainbow.

In the future, we may run additional Rainbow instances outside our primary data centers, similar to how we offer Jetstream instances today. This distributes network traffic over more routers, and could improve connection reliability and throughput for subscribers in regions outside North America.

Going forward, we recommend that software which consumes from firehoses (including atproto SDKs) support HTTP redirects for WebSocket connections. This enables "pooling" behavior where a single hostname could route clients to multiple distinct servers, without use of a load-balancer.

What Else?​

As a reminder, the relay synchronization API is the same as the PDS synchronization API, including the firehose. Relays are essential to the functionality of the atproto network: they are an operational and developer convenience, allowing consuming services to skip keeping track of which PDS instances are active.

A classic way to scale services like the firehose is sharding, where the stream is split into multiple parallel streams. atproto has a natural sharding key (the account DID), which means related events can be consistently routed to the correct shard. This will allow near-indefinite scaling of the firehose event rate, assuming compute and network resources are available.

Part of what makes relays difficult to operate at scale is that they function as both a "relay" (rebroadcasting events from PDS instances) and a full-network mirror (storing all repo contents). In the current protocol, it is necessary to combine both functions to fully verify repository operations, especially deletion of events. We think that clever implementations could make this work less resource intensive (for example, storing just record CIDs instead of full data). It is also possible to validate most aspects of the event stream without a full copy of the repo tree, and there could be a role in the network for "non-mirroring" relays.

Lastly, there is a particular need for efficient consumption and backfill of full-network content for only content and accounts which make use of specific record types. For example, “only whtwnd blog post records”, or “only accounts with labeling service declarations”. We are planning new features to make this type of network subscription and backfill much more efficient.

One of most popular aspects of atproto for developers is the firehose: an aggregated stream of all the public data updates in the network. Independent developers have used the firehose to build real-time monitoring tools (like Firesky), feed generators, labeling services, bots, entire applications, and more.

But the firehose wire format is also one of the more complex parts of atproto, involving decoding binary CBOR data and CAR files, which can be off-putting to new developers. Additionally, the volume of data has increased rapidly as the network has grown, consistently producing hundreds of events per second.

The full synchronization firehose is core network infrastructure and not going anywhere, but to address these concerns we developed an alternative streaming solution, Jetstream, which has a few key advantages:

• simple JSON encoding
• reduced bandwidth, and compression
• ability to filter by collection (NSID) or repo (DID)

A Jetstream server consumes from the firehose and fans out to many subscribers. It is open source, implemented in Go, simple to self-host. There is an official client library included (in Go), and community client libraries have been developed.

Jetstream was originally written as a side project by one of our engineers, Jaz. You can read more about their design goals and efficiency gains on their blog. It has been successful enough that we are promoting it to a team-maintained project, and are running several public instances:

• jetstream1.us-east.bsky.network
• jetstream2.us-east.bsky.network
• jetstream1.us-west.bsky.network
• jetstream2.us-west.bsky.network

You can read more technical details about Jetstream in the Github repo.

Why Now?​

Why are we promoting Jetstream at this time?

Two factors came to a head in early September: we released an example project for building new applications on atproto (Statusphere), and we had an unexpectedly large surge in traffic in Brazil. Suddenly we had a situation where new developers would be subscribing to a torrential full-network firehose (over a thousand events per second), just to pluck out a handful of individual events from a handful of accounts. Everything about this continued to function, even on a laptop on a WiFi connection, but it feels a bit wild as an introduction to the protocol.

We knew from early on that while the current firehose is extremely powerful, it was not well-suited to some use cases. Until recently, it hadn’t been a priority to develop alternatives. The firehose is a bit overpowered, but it does Just Work.

Has the Relay encountered scaling problems or become unaffordable to operate?

Nope! The current Relay implementation ('bigsky', written in Go, in the indigo git repo) absorbed a 10x surge in daily event rate, with over 200 active subscribers, and continues to chug along reliably. We have demonstrated how even a full-network Relay can be operated affordably.

We do expect to refactor our Relay implementation and make changes to the firehose wire format to support sharding. But the overall network architecture was designed to support global scale and millions of events per second, and we don't see any serious barriers to reaching that size. Bandwidth costs are manageable today. At larger network size (events times subscribers), bandwidth will grow in cost. We expect that the economic value of the network will provide funding and aligned incentives to cover the operation of core network infrastructure, including Relays. In practical terms, we expect funded projects and organizations depending on the firehose to pay infrastructure providers to ensure reliable operation (eg, an SLA), or to operate their own Relay instances.

Tradeoffs and Use Cases​

Jetstream has efficiency and simplicity advantages, but they come with some tradeoffs. We think it is a pragmatic option for many projects, but that developers need to understand what they are getting into.

Events do not include cryptographic signatures or Merkle tree nodes, meaning the data is not self-authenticating. "Authenticated Transfer" is right in the AT Protocol acronym, so this is a pretty big deal! The trust relationship between a Jetstream operator and a consuming client is pretty different from that of a Relay. Not all deployment scenarios and use-cases require verification, and we suspect many projects are already skipping that aspect when consuming from the firehose. If you are running Jetstream locally, or have a tight trust relationship with a service provider, these may be acceptable tradeoffs.

Unlike the firehose (aka, Repository Event Stream), Jetstream is not formally part of the protocol. We are not as committed to maintaining it as a stable API or critical piece of infrastructure long-term, and we anticipate adopting some of the advantages it provides into the protocol firehose over time.

On the plus side, Jetstream is easier and cheaper to operate than a Relay instance. Folks relying on Jetstream can always run their own copy on their own servers.

Some of the use cases we think Jetstream is a good fit for:

• casual, low-stakes projects and social toys: interactive bots, and "fun" badging labelers (eg, Kiki/Bouba)
• experimentation and prototyping: student projects, proofs of concept, demos
• informal metrics and visualizations
• developing new applications: filtering by collection is particularly helpful when working with new Lexicons and debugging
• internal systems: if you have multiple services consuming from the firehose, a single local Jetstream instance can be used to fan out to multiple subscribers

Some projects it is probably not the right tool for:

• mirroring, backups, and archives
• any time it is important to know "who said what"
• moderation or anti-abuse actions
• research studies

What Else?​

The ergonomics of working with the firehose and "backfilling" bulk data from the network are something we would like to improve in the protocol itself. This might include mechanisms for doing "selective sync" of specific collections within a repo, while still getting full verification of authenticity.

It would be helpful to have a mechanism to identify which repos in the network have any records of a specific type, without inspecting every account individually. For example, enumerating all of the labelers or feed generators in the network. This is particularly important for new applications with a small initial user base.

We are working to complete the atproto specifications for the firehose and for account hosting status.

As the AT Protocol matures, developers are building alternative Bluesky clients and entirely novel applications with independent Lexicions. We love to see it! This is very aligned with our vision for the ATmosphere, and we intend to encourage more of this through additional developer documentation and tooling.

One of the major components of the protocol is the concept of "Lexicons," which are machine-readable schemas for both API endpoints and data records. The goal with Lexicons is to make it possible for independent projects to work with the same data types reliably. Users should be able to choose which software they use to interact with the network, and it is important that developers are able to call shared APIs and write shared data records with confidence.

While the Lexicon concept has been baked into the protocol from the beginning, some aspects are still being finalized, and best practices around extensions, collaboration, and governance are still being explored.

A recent incident in the live network brought many of these abstract threads into focus. Because norms and precedent are still being established, we thought it would be good to dig into the specific situation and give some updates.

What Happened?​

On October 10, Bluesky released version 1.92 of our main app. This release added support for "pinned posts," a long-requested feature. This update added a pinnedPost field to the app.bsky.actor.profile record. This field is declared as a com.atproto.repo.strongRef, which is an object containing both the URL and a hash (CID) of the referenced data record.

📢 App Version 1.92 is rolling out now (1/5)Pinned posts are here! Plus lots of UI improvements, including new font options, and the ability to filter your searches by language.Open this thread for more details. 🧵

[image or embed]

— Bluesky (@bsky.app) Oct 10, 2024 at 3:24 PM

All the way back in April 2024, independent developers had already implemented pinned posts in a handful of client apps. They did so by using a pinnedPost field on the app.bsky.actor.profile record, as a simple string URL. This worked fine for several months, and multiple separate client apps (Klearsky, Tokimeki, and Hagoromo) collaborated informally and used this same extension of the profile record type.

やっていることは簡単で、 app.bsky.actor.profile に pinnedPost というカスタムフィールドを作り、これにポストのAT URIを設定しているだけ…なんですが getProfile がカスタムフィールドを返してくれない（それはそう）のがちょっとあれでまだ調整中です

— mimonelu 🦀 みもねる (@mimonelu.net) Apr 29, 2024 at 8:45 AM

One of the interesting dynamics was that multiple independent Bluesky apps were collaborating to use the same extension field.

Blueskyクライアントの一覧を更新しました！🆕Features!PinnedPost (3rd party non-official feature)・Klearsky・TOKIMEKI・羽衣-Hagoromo-

[image or embed]

— どるちぇ (@l-tan.blue) May 3, 2024 at 9:36 AM

Which all worked great! Until the Bluesky update conflicted with the existing records, causing errors for some users. Under the new schema, the previously-written records suddenly became "invalid". And new records, valid under the new schema, could be invalid from the perspective of independent software.

Analysis​

The issue with conflicting records  was an unintentional mistake on our part. While we knew that other apps had experimented with pinned posts, and separately knew that conflicts with Lexicon extension fields were possible in theory, we didn't check or ask around for feedback when updating the profile schema. While the Bluesky app is open-source and this new schema had even been discussed by developers in the app ahead of time, we didn't realize we had a name collision until the app update was shipped out to millions of users. If we had known about the name collision in advance, we would have chosen a different field name or worked with the dev community to resolve the issue.

There has not been clear guidance to developers about how to interoperate with and extend Lexicons defined by others. While we have discussed these questions publicly a few times, the specifications are somewhat buried, and we are just starting to document guidance and best practices.

At the heart of this situation is a tension over who controls and maintains Lexicions. The design of the system is that authority is rooted in the domain name corresponding to the schema NSID (in reverse notation). In this example, the app.bsky.actor.profile schema is controlled by the owners of bsky.app – the Bluesky team. Ideally schema maintainers will collaborate with other developers to update the authoritative schemas with additional fields as needed.

There is some flexibility in the validation rules to allow forwards-compatible evolution of schemas. Off-schema attributes can be inserted, ignored during schema validation, and passed through to downstream clients. Consequently it’s possible (and acceptable) for other clients to use off-schema attributes, which is the situation that happened here.

While this specific case resulted in interoperability problems, we want to point out that these same apps are separately demonstrating a strong form of interoperation by including data from multiple schemas (whtwnd.com, linkat.blue, etc) all in a single app. This is exactly the kind of robust data reuse and collaboration we hoped the Lexicon system would enable.

🌈 TOKIMEKI UPDATE!!!(Web/Android v1.3.5/iOS TF)🆕 プロフィール画面に Atmosphere スペースを追加！- AT Protocol では Bluesky 以外にも様々なサービスを自由に開発することができ、実際にいくつかの便利なサービスが公開されています。- ユーザーが利用しているBluesky以外のサービスへのリンクを見ることができます。- 現在は、Linkat (リンク集) と WhiteWind (ブログ) の2つに対応。- 設定→全般から非表示にできます。Web | Android

[image or embed]

— 🌈 TOKIMEKI Bluesky (@tokimeki.blue) Oct 10, 2024 at 10:50 PM

Current Recommendations​

What do we recommend to developers looking to extend record schemas today?

Our current recommendation is to define new Lexicons for "sidecar" records. Instead of adding fields to app.bsky.actor.profile, define a new record schema (eg com.yourapp.profile) and put the fields there. When rendering a profile view, fetch this additional record at the same time. Some records always have a fixed record key, like self, so they can be fetched with a simple GET. For records like app.bsky.feed.post, which have TID record keys, the sidecar records can have the same record key as the original post, so they also can be fetched with a simple GET. We use this pattern at scale in the bsky Lexicons with app.bsky.feed.threadgate, which extends the post schema, and allows data updates without changing the version (CID) of the post record itself.

There is some overhead to doing additional fetches, but these can be mitigated with caching or building a shim API server (with updated API Lexicions) to blend in the additional data to "view" requests. If needed, support could be improved with generic APIs to automatically hydrate "related records" with matching TIDs across collections in the same repository.

If sidecar records are not an option, and developers feel they must add data directly to existing record types, we very strongly recommend against field names that might conflict. Even if you think other developers might want to use the same extension, you should intentionally choose long unique prefixes for field names to prevent conflicts both with the "authoritative" Lexicon author, and other developers who might try to make the same extension. What we currently recommend is using a long, unique, non-generic project name prefix, or even a full NSID for the field name. For example, app.graysky.pinnedPost or grayskyPinnedPost are acceptable, but not pinnedPost or extPinnedPost.

While there has been some clever and admirable use of extension fields (the SkyFeed configuration mechanism in app.bsky.feed.generator records comes to mind), we don't see inserting fields into data specified by other parties as a reliable or responsible practice in the long run. We acknowledge that there is a demonstrated demand for a simple extension mechanism, and safer ways to insert extension data in records might be specified in the future.

Proposals and discussion welcome! There is an existing thread on Github.

Progress with Lexicons​

While not directly related to extension fields, we have a bunch of ongoing work with the overall system.

We are designing a mechanism for Lexicon resolution. This will allow anybody on the public internet to authoritatively resolve the schema for a given NSID. This process should not need to happen very often, and we want to incorporate lessons from previous live schema validation systems (including XML), but there does need to be a way to demonstrate authority.

We are planning to build an aggregator and automated documentation system for Lexicons, similar to package management systems like pkg.go.dev and lib.rs. These will make it easier to discover and work with independent Lexicons across the ATmosphere and provide baseline documentation of schemas for developers. They can also provide collective benefits such as archiving, flagging abuse and security problems, and enabling research.

We are writing a style guide for authoring Lexicons, with design patterns, tips and common gotchas, and considerations for evolution and extensibility.

The validation behaviors for the unknown and union Lexicon types have been clarified in the specifications.

The schema validation behavior when records are created at PDS instances has been updated, and will be reflected in the specifications soon (a summary is available).

Generic run-time Lexicon validation support was added to the Go SDK (indigo), and test vectors were added to the atproto interop tests repository.

Finally, an end-to-end tutorial on building an example app ("Statusphere") using custom Lexicons was added to the updated atproto documentation website.

Overall, the process for designing and publishing new schemas from scratch should be clearer soon, and the experience of finding and working with existing schemas should be significantly improved as well.

We are very happy to release the initial specification of OAuth for AT Protocol! This is expected to be the primary authentication and authorization system between atproto client apps and PDS instances going forward, replacing the current flow using App Passwords and createSession over time.

OAuth is a framework of standards under active development by the IETF. We selected a particular "profile" of RFCs, best practices, and draft specifications to preserve security in the somewhat unique atproto ecosystem. In particular, unlike most existing OAuth deployments and integrations, the atproto network is composed of many independent server instances, client apps, developers, and end users, who generally have not cross-registered their client software ahead of time. This necessitates both automated discovery of the user’s Authorization Server, and automated registration of client metadata with the server. In some ways this situation is closer to that between email clients and email providers than it is between traditionally pre-registered OAuth or OIDC clients (such as GitHub apps or "Sign In With Google"). This unfortunately means that generic OAuth client libraries may not work out-of-the-box with the atproto profile yet. We have built on top of draft standards (including "OAuth Client ID Metadata Document") and are optimistic that library support will improve with time.

We laid out an OAuth Roadmap earlier this summer, and are entering the "Developer Preview Phase". In the coming months we expect to tweak the specification based on feedback from developers and standards groups, and to fill in a few details. We expect the broad shape of the specification to remain the same, and encourage application and SDK developers to start working with the specification now, and stop using the legacy App Password system for new projects.

What is Ready Today?​

OAuth has been deployed to several components of the atproto network over the past weeks. The Bluesky-developed PDS implementation implements the server component (including the Authorization Interface), the TypeScript client SDK now supports the client components, and several independent developers and projects have implemented login flows. At this time the Bluesky Social app has not yet been updated to use OAuth.

We have a a few resources for developers working with OAuth:

• The TypeScript SDK (@atproto/api) has been updated to support OAuth. READMEs are available for a browser-specific package and a Node.js-specific package. There is also a browser-based example project. You can read more about upgrading in the Typescript API Package Auth Refactor blog post.
• OAuth Client Implementation Guide, a developer-oriented document.
• A Python (Flask) Web Service Demo which shows how to implement a client "the hard way", without using a supporting SDK. A live version of the demo is running at https://oauth-flask.demo.bsky.dev/ (it may not be operated indefinitely).
• The AT Protocol OAuth Specification, which is the authoritative reference.

The current OAuth profile does not specify how granular permissions ("scopes") work with atproto. Instead, we have defined a small set of "transitional" scopes which provide the same levels of client access as the current auth system:

• transition:generic the same level of permissions as an App Password
• transition:chat.bsky is an add-on (must be included in combination with transition:generic) which adds access to the chat.bsky.* Lexicons for DMs. Same behavior as an App Password with the DM access option selected.

Next Steps​

There are two larger components which will integrate OAuth into the atproto ecosystem:

• The first is to expand the PDS account web interface to manage active OAuth sessions. This will allow users to inspect active sessions, the associated clients, and to revoke those sessions (eg, "remote log out"). This is user interface work specific to the PDS implementation, and will not change or impact the existing OAuth specification.
• The second is to design an atproto-native scopes system which integrates with Lexicons, record collections, and other protocol features. This will allow granular app permissions in an extensible manner. This is expected to be orthogonal to the current OAuth specification, and it should be relatively easy for client apps to transition to more granular permissions, though it will likely require logout and re-authentication by users.

These are big priorities for user security, and the path to implementation and deployment is clear.

While that work is in progress, we are interested in feedback from SDK developers and early adopters. What pain points do you encounter? Are there requirements which could be relaxed without reducing user security?

The overall design of this OAuth profile is similar to that of other social web protocols, such as ActivityPub. There are some atproto-specific aspects, but we are open to collaboration and harmonization between profiles to simplify and improve security on the web generally.

Finally, a number of the specifications we adopt and build upon are still drafts undergoing active development. We are interested in feedback on our specification, and intend to work with standards bodies (including the IETF) and tweak our profile if necessary to ensure compliance with final versions of any relevant standards and best practices.

Today we are merging some changes to how the TypeScript @atproto/api package works with authentication sessions. The changes are mostly backwards compatible, but some parts are now deprecated, and there are some breaking changes for advanced uses.

The motivation for these changes is the need to make the @atproto/api package compatible with OAuth session management. We don't have OAuth client support "launched" and documented quite yet, so you can keep using the current app password authentication system. When we do "launch" OAuth support and begin encouraging its usage in the near future (see the OAuth Roadmap), these changes will make it easier to migrate.

In addition, the redesigned session management system fixes a bug that could cause the session data to become invalid when Agent clones are created (e.g. using agent.withProxy()).

New Features​

We've restructured the XrpcClient HTTP fetch handler to be specified during the instantiation of the XRPC client, through the constructor, instead of using a default implementation (which was statically defined).

With this refactor, the XRPC client is now more modular and reusable. Session management, retries, cryptographic signing, and other request-specific logic can be implemented in the fetch handler itself rather than by the calling code.

A new abstract class named Agent, has been added to @atproto/api. This class will be the base class for all Bluesky agents classes in the @atproto ecosystem. It is meant to be extended by implementations that provide session management and fetch handling. Here is the class hierarchy:

As you adapt your code to these changes, make sure to use the Agent type wherever you expect to receive an agent, and use the AtpAgent type (class) only to instantiate your client. The reason for this is to be forward compatible with the OAuth agent implementation that will also extend Agent, and not AtpAgent.

import { Agent, AtpAgent } from '@atproto/api'

async function setupAgent(service: string, username: string, password: string): Promise<Agent> {
  const agent = new AtpAgent({
    service,
    persistSession: (evt, session) => {
      // handle session update
    },
  })

  await agent.login(username, password)

  return agent
}

import { Agent } from '@atproto/api'

async function doStuffWithAgent(agent: Agent, arg: string) {
  return agent.resolveHandle(arg)
}

import { Agent, AtpAgent } from '@atproto/api'

class MyClass {
  agent: Agent

  constructor () {
    this.agent = new AtpAgent()
  }
}

Breaking changes​

Most of the changes introduced in this version are backward-compatible. However, there are a couple of breaking changes you should be aware of:

• Customizing fetch: The ability to customize the fetch: FetchHandler property of @atproto/xrpc's Client and @atproto/api's AtpAgent classes has been removed. Previously, the fetch property could be set to a function that would be used as the fetch handler for that instance, and was initialized to a default fetch handler. That property is still accessible in a read-only fashion through the fetchHandler property and can only be set during the instance creation. Attempting to set/get the fetch property will now result in an error.
• The fetch() method, as well as WhatWG compliant Request and Headers constructors, must be globally available in your environment. Use a polyfill if necessary.
• The AtpBaseClient has been removed. The AtpServiceClient has been renamed AtpBaseClient. Any code using either of these classes will need to be updated.
• Instead of wrapping an XrpcClient in its xrpc property, the AtpBaseClient (formerly AtpServiceClient) class - created through lex-cli - now extends the XrpcClient class. This means that a client instance now passes the instanceof XrpcClient check. The xrpc property now returns the instance itself and has been deprecated.
• setSessionPersistHandler is no longer available on the AtpAgent or BskyAgent classes. The session handler can only be set though the persistSession options of the AtpAgent constructor.
• The new class hierarchy is as follows:
  • BskyAgent extends AtpAgent: but add no functionality (hence its deprecation).
  • AtpAgent extends Agent: adds password based session management.
  • Agent extends AtpBaseClient: this abstract class that adds syntactic sugar methods app.bsky lexicons. It also adds abstract session management methods and adds atproto specific utilities (labelers & proxy headers, cloning capability) - AtpBaseClient extends XrpcClient: automatically code that adds fully typed lexicon defined namespaces (instance.app.bsky.feed.getPosts()) to the XrpcClient.
  • XrpcClient is the base class.

Non-breaking changes​

• The com.* and app.* namespaces have been made directly available to every Agent instances.

Deprecations​

• The default export of the @atproto/xrpc package has been deprecated. Use named exports instead.
• The Client and ServiceClient classes are now deprecated. They are replaced by a single XrpcClient class.
• The default export of the @atproto/api package has been deprecated. Use named exports instead.
• The BskyAgent has been deprecated. Use the AtpAgent class instead.
• The xrpc property of the AtpClient instances has been deprecated. The instance itself should be used as the XRPC client.
• The api property of the AtpAgent and BskyAgent instances has been deprecated. Use the instance itself instead.

Migration​

The @atproto/api package​

If you were relying on the AtpBaseClient solely to perform validation, use this:

import { AtpBaseClient, ComAtprotoSyncSubscribeRepos } from '@atproto/api'

const baseClient = new AtpBaseClient()

baseClient.xrpc.lex.assertValidXrpcMessage('io.example.doStuff', {
  // ...
})

import { lexicons } from '@atproto/api'

lexicons.assertValidXrpcMessage('io.example.doStuff', {
  // ...
})

If you are extending the BskyAgent to perform custom session manipulation, define your own Agent subclass instead:

import { BskyAgent } from '@atproto/api'

class MyAgent extends BskyAgent {
  private accessToken?: string

  async createOrRefreshSession(identifier: string, password: string) {
    // custom logic here

    this.accessToken = 'my-access-jwt'
  }

  async doStuff() {
    return this.call('io.example.doStuff', {
      headers: {
        'Authorization': this.accessToken && `Bearer ${this.accessToken}`
      }
    })
  }
}

import { Agent } from '@atproto/api'

class MyAgent extends Agent {
  private accessToken?: string
  public did?: string

  constructor(private readonly service: string | URL) {
    super({
      service,
      headers: {
        Authorization: () =>
          this.accessToken ? `Bearer ${this.accessToken}` : null,
      }
    })
  }

  clone(): MyAgent {
    const agent = new MyAgent(this.service)
    agent.accessToken = this.accessToken
    agent.did = this.did
    return this.copyInto(agent)
  }

  async createOrRefreshSession(identifier: string, password: string) {
    // custom logic here

    this.did = 'did:example:123'
    this.accessToken = 'my-access-jwt'
  }
}

If you are monkey patching the xrpc service client to perform client-side rate limiting, you can now do this in the FetchHandler function:

import { BskyAgent } from '@atproto/api'
import { RateLimitThreshold } from "rate-limit-threshold"

const agent = new BskyAgent()
const limiter = new RateLimitThreshold(
  3000,
  300_000
)

const origCall = agent.api.xrpc.call
agent.api.xrpc.call = async function (...args) {
  await limiter.wait()
  return origCall.call(this, ...args)
}

import { AtpAgent } from '@atproto/api'
import { RateLimitThreshold } from "rate-limit-threshold"

class LimitedAtpAgent extends AtpAgent {
  constructor(options: AtpAgentOptions) {
    const fetch = options.fetch ?? globalThis.fetch
    const limiter = new RateLimitThreshold(
      3000,
      300_000
    )

    super({
      ...options,
      fetch: async (...args) => {
        await limiter.wait()
        return fetch(...args)
      }
    })
  }
}

If you configure a static fetch handler on the BskyAgent class - for example to modify the headers of every request - you can now do this by providing your own fetch function:

import { BskyAgent, defaultFetchHandler } from '@atproto/api'

BskyAgent.configure({
  fetch: async (httpUri, httpMethod, httpHeaders, httpReqBody) => {

    const ua = httpHeaders["User-Agent"]

    httpHeaders["User-Agent"] = ua ? `${ua} ${userAgent}` : userAgent

    return defaultFetchHandler(httpUri, httpMethod, httpHeaders, httpReqBody)
  }
})

import { AtpAgent } from '@atproto/api'

class MyAtpAgent extends AtpAgent {
  constructor(options: AtpAgentOptions) {
    const fetch = options.fetch ?? globalThis.fetch

    super({
      ...options,
      fetch: async (url, init) => {
        const headers = new Headers(init.headers)

        const ua = headersList.get("User-Agent")
        headersList.set("User-Agent", ua ? `${ua} ${userAgent}` : userAgent)

        return fetch(url, { ...init, headers })
      }
    })
  }
}

The @atproto/xrpc package​

The Client and ServiceClient classes are now deprecated. If you need a lexicon based client, you should update the code to use the XrpcClient class instead.

The deprecated ServiceClient class now extends the new XrpcClient class. Because of this, the fetch FetchHandler can no longer be configured on the Client instances (including the default export of the package). If you are not relying on the fetch FetchHandler, the new changes should have no impact on your code. Beware that the deprecated classes will eventually be removed in a future version.

Since its use has completely changed, the FetchHandler type has also completely changed. The new FetchHandler type is now a function that receives a url pathname and a RequestInit object and returns a Promise<Response>. This function is responsible for making the actual request to the server.

export type FetchHandler = (
  this: void,
  /**
   * The URL (pathname + query parameters) to make the request to, without the
   * origin. The origin (protocol, hostname, and port) must be added by this
   * {@link FetchHandler}, typically based on authentication or other factors.
   */
  url: string,
  init: RequestInit,
) => Promise<Response>

A noticeable change that has been introduced is that the uri field of the ServiceClient class has not been ported to the new XrpcClient class. It is now the responsibility of the FetchHandler to determine the full URL to make the request to. The same goes for the headers, which should now be set through the FetchHandler function.

If you do rely on the legacy Client.fetch property to perform custom logic upon request, you will need to migrate your code to use the new XrpcClient class. The XrpcClient class has a similar API to the old ServiceClient class, but with a few differences:

• The Client + ServiceClient duality was removed in favor of a single XrpcClient class. This means that:
  • There no longer exists a centralized lexicon registry. If you need a global lexicon registry, you can maintain one yourself using a new Lexicons (from @atproto/lexicon).
  • The FetchHandler is no longer a statically defined property of the Client class. Instead, it is passed as an argument to the XrpcClient constructor.
• The XrpcClient constructor now requires a FetchHandler function as the first argument, and an optional Lexicon instance as the second argument.
• The setHeader and unsetHeader methods were not ported to the new XrpcClient class. If you need to set or unset headers, you should do so in the FetchHandler function provided in the constructor arg.

import client, { defaultFetchHandler } from '@atproto/xrpc'

client.fetch = function (
  httpUri: string,
  httpMethod: string,
  httpHeaders: Headers,
  httpReqBody: unknown,
) {
  // Custom logic here
  return defaultFetchHandler(httpUri, httpMethod, httpHeaders, httpReqBody)
}

client.addLexicon({
  lexicon: 1,
  id: 'io.example.doStuff',
  defs: {},
})

const instance = client.service('http://my-service.com')

instance.setHeader('my-header', 'my-value')

await instance.call('io.example.doStuff')

import { XrpcClient } from '@atproto/xrpc'

const instance = new XrpcClient(
  async (url, init) => {
    const headers = new Headers(init.headers)

    headers.set('my-header', 'my-value')

    // Custom logic here

    const fullUrl = new URL(url, 'http://my-service.com')

    return fetch(fullUrl, { ...init, headers })
  },
  [
    {
      lexicon: 1,
      id: 'io.example.doStuff',
      defs: {},
    },
  ],
)

await instance.call('io.example.doStuff')

If your fetch handler does not require any "custom logic", and all you need is an XrpcClient that makes its HTTP requests towards a static service URL, the previous example can be simplified to:

import { XrpcClient } from '@atproto/xrpc'

const instance = new XrpcClient('http://my-service.com', [
  {
    lexicon: 1,
    id: 'io.example.doStuff',
    defs: {},
  },
])

If you need to add static headers to all requests, you can instead instantiate the XrpcClient as follows:

import { XrpcClient } from '@atproto/xrpc'

const instance = new XrpcClient(
  {
    service: 'http://my-service.com',
    headers: {
      'my-header': 'my-value',
    },
  },
  [
    {
      lexicon: 1,
      id: 'io.example.doStuff',
      defs: {},
    },
  ],
)

If you need the headers or service url to be dynamic, you can define them using functions:

import { XrpcClient } from '@atproto/xrpc'

const instance = new XrpcClient(
  {
    service: () => 'http://my-service.com',
    headers: {
      'my-header': () => 'my-value',
      'my-ignored-header': () => null, // ignored
    },
  },
  [
    {
      lexicon: 1,
      id: 'io.example.doStuff',
      defs: {},
    },
  ],
)

We’re launching microgrants for labeling services on Bluesky!

Moderation is the backbone of healthy social spaces online. Bluesky has our own moderation team dedicated to providing around-the-clock coverage to uphold our community guidelines, and additionally, we recognize that there is no one-size-fits-all approach to moderation. No single company can get online safety right for every country, culture, and community in the world. So we’ve also been building something bigger — an ecosystem of moderation and open-source safety tools that gives communities power to create their own spaces, with their own norms and preferences.

Labeling services on Bluesky allow users and communities to participate in a stackable ecosystem of services. Users can create and subscribe to filters from independent moderation services, which are layered on top of Bluesky’s own service. You can read more about how stackable moderation works on Bluesky here.

To support the first labelers in our ecosystem and encourage more, we are launching a microgrants program for labeling services. To apply for a grant, please fill out this form.​

Program Details​

For this program, we have an allocation of $10,000. We will be distributing $500 per labeling service that is approved for a grant. Please submit an application here.

The application has a rolling deadline, and we will announce both the recipients of the grants and when all of the grants have been distributed. We pay out grants via public GitHub Sponsorships.

In addition, we’ve also partnered with Amazon Web Services (AWS) to offer $5,000 in AWS Activate¹ credits to labeling services as well. These credits are applied to your AWS bill to help cover costs from cloud services, including machine learning, compute, databases, storage, containers, dev tools, and more. Simply check a box in your grant application if you’re interested in receiving these credits as well.

If you’re an organization interested in running a labeler but do not currently have the technical capacity to implement one, please reach out to our team at partnerships@blueskyweb.xyz. We may be able to assist in matching you with a developer.

Initial Labeling Grant Recipients​

We're kicking off the program with grants to three initial recipients:

XBlock

@XBlock.aendra.dev is an attempt to help give users control over the types of content they see on Bluesky. Screenshots serve a variety of uses on social media, but quite often are intended to create discourse or drive dogpiles. By letting users toggle the visibility of screenshots from various platforms, XBlock aims to give users a "volume dial" for certain types of content.

Aegis

@aegis.blue is a volunteer-run labeling service providing community moderation predominantly to Bluesky's LGBTQIA+ and marginalized users. Featuring a diverse team of both industry and aspiring experts, Aegis lives by the motto, We Keep Us Safe. More info can be found on their website at https://aegis.blue/Home.

News Detective

News Detective fights misinformation by combining the experience of professional factcheckers with the wisdom of crowds. A crowd of volunteer factcheckers transparently investigates posts, and professional factcheckers make sure only the highest quality factchecks make it through the system. Users who use News Detective will be able to see factchecks (including explanations and sources) on posts they come across and request factchecks on posts they find questionable. They can also watch News Detectives discuss the posts and even participate in factchecking to create a more honest, democratic, and transparent internet. Incubated at MIT DesignX, MIT Sandbox, and HacksHackers.

Contact​

Please feel free to leave questions or comments on the GitHub discussion for this announcement here, or on the Bluesky post here.

Discuss this post in our Github Discussion forums here

This roadmap is an update on our progress and lays out our general goals and focus for the coming months. This document is written for developers working on atproto clients, implementations, and applications (including Bluesky-specific projects). This is not a product announcement: while some product features are hinted at, we aren't promising specific timelines here. As always, most Bluesky software is free and open source, and observant folks can follow along with our progress week by week in GitHub.

In the big picture, we made a lot of progress on the protocol in early 2024. We opened up federation on the production network, demonstrated account migration, specified and launched stackable moderation (labeling and Ozone), shared our plan for OAuth, specified a generic proxying mechanism, built a new API documentation website (docs.bsky.app), and more.

After this big push on the protocol, the Bluesky engineering team is spending a few months catching up on some long-requested features like GIFs, video, and DMs. At the same time, we do have a few "enabling" pieces of protocol work underway, and continue to make progress towards a milestone of protocol maturity and stability.

Summary-level notes:

• Federation is now open: you don't need to pre-register in Discord any more. 
• It is increasingly possible to build independent apps and integrations on atproto. One early example is https://whtwnd.com/, a blogging web app built on atproto.
• The timeline for a formal standards body process is being pushed back until we have additional independent active projects building on the protocol.

Current Work​

Proxying of Independent Lexicons: earlier this year we added a generic HTTP proxying mechanism, which allows clients to specify which onward service (eg, AppView) instance they want to communicate with. To date this has been limited to known Lexicons, but we will soon relax this restriction and make arbitrary XRPC query and procedure requests. Combined with allowing records with independent Lexicon schemas (now allowed), this finally enables building new independent atproto applications. PR for this work

Open Federation: the Bluesky Relay service initially required pre-registration before new PDS instances were crawled. This was a very informal process (using Discord) to prevent automated abuse, but we have removed this requirement, making it even easier to set up PDS instances. We will also bump the per-PDS account limits, though we will still enforce some limits to minimize automated abuse; these limits can be bumped for rapidly growing communities and projects.

Email 2FA: while OAuth is our main focus for improving account security (OAuth flows will enable arbitrary MFA, including passkeys, hardware tokens, authenticators, etc), we are rapidly rolling out a basic form of 2FA, using an emailed code in addition to account password for sign-in. This will be an optional opt-in functionality. Announcement with details

OAuth: we continue to make progress implementing our plan for OAuth. Ultimately this will completely replace the current account sign-up, session, and app-password API endpoints, though we will maintain backwards compatibility for a long period. With OAuth, account lifecycle, sign-in, and permission flows will be implementation-specific web views. This means that PDS implementations can add any sign-up screening or MFA methods they see fit, without needing support in the com.atproto.* Lexicons. Detailed Proposal

Product Features​

These are not directly protocol-related, but are likely to impact many developers, so we wanted to give a heads up on these.

Harassment Mitigations: additional controls and mechanisms to reduce the prevalence, visibility, and impact of abusive mentions and replies, particularly coming from newly created single-purpose or throw-away accounts. May expand on the existing thread-gating and reply-gating functionality.

Post Embeds: the ability to embed Bluesky posts in external public websites. Including oEmbed support. This has already shipped! See embed.bsky.app

Basic "Off-Protocol" Direct Messages (DMs): having some mechanism to privately contact other Bluesky accounts is the most requested product feature. We looked closely at alternatives like linking to external services, re-using an existing protocol like Matrix, or rushing out on-protocol encrypted DMs, but ultimately decided to launch a basic centralized system to take the time pressure off our team and make our user community happy. We intend to iterate and fully support E2EE DMs as part of atproto itself, without a centralized service, and will take the time to get the user experience, security, and privacy polished. This will be a distinct part of the protocol from the repository abstraction, which is only used for public content.

Better GIF and Video support: the first step is improving embeds from external platforms (like Tenor for GIFs, and YouTube for video). Both the post-creation flow and embed-view experience will be improved.

Feed Interaction Metrics: feed services currently have no feedback on how users are interacting with the content that they curate. There is no way for users to tell specific feeds that they want to see more or less of certain kinds of content, or whether they have already seen content. We are adding a new endpoint for clients to submit behavior metrics to feed generators as a feedback mechanism. This feedback will be most useful for personalized feeds, and less useful for topic or community-oriented feeds. It also raises privacy and efficiency concerns, so sending of this metadata will both be controlled by clients (optional), and will require feed generator opt-in in the feed declaration record.

Topic/Community Feeds: one of the more common uses for feed generators is to categorize content by topic or community. These feeds are not personalized (they look the same to all users), are not particularly "algorithmic" (posts are either in the feed or not), and often have relatively clear inclusion criteria (though they may be additionally curated or filtered). We are exploring ways to make it easier to create, curate, and explore this type of feed.

User/Labeler Messaging: currently, independent moderators have no private mechanism to communicate with accounts which have reported content, or account which moderation actions have been taken against. All reports, including appeals, are uni-directional, and accounts have no record of the reports they have submitted. While Bluesky can send notification emails to accounts hosted on our own PDS instance, this does not work cross-provider with self-hosted PDS instances or independent labelers.

Protocol Stability Milestone​

A lot of progress has been made in recent months on the parts of the protocol relevant to large-scale public conversation. The core concepts of autonomous identity (DIDs and handles), self-certifying data (repositories), content curation (feed generators), and stackable moderation (labelers) have now all been demonstrated on the live network.

While we will continue to make progress on additional objectives (see below), we feel we are approaching a milestone in development and stability of these components of the protocol. There are a few smaller tasks to resolve towards this milestone.

Takedowns: we have a written proposal for how content and account takedowns will work across different pieces of infrastructure in the network. Takedowns are a stronger intervention that complement the labeling system. Bluesky already has mechanisms to enact takedowns on our own infrastructure when needed, but there are some details of how inter-provider takedown requests are communicated.

Remaining Written Specifications: a few parts of the protocol have not been written up in the specifications at atproto.com.

Guidance on Building Apps and Integrations: while we hope the protocol will be adopted and built upon in unexpected ways, it would be helpful to have some basic pointers and advice on creating new applications and integrations. These will probably be informal tutorials and example code to start.

Account and Identity Firehose Events: while account and identity state are authoritatively managed across the DID, DNS, and PDS systems, it is efficient and helpful for changes to this state to be broadcast over the repository event stream ("firehose"). The semantics and behavior of the existing #identity event type will be updated and clarified, and an additional #account event type will be added to communicate PDS account deletion and takedown state to downstream services (Relay, and on to AppView, feed generator, labelers, etc). Downstream services might still need to resolve state from an authoritative source after being notified on the firehose.

Private Account Data Iteration: the app.bsky Lexicons currently include a preferences API, as well as some additional private state like mutes. The design of the current API is somewhat error-prone, difficult for independent developers to extend, and has unclear expectations around providing access to service providers (like independent AppViews). We are planning to iterate on this API, though it might not end up part of the near-term protocol milestone.

Protocol Tech Debt: there are a few other small technical issues to resolve or clean up; these are tracked in this GitHub discussion

On the Horizon​

There are a few other pieces of protocol work which we are starting to plan out, but which are not currently scheduled to complete in 2024. It is very possible that priorities and schedules will be shuffled, but we mostly want to call these out as things we do want to complete, but will take a bit more time.

Protocol-Native DMs: as mentioned above, we want to have a "proper" DM solution as part of atproto, which is decentralized, E2EE, and follows modern security best practices.

Limited-Audience (Non-Public) Content: to start, we have prioritized the large-scale public conversation use cases in our protocol design, centered around the public data repository concept. While we support using the right tool for the job, and atproto is not trying to encompass every possible social modality, there are many situations and use-cases where having limited-audience content in the same overall application would be helpful. We intend to build a mechanism for group-private content sharing. It will likely be distinct from public data repositories and the Relay/firehose mechanism, but retain other parts of the protocol stack.

Firehose Bandwidth Efficiency: as the network grows, and the volume and rate of repository commits increases, the cost of subscribing to the entire Relay firehose increases. There are a number of ways to significantly improve bandwidth requirements: removing MST metadata for most use-cases; filtering by record types or subsets of accounts; batch compression; etc.

Record Versioning (Post Editing): atproto already supports updating records in repositories: one example is updating bsky profile records. And preparations were made early in the protocol design to support post editing while avoiding misleading edits. Ideally, it would also be possible to (optionally) keep old versions of records around in the repository, and allow referencing and accessing multiple versions of the same record.

PLC Transparency Log: we are exploring technical and organizational mechanisms to further de-centralize the DID PLC directory service. The most promising next step looks to be publishing a transparency log of all directory operations. This will make it easier for other organizations to audit the behavior of the directory and maintain verifiable replicas. The recent "tiling" transparency log design used for https://sunlight.dev/ (described here) is particularly promising. Compatibility with RFC 6962 (Certificate Transparency) could allow future integration with an existing ecosystem of witnesses and auditors.

Identity Key Self-Management UX: the DID PLC system has a concept of "rotation keys" to control the identity itself (in the form of the DID document). We would like to make it possible for users to optionally register additional keys on their personal devices, password managers, or hardware security keys. If done right, this should improve the resilience of the system and reduce some of the burden of responsibility on PDS operators. While this is technically possible today, it will require careful product design and security review to make this a safe and widely-adopted option.

Standards Body Timeline​

As described in our 2023 Protocol Roadmap, we hope to bring atproto to an existing standards body to solidify governance and interoperability of the lower levels of the protocol. We had planned to start the formal process this summer, but as we talked to more people experienced with this process, we realized that we should wait until the design of the protocol has been explored by more developers. It would be ideal to have a couple organizations with atproto experience collaborate on the standards process together. If you are interested in being part of the atproto standards process, leave a message in the discussion thread for this post, or email protocol@blueskyweb.xyz.

While there has been a flowering of many projects built around the app.bsky microblogging application, there have been very few additional Lexicons and applications built from scratch. Some of this stemmed from restrictions on data schemas and proxying behavior on the Bluesky-hosted PDS instances, only relaxed just recently. We hope that new apps and Lexicons will exercise the full capabilities and corner-cases of the protocol.

We will continue to participate in adjacent standards efforts to make connections and get experience. Bluesky staff will attend IETF 120 in July, and are always happy to discuss responsible DNS integrations, OAuth, and HTTP API best practices.