Background

A few weeks ago I got lightly involved in the community response to the supply chain attack against the Trivy vulnerability scanner. On March 19th, 2026 malicious versions of the Trivy binary and GitHub Action were published as part of a larger attack that has continued to expand in scope to include popular NPM packages, LiteLLM, etc in the following weeks.

When the Trivy attack was first discovered, initial triage by the community began as a thread on the (now deleted) discussion topic from the previous compromise of Trivy that had occurred in late February. As TeamPCP realized they had been discovered, they used their GitHub access to delete the entire discussion thread, presumably in an attempt to slow down incident response by impacted users/companies.

When I added a comment preserving the original IoCs on a new GitHub discussion, TeamPCP was so kind as to send me 1000+ spam comments/emails from compromised GitHub accounts:

Of course the Trivy incident has kicked off the same conversations as previous supply-chain attacks about dependency cooldowns, trusted publishers, immutable releases, etc. However, very little of the conversation has focused on what Aqua Security or the companies/users who unintentionally executed the malicious Trivy binary/GitHub Action (which included extensive credential harvesting malware) could have done to detect the attack sooner.

In particular, I’ve haven’t seen many mentions of my favorite tools, Canarytokens (aka Honeytokens). These are digital “tripwires” disguised as attractive targets such as fake files, API keys, or database entries that trigger an alert the moment a threat actor touches or opens them. They act as a proactive security alarm, notifying you of a breach by revealing exactly when and where an unauthorized user is snooping in your system.

Goals

There is already an excellent list of supported Canarytokens including AWS keys and Kubeconfig both of which were exfiltrated and later abused by TeamPCP. However, notably missing from this list of supported credentials (and from the various competitors) is a GitHub Canarytoken, the primary credential type used to stage the compromise of Trivy.

Our goal is to create an SSH key and API token for github.com and then leave it somewhere an attacker/malware is likely to stumble upon it, for example:

• Within a GitHub Actions secret with an interesting name like GH_ADMIN_TOKEN
• Inside the ~/.ssh directory on a self-hosted GitHub actions runner image
• A machine entry in the ~/.netrc file on a developer’s workstation
• Inside Kubernetes Secret/ConfigMap within a development environment

If an attacker attempts to use the token/key, it should trigger an alert as quickly as possible, capturing the source IP address, User-Agent header, etc. Ideally, it shouldn’t be possible for an attacker to identify these are canary credentials without first detonating them.

The implementation will look something like this:

• Identify the types of credentials that are best suited for canarytokens (i.e. read-only) and how they are provisioned (ideally they can be created programmatically).
• Create fake resources (repositories, issues, pull-requests), accessible via the credentials, that will look attractive for an attacker to interact with.
• Configure audit logging using available features from the vendor, filter for events related to the canarytokens and trigger an alert, for example ‘paging’ via Slack.
• If the audit logs are incomplete/broken (foreshadowing), additional complexity/hacks may be required to alert on any usage…

Security log vs Audit log

If you’ve ever dug into your GitHub account settings (or had the misfortune of doing incident response on a GitHub org/user), you may already be familiar with the ‘Security log’ which records actions taken on a personal account, ex:

While there are quite a few different event types in this log, notably missing are any events for the use of an API token/SSH key, only the initial creation or deletion of credential resources will emit a ‘Security log’ event. Additionally, there is no API* for programmatically accessing ‘Security log’ events, only a manual export button in the GUI.

Of course this lack of visibility was not acceptable for ‘Enterprise’ customers so GitHub introduced a related feature: Audit logs. These logs track actions taken against an Organization instead of a User (more on the limitations of this later). ‘Audit logs’ are far more verbose than ‘Security logs’ and can be accessed programmatically*. In particular, we will be interested in the git.clone or git.fetch events which were introduced in December 2020, as well as the api.request event added in February 2023 which became generally available in January 2025.

Unfortunately, GitHub Audit logs are not free. In order to gain access the events we care about, our Organization(s) must be part of a GitHub Enterprise Cloud (GHEC) Enterprise. This will run a minimum of $21/month (after the 30 day trial expires) for the single admin user required to setup the pipelines/credentials:

Configuring and Streaming Audit Logs

Once our GitHub Enterprise is created, we need to configure the Audit Log settings which will be shared across every Organization in our Enterprise. In particular we need to enable source IP disclosure and API request events:

It’s worth noting that GitHub has some (frankly absurd) opinions about which audit events can contain source IPs. You cannot see source IPs for any activity involving a public repository (you know, the ones being targeted in all these supply chain attacks). Generally, it’s a reasonable stance that I shouldn’t get to see the source IP/actor behind some unassociated GitHub user performing a ‘git clone’ of my public repository.

However, refusing to include source IPs on write actions like a git.push to a protected branch is unreasonable. Especially when the user is inherently a member/admin of the organization, acting on a public repository owned by said organization. This will force us to exclusively use private/internal repositories as our targets for our canarytokens.

Although git audit log events can be obtained by polling the REST API, the api.request event is only available when using audit log streaming, largely due to the event volume. The audit log streaming feature supports a variety of destinations including AWS S3 buckets, Google Cloud Storage, Azure Event Hub, etc.

For the purposes of building a PoC, we want minimal setup/dependencies so I’m going to focus on the Splunk HTTP Event Collector (HEC) destination which can be configured with an arbitrary hostname/port. When events arrive, GitHub will emit them as an HTTP POST to the /services/collector POST endpoint:

POST /services/collector HTTP/1.1
Host: cf-github-canarytokens.bored-engineer.workers.dev:443
Accept-Encoding: gzip
Authorization: Splunk hunter2
Connection: close
Content-Length: 1337
Content-Type: application/json
User-Agent: Go-http-client/1.1

{"event":{...},"time":"1776463522.955"}{"event":{...},"time":"1776463524.123"}

We can pretend to be a compliant Splunk HEC collector by returning a successful JSON response (as well as answering health-check requests). Annoyingly, the events are delivered without any delimiter like a newline separating them which makes parsing using JSON.parse fail. I wrote a simple Cloudflare worker which handles this format and just logs each incoming event:

A few seconds later, audit events begin to arrive:

Organization and Repository Setup

With audit logs configured, we need to create our fake Organization under our Enterprise. I suggest a name that closely matches your actual public GitHub organization and repository which will appear enticing enough for an attacker to clone/access.

Next, we need to create an internal/private repository. I’d suggest populating the repository with some amount of content (ex: AI slop terraform) or at least a README.md so it’s not an empty/uninitialized repository:

Creating a SSH Deploy Key

Now that we have all the audit logging components setup, we can create our first canary credential, starting with an SSH key as they are the easier to reason about than API tokens. Specifically, deploy keys are an SSH key that is specific to an individual GitHub repository instead of a GitHub user and default to read-only access:

If we SSH to github.com using the newly created key (which can also be created via the REST API), the response will direct the attacker to our repository:

When we use this SSH key to clone the repository, we receive a git.clone audit event that includes the source IP and even the build/version of git used!

{
  "repository_public": false,
  "_document_id": "PHelWiQ6to2YlxnWOph03A==",
  "action": "git.clone",
  "actor": "deploy_key",
  "actor_ip": "203.0.113.37",
  "actor_location": {
    "country_code": "US"
  },
  "business": "canary-bored-engineer",
  "hashed_token": "+qycZFngYHBoZcx/+Ri5lIYs8FnI8t0DPZpEECxeuLc",
  "org": "contoso-private",
  "programmatic_access_type": "Public Key (User/Deploy)",
  "repo": "contoso-private/infra-secrets",
  "repository": "contoso-private/infra-secrets",
  "request_access_security_header": "",
  "request_id": "d389dcd282ff50415c7b41c0d1fd09ed",
  "transport_protocol_name": "ssh",
  "user": "",
  "user_agent": "git/2.53.0-Darwin",
  "@timestamp": 1776398159553,
  "actor_id": 0,
  "business_id": 588174,
  "org_id": 275559016,
  "repository_id": 1208913072,
  "token_id": 0,
  "transport_protocol": 2,
  "user_id": 0
}

What about User SSH Keys?

As a (more popular) alternative to deploy keys, every GitHub user can configure SSH keys associated with their account instead of a repository. While this can also be done programmatically via the REST API, these SSH keys are more powerful and are publicly accessible making them poorly suited for canarytokens:

Specifically, there are no fine-grained authorization controls for SSH keys on GitHub, every SSH key attached to your user will have full read/write access to every repository your GitHub user has access to (with the exception of organizations that enforce SSO which require a one-time approval/authorization before the SSH key can be used).

Additionally, it’s entirely possible for an attacker who has found an SSH private key to check if it’s valid and what GitHub user it corresponds to without actually using it. The SSH (public) keys for a user can be trivially accessed by adding ‘.keys’ to their GitHub URL, ex: https://github.com/bored-engineer.keys

This feature is what allows you to quickly add your SSH keys to a new Ubuntu install and what powers the wonderful whoami.filippo.io. Some secret scanners like trufflehog will do this out of the box. An educated attacker may be able to identify patterns in our ‘canary’ GitHub users, especially if they are being shared across multiple unrelated GitHub organizations. This violates our goal of making it difficult to check the validity of the credential without detonating it.

A final problem is discoverability, when an SSH key is found by a secret scanner, there’s nothing that will immediately lead the attacker to the canary repository where we get audit events for clones. Some tools like trufflehog will attempt to directly SSH to GitHub to obtain the username, but our attacker still won’t know the name of our private repository to clone and trigger an alert unlike with deploy keys:

As such, I would strongly suggest using deploy SSH keys instead of user SSH keys for canarytokens.

Personal Access Tokens

With SSH credentials figured out, we can move on to personal access tokens (PATs). These are a far more interesting target for attackers as it allows them to enumerate private repositories, pulls, issues, etc. However, as we’ll soon discover, they are quite a bit messier to build alerting for compared to SSH keys.

Unfortunately, one of the biggest limitations with PATs is they cannot be created programmatically (at least without screen scraping the main github.com UI). This will significantly limit the ability for commercial Canary vendors to offer the tokens as it requires manual steps by a human to provision.

Most GitHub API users are familiar with “classic” PATs, identified by the ‘ghp_’ prefix. These PATs use broad permissions such as ‘repo’ which grants read/write to all private repositories accessible to a user. Because this also permits dangerous actions like deleting the canary repository, I would not recommend classic PATs. With the said, it is possible to create “safer” classic PATs by using a dedicated GitHub user who has restricted permissions (read-only) for the canary GitHub organization.

Instead of classic PATs, fine-grained personal access tokens (identified by the github_pat_ prefix) can be used which grant permissions on specific GitHub repos:

We can then use this token via the REST API or GraphQL API to access any details about our canary repository (or clone the contents) the same way an attacker might if they had discovered it:

Shortly after, an audit event will be sent containing the source IP, User-Agent and normalized request URL/body:

{
  "actor_is_bot": false,
  "public_repo": false,
  "_document_id": "tWjPbax39pvd8MyeK-rZCA",
  "action": "api.request",
  "actor": "canary-bored-engineer",
  "actor_ip": "203.0.113.37",
  "actor_location": {
    "country_code": "US"
  },
  "business": "canary-bored-engineer",
  "hashed_token": "CUQ+aPiFMrcEOLXuuZ+0MDlCyaV5YXRoTkNgQMke5BE=",
  "operation_type": "access",
  "org": "contoso-private",
  "programmatic_access_type": "Fine-grained personal access token",
  "query_string": "",
  "repo": "contoso-private/infra-secrets",
  "request_body": "",
  "request_id": "A381:1F948A:17901F3:17F0724:69E1D1D7",
  "request_method": "GET",
  "route": "/repositories/:repository_id/pulls",
  "url_path": "/repositories/1208913072/pulls",
  "user": "canary-bored-engineer",
  "user_agent": "GitHub CLI 2.89.0",
  "user_programmatic_access_name": "ci-canary",
  "@timestamp": 1776407000157,
  "actor_id": 272395934,
  "business_id": 588174,
  "created_at": 1776407000157,
  "org_id": 275559016,
  "rate_limit_remaining": 4994,
  "repo_id": 1208913072,
  "status_code": 200,
  "token_id": 13603103,
  "user_id": 272395934
}

Cracks in API Audit Logs

Unfortunately, the gaps in GitHub audit logs begin to surface as we move on to PATs. Because audit logs are associated with a GitHub Organization, we will only see events related to Organization resources.

For example, the first request made using a compromised API token is typically the /user endpoint to obtain the id/login of the user (and associated permissions via the ‘x-oauth-scopes’ header). This is NOT an organization resource and as such no audit log event will be generated when this request is made.

What’s unexpected, is that some endpoints like /user/repos will list every repository a user has access to (including organization ones) without generating an audit log event. This is why it’s critical to create legitimate looking orgs/repositories that an attacker will be interesting in exploring further.

Interestingly, forked repositories are a weird middle-ground to this logic. If you fork a private organization repository into your personal account, it will continue to emit organization audit logs when you interact with the fork via git or the REST/GraphQL API.

As best I can tell, GitHub only emits api.request audit logs from the REST API when the URL path includes /repos/{owner}/{repo} or /orgs/{org} (where the organization has audit logs enabled). This means in addition to /user/repos, the entire /search family of endpoints can be used to access private organization code, commits, issues, pulls, etc using a basic org:contoso-private query without generating a single audit event.

Even within this simple logic, there appears to be gaps! Reading repository files via /repos/{owner}/{repo}/contents/{path}, does not generate any audit events, despite the tarball and zipball APIs right next to it doing so!

The GraphQL API is even worse, as long as you avoid directly returning a Repository or Organization object, you can perform all the queries in the world without generating audit events. For example, using the search query with any ISSUES or DISCUSSION type grants access to issues, PRs (a sub-type of Issue), discussions, etc.

With a bit of knowledge about how GitHub GraphQL node_id values are constructed, as well as a repo ID (ex: 12345) obtained from /user/repos, we can skip right past the Repository object. For example, the node_id of a git Ref is simply "REF_" + base64url(msgpack([0,12345,"refs/heads/main"]) which can be directly used in a node query to read repository contents:

query {
  node(id:"REF_kwDNMDmvcmVmcy9oZWFkcy9tYWlu") {
    ... on Ref {
      target {
        ... on Commit {
          file(path: "README.md") {
            object {
              ... on Blob {
                text
              }
            }
          }
        }
      }
    }
  }
}

These gaps extend to GraphQL mutations as well, such as updateRef which can be used to effectively force push branches/tags without any audit events (api.request or git.push) being emitted!

A draft of this post highlighting the gaps in audit logs was shared with GitHub on 4/19/2026 who stated they will take this research into account as they continue to build out the audit logging roadmap.

Building a (broken) Safety Net

Sadly, these sort of audit logging gaps are pretty common problem for multi-tenant SaaS providers. For example, in AWS there are multiple ways to test IAM credentials without generating a CloudTrail event (see Nick Frichette’s wonderful talk at fwd:cloudsec).

To solve this gap with AWS canarytokens, Thinkist developed the “AWS Safety Net” which periodically polls the credential report API that returns a last_used_date for each IAM access token and is always updated when a token is used. You won’t get access to the IP address/User-Agent, but at least you’ll know the credential was compromised wherever it was stored and can begin to roll incident response.

We can see in the GitHub UI that a similar “last used” timestamp is tracked for PATs and SSH keys:

After some digging, the raw timestamp is accessible via the /orgs/{org}/personal-access-tokens REST API which returns a token_last_used_at field. This API does come with a few restrictions, it only works when authenticated using a GitHub App installation and only returns fine-grained PATs (no classic PATs). The /user/keys and /repos/{owner}/{repo}/keys REST APIs can be used for similar purposes for SSH keys though it has slightly less value.

However, it seems the “last used” tracking for PATs is broken. For classic PATs the field is just never updated at all anymore, even when a token is actively being used to read (or even write) organization resources (hope you haven’t relied on that for any incident response!)

For fine-grained PATs, the behavior is inconsistently broken. Some critical actions such as git clone/git fetch/git push or using the previously mentioned contents REST API do not update the token_last_used_at field (the token will incorrectly show as “Never used” in the UI). Most other API calls do populate the field, however after it’s been populated it remains fixed at that initial timestamp.

What I suspect is happening here is that GitHub has implemented a roughly 1-week backoff for updates to this field, otherwise every API request (even read-only ones able to be served from cache) require a DB write. We can see this in the UI with the wording “Last used within the last week” rather than an exact timestamp, they’ve just forgotten that the raw timestamp is exposed via the API which appears to return per-second precision...

A slow descent into madness

At this point I was ready to give up, publish this blog post, and claim it was impossible to build the “perfect” GitHub canarytokens. At best we can use audit-logs to alert on most common actions, and maybe we can detect some other usage via the polling the “last used” for each credential with a granularity of ~1 week. To be clear, this is still pretty good and worth implementing, but we can do better….

In the middle of the night, the solution hit me: GitHub may not be willing to handle a DB write for every API request just for some “last used” field in the UI, but they can’t afford to ignore API abuse and scraping...

The GitHub REST and GraphQL APIs both have primary rate limits which limit the number of HTTP requests you can make per-hour. By default, this is 5,000 requests/points per hour for core/graphql resources. Every API request will consume a rate-limit by at least 1 point.

We can use the raw/plaintext canarytoken PATs to frequently poll the /rate_limit REST API (which ironically does not consume any rate-limits) and compare with the last seen value. If the used field for any rate type (core, graphql, etc) increases, the token has been compromised and we need to trigger an alert! Of course in practice, there are few other edge cases to account for:

The rate limit resets every hour so there’s a small window where an attacker might use a token just before it resets and we’d never know as it’s still at 5,000 during the next poll. Thankfully, GitHub uses redis to store these rate-limits, populating an expires_at property with the timestamp when the limit will reset. If a PAT is never actually used, no redis key gets created, and no reset gets scheduled. You can see this if you use a brand-new PAT to poll /rate_limit, the reset timestamp is always 1 hour in the future, effectively a placeholder.

A really crafty attacker might think to abuse our PAT and then quickly use the /credentials/revoke API to invalidate it before our next scheduled poll of /rate_limit, preventing us from identifying the number of requests made. So we’ll need to alert if the API response indicates a credential was revoked as well.

Finally, rate-limits (for PATs) are per-user, not per-token which means if we have multiple tokens owned by the same GitHub user, we won’t be able to identify which specific token was compromised/used. This also makes any canarytokens owned by a “real” user who consumes API rate-limit normally (such as via the GitHub CLI) impractical.

Putting it all together into a PoC

To build a end-to-end PoC, I modified the previously mentioned Cloudflare worker to publish basic alerts to Slack using incoming webhooks:

GitHub - bored-engineer/cf-github-canarytokens: GitHub Canarytoken PoC

Alerting on specific audit events is as simple as filtering on the incoming event.hashed_token field for the known canarytoken credentials (specified via GITHUB_CREDENTIALS enviroment variable/secret):

To implement the “Safety Net” functionality I used Workers KV to track the last response from /rate_limit per each token. When the Cron Trigger runs (every 5 minutes) it alerts if the used field has increased for any resource type:

Revocation is also detected within the same process when polling the rate-limits fails:

I would love to see the Canarytoken vendors take this research and implement it into their products, if you’d like to do so and want to chat more, feel free to reach out on LinkedIn. For corporations that are already using the GitHub Audit Log streaming feature, it should be trivial to build similar alerting functionality directly in their SIEM with a small CronJob to implement the safety check functionality.

Recommendations for GitHub organizations

If you own a GitHub organization/enterprise, the visibility gaps in audit logs discussed in this post may have scared you (and they probably should!) but there are still some things you can do today to reduce risk:

• Enable audit log streaming on your enterprise including source IPs and API requests, even if it’s just going to an S3 bucket nobody looks at it, your incident response team will thank you later.
• Enforce the use of SSO on your GitHub organization, not just because SSO is good but because it forces an explicit authorization action by users to grant an SSH key/PAT access to your organization resources, instead of granting access implicitly. That way the PAT created for someone’s weekend project won’t have access to your organization resources.
• Enforce an IP allowlist for your organization from a set of known trusted VPN/corporate IPs. This is by-far the strongest control (and the most painful to rollout) as it will prevent stolen credentials (even if still valid) from being used by an attacker except on the intended systems where you (hopefully) have other visibility/alerting via EDR or related tooling.
• If you can, restrict access from personal access tokens to your organization resources. Blocking classic PATs and enforcing a maximum expiration (ex: 3 months) on fine-grained PATs is a great way to reduce risk if you can’t eliminate PATs altogether.
• If you use GitHub enterprise (on-prem), configure collection of the raw HTTP access logs in addition to native GitHub audit logs, it may prove critical during incident response.

If you are some random person in Nebraska who maintains a critical repository under your GitHub user, please consider moving it under an organization so you can gain access to the above controls.

My requests for GitHub

In the hopes someone who works on audit logs at GitHub has made it this far into the post, please consider my (roughly prioritized) list of feature requests/fixes:

• Resolve the gaps in api.request audit log events from the REST API when the response returns any organization-owned resources (ex: /user/repos, /search, /repos/{owner}/{repo}/contents/{path}, etc)
• Resolve the gaps in api.request audit log events from the GraphQL API when the response returns or modifies organization-owned resources (ex: node, nodes, resource queries or mutations that accept a node ID)
• Resolve the bugs that are preventing updates to the last_used fields for SSH keys and PATs.
  - For SSH keys this should include ssh git@github.com sessions used by secrets scanners that don’t invoke git-shell.
  - For API keys this should include the /user endpoint used by secret scanners.
  - If the ~1 week backoff for updates to this field is intentional/necessary, please document it clearly.
• Please reconsider your position on source IP disclosure for audit log events involving public repositories, in particular if the actor is a member of the organization and/or has any privileged access to the repository.
• Add more details to the git.{clone,fetch,push} audit events, in particular a list of OIDs being fetched/pushed could be incredibly helpful during incident response.
• Emit a git.push audit event when a repository is modified using the REST/GraphQL APIs such as via the updateRef mutation, or the /repos/{owner}/{repo}/contents/{path} endpoint.
• Consider adding an API for provisioning of fine-grained PATs, this would at least allow automated rotation for APIs where a GitHub app cannot be used (and it would allow automated canarytoken provisioning).
• Consider adding an audit log (streaming) feature for api.request and git.{clone,fetch,push} events by a GitHub app owners. This would allow app owners to pro-actively monitor for abuse of their GitHub app installation tokens/OAuth tokens (ex: the 2022 Heroku breach)
  - At the very least if a GitHub app has done the right thing and configured IP allowlisting, send an email/alert when an otherwise valid token is blocked due to an invalid source IP, indicating compromise.
• Consider allowing users to opt-in to audit logs as well (not just security logs). I don’t care if you have to charge money for it, or gate it behind owning repositories with enough stars, as we continue to see more supply chain attacks there’s too many high-value repositories owned by GitHub users instead of organizations.
• I’ve avoided mention of Enterprise Managed Users (EMUs) (which actually allows programatic access to security log events via audit log streaming) to reduce confusion and because they don’t add much additional value in the context of canarytokens. But they could! I would love to receive an api.request for every API call (regardless of if it targets an organization resource) when the actor is an EMU user and I think there’s a strong argument for supporting this.

If you’re at GitHub and want to chat more about these, please don’t hesitate to reach out!

Building GitHub Canarytokens: A rant about Audit Log gaps was originally published in bored.engineer on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background

Various Riot Games ‘League of Legends’ webpages need to access metadata about the currently logged-in player to properly function. Sometimes these webpages are not located on the same (sub)domain as account.leagueoflegends.com and therefore must access the information cross-origin.

These days, this would be accomplished via Cross-Origin Resource Sharing (CORS) and/or window.postMessage, however back in 2016 browser support was inconsistent, particularly if you needed to support users on much older browsers. Many companies at the time, including Riot Games, turned to a library called easyXDM:

easyXDM is a Javascript library that enables you as a developer to easily work around the limitation set in place by the Same Origin Policy, in turn making it easy to communicate and expose javascript API’s across domain boundaries. — easyXDM.net

This JavaScript library provides a normalized interface for cross-origin communication that is backed by different browser transports (examples include window.postMessage, Flash's LocalConnection, HashTransport, etc). The transport is selected based on which is “best” supported by the user’s browser.

When using easyXDM, there is a producer and a consumer webpage. The producer page exports one or more JavaScript functions which can then be invoked (from another origin) by the consumer page which receives the result. For Riot Games the producer was account.leagueoflegends.com/pm.html which exported the following methods:

• send: A wrapper function around jQuery.ajax allowing access to make requests/responses cross-origin
• get-cookies: Retrieves cookies by name from document.cookie
• set-cookies: Sets a cookie on the base domain via document.cookie

Even on the surface, these functions appear quite dangerous so predictably there were some protections against access/abuse by arbitrary webpages:

When the pm.html page was loaded, the document.referrer was checked to verify the top level domain matched an allowlist:

• leagueoflegends.com
• riotgames.com
• lolesports.com
• pvp.net
• leagueoflegends.com.tr
• lolespor.com
• lolguilds.ru

Additionally, when receiving a cross-origin message from the easyXDM transport, the message origin (as reported by easyXDM) was checked against the same allowlist before executing the corresponding function.

A quick introduction to easyXDM

Before launching into the vulnerabilities, I need to take a moment to explain how easyXDM works and establish some terminology. easyXDM webpages obtain context about their configuration from a series of query parameters in the URL:

• xdm_e (config.remote): The URL to load if the current page is a consumer or the URL of the parent page if the current page is a producer
• xdm_c (config.channel): The channel to use when sending messages
• xdm_s (config.secret): The secret to use to validate both parties are known
• xdm_p (config.protocol): The id of which protocol transport to use for communication as defined in Core.js#L695

Because this context/configuration is obtained from the query parameters it is possible for a malicious actor to manipulate these values which I’ll need later.

Bypassing the Referrer Check

First I needed to bypass the referrer check. One way this could be accomplished is by posting a link on the boards.na.leagueoflegends.com forum (which matches the *.leagueoflegends.com referrer check) and hoping that a player clicks the link. However, this significantly limits the possible impact of any exploit as it would require specific user interaction.

A better exploit would be to utilize an open-redirect on any of the allowlisted domains. Unfortunately (or fortunately in this case for Riot) browsers have stopped carrying the Referrer header most on 301, 302, etc Location: based redirects. This leaves me looking for a JavaScript-based open redirect such as: window.location.href = "${open_redirect}";.

Thankfully, it is possible to abuse easyXDM to accomplish this. There is a vulnerability with option handling in FrameElementTransport.js. After the transport has loaded it will perform the following check:

Since document.referrer won't match the provided xdm_e parameter it will redirect the top window. This is intended to prevent spoofing origins when using the FrameElementTransport. I can force easyXDM to use this vulnerable transport by specifying a xdm_p parameter like this:

http://provider.easyxdm.net/current/example/remotetransport.html?xdm_e=https%3A%2F%2Fattackerdoma.in&xdm_c=channel&xdm_p=5

This method requires document.referrer to have a value for the payload to work. Since 2016, browsers have gotten more aggressive about removing referrers so this may no longer work in 2022, at least without further changes.

Great, now I have a JS-based open-redirect, however I can’t use it on pm.html because there is a catch-22 problem with the referrer checks. I needed to locate a different easyXDM consumer on one of the allowlisted domains to abuse. Thankfully there was another one at: apollo.na.leagueoflegends.com/apollo/cors/index.html.

Putting it all together, I have the following PoC which will load pm.html via the easyXDM open redirect on apollo.na.leagueoflegends.com, bypassing the referrer check:

Bypassing Origin Check

Next, I needed to bypass the origin checks on each message. Since the origin is provided by easyXDM, I needed to identify another bug/vulnerability in one of the transport implementations.

Eventually, I found one in HashTransport which works by passing data from a child iFrame to a parent window via window.location.hash. It works a little something like this:

Because this implementation is quite a workaround, it is not possible for the parent page (subdomain1) to determine who updated location.hash, unlike other transports such as window.parent.postMessage which have a event.origin property. To get around this, easyXDM assumes all messages come from config.remote (see HashTransport.js).

This is great and exactly the type of bug I needed, however there is an issue with where our exploit code gets loaded in the chain. If I set config.remote to attackerdoma.in to trigger our malicious page to load, all messages will have a non-allowlisted origin. But if I set it to a webpage on an allowlisted domain, our own webpage will never be loaded leaving us with no way to exploit the bug...

Callback hell with iFrames:

Looping back around to our referrer bypass, I can use a very similar technique to get around this issue. I can’t use the exact same open redirect bug since it calls window.top.location (replacing the top level window) which would break our exploit chain. However, I can use one of the other protocols (in this case HashTransport again) to force the apollo consumer to frame our attacker page, like this:

This results in the following nested frame layout:

https://account.leagueoflegends.com/pm.html
└── https://apollo.na.leagueoflegends.com/apollo/cors/index.html
    └── https://attackerdoma.in/...

At which point my script on attackerdoma.in can send messages to the parent frame like this:

This approach isn’t perfect though: when pm.html wants to send a message back to apollo it will set location.hash for the second level iframe which I can't access from attackerdoma.in context leaving me blind; I can send messages but not receive the reply...

Bypassing VerifyBehavior.js:

To throw another roadblock into the mix, easyXDM has already considered this attack and created VerifyBehavior.js. In their own words:

This behavior will verify that communication with the remote end is possible, and will also sign all outgoing, and verify all incoming messages. This removes the risk of someone hijacking the iframe to send malicious messages.

The good news is the implementation is also vulnerable and only protects against a scenario where the second level iframe is replaced mid-communication, not replaced from the start. The implementation looks roughly like this:

By sending a message first to establish a value for theirSecret:

1_1,2_0_theirSecret

Then waiting a few hundred milliseconds before sending a second message the request will continue:

1_1,3_0_theirSecret_${encodeURIComponent(message_payload)}

Turning jQuery.ajax into XSS:

At this point I can call any of the exposed methods mentioned earlier, but not receive the result. This leaves me with the ability to set cookies and make XHR requests but not read the response which is not particularly impactful. Thankfully, I can use a little-known “feature” of jQuery to abuse the XHR method. When calling jQuery.ajax if the url ends in =? jQuery will attempt to load the request as a JSONP call (even if dataType: "json" is set). You can test this out yourself (at least until jQuery v4):

So the crafted message payload to pm.html to trigger XSS becomes:

Putting it all together

At this point I have a rather complex chain of vulnerabilities:

1. Victim opens 85f32147–76c1–44e2–8aa8-a1f9fd8e2ed3.html
2. Redirect to https://apollo.na.leagueoflegends.com/apollo/cors/index.html occurs
3. Apollo redirects to https://account.leagueoflegends.com/pm.html with document.referrer set to a whitelisted domain
4. pm.html loads https://apollo.na.leagueoflegends.com/apollo/cors/index.html which is a whitelisted domain for messages
5. Apollo loads the nested frame of 8723f98f-e1f1–41cc-bc1e-f2afb4c8d933.html
6. 8723... sends a message to pm.html setting theirSecret for the session
7. 8723... sends a message to pm.html using theirSecret to call the send method
8. pm.html calls send which triggers a JSONP call to e26e42c0-08b7-4998-8e62-e9d8d6025d9e.js
9. XSS Payload fires

All of these vulnerabilities were privately reported to Riot Games with the above description and functional proof of concept.

Bypassing the mitigation

After some time, the Riot team indicated a partial mitigation had been rolled out, with more fixes on the way. This mitigation was an update to /apollo/cors/index.html to check if document.referrer is an allowlisted domain:

To bypass this, I needed to find another open-redirect from one of the allowlisted domains. Taking a look at the login process for leagueoflegends.com, the user is directed through an auth flow on auth.riotgames.com via login.riotgames.com using the following URL (assuming the user is in the NA region):

https://login.leagueoflegends.com/?region=na&lang=en_US&redirect_uri=http%3A%2F%2Fna.leagueoflegends.com%2F

While the redirect_uri parameter can't be manipulated to any URL making it an open-redirect, it could be modified to any subdomain of leagueoflegends.com, including apollo.na.leagueoflegends.com. This meant I could use this endpoint to redirect to the apollo page which will bypass the allowlist check like this:

https://login.leagueoflegends.com/?region=na&lang=en_US&redirect_uri=https%3A%2F%2Fapollo.na.leagueoflegends.com%2Fapollo%2Fcors%2Findex.html

Because this is abusing the login functionality, the victim has to be already logged-in to trigger the PoC.

This left me with a final vulnerability chain:

1. Victim opens a06250dd-ffd0–4a7e-8fb2-cf163021fe61.html
2. Redirect to https://login.leagueoflegends.com/
3. Redirect to https://auth.riotgames.com/authorize
4. Redirect to https://login.leagueoflegends.com/oauth2-callback
5. Redirect to https://login.lolesports.com/sso/login
6. Redirect to https://login.leagueoflegends.com/sso/callback
7. Redirect to https://apollo.na.leagueoflegends.com/apollo/cors/index.html occurs
8. Apollo redirects to https://account.leagueoflegends.com/pm.html with document.referrer set to a whitelisted domain
9. pm.html loads https://apollo.na.leagueoflegends.com/apollo/cors/index.html which is a whitelisted domain for messages
10. Apollo loads the nested frame of 8723f98f-e1f1–41cc-bc1e-f2afb4c8d933.html
11. 8723... sends a message to pm.html setting theirSecret for the session
12. 8723... sends a message to pm.html using theirSecret to call the send method
13. pm.html calls send which triggers a JSONP call to e26e42c0-08b7-4998-8e62-e9d8d6025d9e.js
14. XSS Payload fires

This bypass of the mitigation was also reported to Riot Games.

Accidentally Disclosing an 0-day

When drafting this post and verifying the original payloads in 2022, I realized that the latest release of easyXDM was still vulnerable to the message origin spoofing via HashTransport as well as the (limited) open-redirect vulnerabilities!

Before publishing these vulnerabilities publicly, I reached out to the easyXDM author to see if the project was still maintained and if a new security release would even make sense, they responded:

This project is not actively maintained as browsers have caught up to provide the core features natively.

And I would agree with this stance, the browser support for window.postMessage is pervasive and has been for 10+ years at this point. Newly developed software should be using the native window.postMessage functionality without compatibility shims like easyXDM, even if native postMessage still leaves plenty of room for its own different complex vulnerabilities.

Conclusion

This was a pretty fun chain of vulnerabilities to work on at the time, and really shows the hidden level of complexity you can inherit when you import seemingly straight-forward JavaScript compatibility shims.

P.S. Thanks to the Riot Games security team for giving me their blessing to publicly disclose this report when I reached out 6 years after the fact :)

Timeline

• November 26th, 2016: Initial report to Riot Games via HackerOne
• November 28th, 2016: Report acknowledged by triage team
• November 29th, 2016: Riot confirms successful reproduction
• February 21st, 2017: Riot indicates a patch was deployed
• February 21st, 2017: Response that I am still able to reproduce, presumably because the patch is not completely rolled out yet
• March 18th, 2017: Follow-up that I am still able to reproduce using the initial payload from the report
• April 28th, 2017: Riot apologies for the confusion, indicates the patch is currently deployed
• April 29th, 2017: Bypass of the patch is identified and reported to Riot
• May 8th, 2017: Riot acknowledges the bypass
• November 2nd, 2017: Follow-up on the report asking for updates
• November 6th, 2017: Riot acknowledges the ping and indicates the issue has been fixed, although one individual vulnerability is remaining (but the full chain is broken at this point)
• May 31st, 2018: Bounty ($2,000) awarded and report closed as resolved
• November 3rd, 2022: Email sent to easyXDM author
• November 19th, 2022: LinkedIn message to easyXDM author
• November 19th, 2022: Response from easyXDM author indicating the project is not actively maintained
• December 1st, 2022: Publication of this post

XSS on account.leagueoflegends.com via easyXDM [2016] was originally published in bored.engineer on Medium, where people are continuing the conversation by highlighting and responding to this story.

How I built a click farm to “bypass” Cloudflare’s CAPTCHA killer with some cheap USB security keys, an Arduino, and a bit of python.

Any opinions stated here are my own, not necessarily those of any past, present, or future employer.

What is Attestation of Personhood?

Cloudflare recently published a blog post about a potential replacement for CAPTCHAs by utilizing signatures from hardware security keys and WebAuthn they are calling “Attestation of Personhood”. The post triggered a good bit of discussion online, particularly around the threat of automation mentioned near the end of the post:

We also have to consider the possibility of facing automated button-pressing systems. A drinking bird able to press the capacitive touch sensor could pass the Cryptographic Attestation of Personhood. At best, the bird solving rate matches the time it takes for the hardware to generate an attestation. With our current set of trusted manufacturers, this would be slower than the solving rate of professional CAPTCHA-solving services, while allowing legitimate users to pass through with certainty.

After a bit of brainstorming and discussion on Slack, I decided it would be a fun weekend project to test this out with actual hardware and see just how difficult it would be.

Acquiring compatible hardware

Because WebAuthn is an open standard it’s of course trivial to build a software token and use it to sign requests. However, Cloudflare mitigates this by requiring a “hardware attestation” signature from specific manufacturers:

our initial rollout is limited to a few devices: YubiKeys, which we had the chance to use and test; HyperFIDO keys; and Thetis FIDO U2F keys.

This attestation is essentially a unique key per batch/manufacturer (minimum of 100k devices) baked into the device which (in theory) would be as difficult to extract as any other private keys from a device. I love my Yubikeys but there’s one thing you can’t deny, they’re expensive at $50+ apiece. Thankfully, HyperFIDO keys are much cheaper at $16 each, so I bought $100 worth of keys which were helpfully available with Amazon’s one-day shipping.

Prototyping with a SoloKey

With limited confidence in my soldering abilities, I wanted to prototype the software side of the equation while waiting for the keys to be delivered. Coincidentally, I recently acquired a SoloKey which is an open-source FIDO2 security key that allows the user to recompile and flash replacement user firmware. Predictably this key isn’t trusted by Cloudflare, but it will allow me to soft-disable the user presence test (physically pressing the key) required by the WebAuthn standard while still developing software that interacts with a physical USB-based key.

Thankfully this is a pretty trivial task, replacing the ctap_user_presence_test function with one that always succeeds. In fact, this is already implemented for use during testing via the “SKIP_BUTTON_CHECK_FAST” ifdef. Enabling it and compiling a new firmware release gets us a hardware token that signs any incoming WebAuthn request instantly (or as fast as the hardware will support).

Developing an actual server to expose the USB key and its operations to the internet is quite simple thanks to Yubico’s python-fido2 library, which abstracts the OS-specific USB interactions to a common interface. A basic HTTP server with no threading/locking/error handling can be implemented as follows:

Building an electronic version of the drinking bird

While experimenting with the SoloKey, the actual HyperFIDO keys arrived. Using some YouTube videos it was easy to get the keys apart and the internal circuit board exposed. Thankfully, going cheap on hardware actually paid off: instead of a capacitance-based presence detection like those seen on a Yubikey/SoloKey, they have a physical button that can be “easily” soldered for debugging.

Interestingly the button seems to be normally open at 3.3 volts and is brought down to 0 volts (ground) when pressed. The “proper” way to automate this would be to use a relay connected to either side of the button circuit allowing each button to be independently triggered without interference.

However, it took a solid hour to get all the keys disassembled and “reliably” soldered on one side, and I had no desire to do it twice. My hacky solution was to connect the wire to a GPIO pin on a Raspberry Pi (eventually replaced by an Arduino). By switching the pin between INPUT mode (floating) and OUTPUT_LOW (pull-down) it can cause each USB key to believe a button press has occurred. This works surprisingly well for a prototype but resulted in a few (in retrospect obvious) learnings:

• If the output is ever accidentally set to OUTPUT_HIGH it may damage/destroy the USB key and if the normal/input voltage is higher than expected (3.3v) it may damage the Raspberry Pi.
• It is critical that the ground between the USB key and Raspberry Pi/Arduino is tied together. The easiest way to do this is to power all devices from the same USB hub.

Pressing buttons is tricky

My initial button press script was quite primitive, triggering each button iteratively as fast as possible. However, as I decreased the interval between button presses I quickly ran into a problem: when a user presence test wasn’t pending, each press triggered the “slot 1” behavior on the USB key generating a 6 digit numeric TOTP token (‘888888’ by default). This meant each key was outputting useless key presses to the connected host at a rate that actually started to result in software problems, particularly on macOS where I was doing local development.

To resolve this issue, I replaced the Raspberry Pi with an unused Arduino Uno I had. This allowed me to simplify the setup (no systemd/python just to toggle GPIO pins) and allowed me to move from a loop to an event-driven architecture. By reading which key to “press” via the (USB) serial interface on the Arduino, I could trigger a button press only when necessary. Perhaps a better solution would have been to disable the “slot 1” TOTP behavior altogether (which is possible on a Yubikey), but I don’t believe it’s possible to reprogram this on the HyperFIDO keys.

Putting it all together

At this point, I have all the ingredients necessary to build an internet-facing web service with the ability to answer WebAuthn signing requests that “automatically” bypass the user presence test on-demand. With a few more modifications the server became multi-threaded and supported exclusive locking per device: github.com/bored-engineer/fido-farm. Benchmarking the server I got some surprising (at least to me) results:

The server is capable of reliably handling just under 28 signatures/sec, or roughly 4.6 requests per key/sec which is much better than I expected. However, it is roughly on par with the benchmarking of an individual key multiplied by 6 so it makes sense.

If an attacker has automated attacks (e.g. DDOS, mass goods purchasing, etc.) that need to bypass the attestation of personhood this is a reliable way to do it with relatively low cost and effort. For a few hours of work and a hundred dollars of hardware keys, an attacker could make a reusable system that could support theoretically limitless automated requests that could successfully bypass the challenge.

Does this mean “Attestation of Personhood” broken?

In my opinion, no. Starting with the obvious, Cloudflare has clearly considered this attack vector as they mentioned it in the post and decided it still raises the cost of an attack over the current CAPTCHA model:

Another issue that we keep a close eye on is security. The security of this challenge depends on the underlying hardware provided by trusted manufacturers. We have confidence they are secured. If any breach were to occur, we would be able to quickly deauthorize manufacturers’ public keys at various levels of granularity.

CAPTCHA solving services charge a variety of prices differentiating their services based on price-per-image, response time, and success rate. Of course, our solution is 100% reliable (until the batch of keys is banned) and fast (~430ms/request) but comes at a significant up-front cost. Assuming a similar price point as CAPTCHA solving services of 50 cents per 1000 challenge images, our service would need to answer over 24,000 challenges per key before getting banned/detected just to cover the cost of the key let alone the additional hardware/bandwidth. And given the probability that all of the hardware keys are from the same batch, this wouldn’t be an event that can be easily recovered by just swapping out an individual key.

Another aspect to consider is that challenge flows are often only one of many factors used when determining if a request should be allowed. Factors like the reputation of the source IP/network, capabilities of the browser (ex: is JavaScript enabled), user behavior on the site such as how quickly was a captcha solved or did the user actually click the element or trigger the function automatically can be used before a decision is made to allow/deny the request.

How could you detect this type of abuse?

Perhaps one of the reasons Cloudflare has confidence in this solution is brute-force hardware automation like the kind developed here is likely much easier to detect than you’d think, (and if not, here’s a free suggestion):

While an attestation certificate is shared amongst at least 100k devices, an additional attribute is provided as part of the attestation called a “signature counter”, this is a unique counter that is incremented each time an operation is performed:

Authenticators SHOULD implement a signature counter feature. These counters are conceptually stored for each credential by the authenticator, or globally for the authenticator as a whole. The initial value of a credential’s signature counter is specified in the signCount value of the authenticator data returned by authenticatorMakeCredential.

In the case of the HyperFIDO tokens, this counter is global which means it increments each time the crypto module is used. Unless the attestation key is extracted from the device allowing complete control over the signed authenticator data, this global counter will increment with every credential operation performed:

This type of unnatural/consistent growth within a batch of keys could be very easy for Cloudflare to detect if they chose to track the values across a batch of keys. Even a basic privacy-preserving detection blocking all requests (or sending to an alternate challenge) that denies requests when the signature counter is greater than a reasonable human threshold (ex: 20k) could be effective.

It will be interesting to see if over time Cloudflare finds it necessary to implement additional detections for this type of physical automation, as well as how quickly and how broadly they disable entire batches of keys when inevitably an attestation private key is successfully extracted in the future.

Takeaways

It is easy to build software that intercepts WebAuthn requests and sends them to a remote FIDO hardware key to be solved. With a bit of soldering, hardware FIDO keys can be modified so the user presence test (physically touching the key) is bypassed on-demand.

By combining these components, it is possible to automate Cloudflare’s Attestation of Personhood challenge. However, this comes at a significantly higher cost to attackers than CAPTCHAs and can be relatively easily detected (at least with the keys I tested).

Try it out yourself

For a final bit of fun, I’ve exposed this entire setup to the internet (via Cloudflare of course) and you can test it out yourself. Simply visit cloudflarechallenge.com and run the following in your browser’s developer console, then attempt to complete the challenge, it should be automatically solved by the hardware farm I set up:

Contributions and Thanks

A special thanks to those who helped peer review and make this post as useful as it is: Luke Matarazzo and James Chiappetta

Building a WebAuthn Click Farm — Are CAPTCHAs Obsolete? was originally published in better appsec on Medium, where people are continuing the conversation by highlighting and responding to this story.

linkedin/jaqen

DEF CON 25: Slides and Source Code was originally published in bored.engineer on Medium, where people are continuing the conversation by highlighting and responding to this story.

defcon24.pdf

Update on the slides, these issues have all been resolved, the slides were not updated before upload to the DEF CON server

• bored-engineer/ps-exploits
• bored-engineer/ps-splunk

DEF CON 24: Slides and Exploit was originally published in bored.engineer on Medium, where people are continuing the conversation by highlighting and responding to this story.

git init && git commit -a -m “Initial Commit” was originally published in bored.engineer on Medium, where people are continuing the conversation by highlighting and responding to this story.