Episode 058 Raw Data - Google Sheets

	A	B	C
1	Thanks for your interest in The Question! Sign up to participate in future questions here. This file is the raw data from Episode 58, deep dive hosted on July 24, 2025 with Ben Callahan and Adrianne Daley. There are _ answers. The question this week was: ----- Hello system thinkers! This week, we’re joined by Adrianne Daley, Staff Engineer on the Design Systems & Frontend Platform team at Honeycomb. Given that Adrianne works on a code instrumentation product, they have a unique perspective on how to capture and track really interesting metrics on design system usage at Honeycomb. This week, we want to explore how you are or are not doing the same. Let’s dive into the ways organizations measure deviations from their design system and what to do with that data. Welcome to Episode 058 of The Question! Are you tracking deviations from the design system in any way? If so, how do you do this? If not, what’s holding you back? Tell us something you’ve learned by gathering (or not gathering) this kind of metric.
2	Question 1: Are you tracking deviations from the design system in any way?	Question 2: If so, how do you do this? If not, what’s holding you back?	Question 3: Tell us something you’ve learned by gathering (or not gathering) this kind of metric.
3	Yes	Our big OKR for this year is actually to reduce deviations and inconsistencies. We started with discovery: in design, we're going beyond the (quite terrible) off the shelf Figma detachment analytics. We picked 4 of the strategic products and did a manual assessment based on our criteria, which resulted in a baseline score. In code, we have our proprietary code tracker, so it's easier to find those deviations (or even components that _could_ be using our DS).	We're analyzing the different types of deviations and categorizing them as "justified" vs. "unnecessary". We see a lot of inconsistency across the org, even in areas that have nothing to do with the DS, and should be table staked for designers (e.g. using a grid system).
4	Yes	lol its a google doc pls help but actually, holding us back from making something more robust is a true need for this. Were early on in our maturity and wed like to get reasonable adoption before we focus too much on deviation. at this point, we havent built things that theres a ton of decent reason to deviate from we are also tracking deviations in code pushup reports but the dots have not been connected to design so its not as useful yet. in some cases where we are opting to not migrate a component for now, but we want to come back to it, we are tagging it with a new class name so we can identify all the cases in our repo after were done with main integration of the DS	most of our deviations are just there bc as we integrate, there are other elements of the experience that need to shift for integration to actually happen. For example we have a case where someone is using what they called a "filter pill" as a floating action button. we are leaving this as a deviation bc in order to fix this, the experience needs to be properly thought through and discussed with the product design team. we want to move fast and clear the low hanging fruit first, so were tracking this as a deviation we will come back to after
5	No	We aren’t officially tracking this, other than me seeing stuff on the site and being sad. This would be nice to have, it’s just not at the top of our list of priorities. After talking about it for several years, we JUST got a tracker on Storybook that shows which pages on the site a component lives. Baby steps!	Well, when we’ve tried to update a component globally, it’s been a way bigger deal than we thought it would be because of all the variations introduced on the site. Even something seemingly small like updating the author bylines on articles has turned out to be a large effort.
6	No	Our Design system is still in beta due to some elements being incomplete.	I'm looking forward to learning more about measuring deviations in design systems so I can work towards possible implementation.
7	No	We have begun investigating how to do this. Our first step will be to track prop usage on components to get an indication of how customers are using our components and how they're being configured. This doesn't really answer the question of deviation, though.	We know deviation exists in our products, and I think we kind of understand why it exists; gaps in the system, cultural, people focused on solving for their silo, and not thinking about the bigger picture. What we don't yet fully understand is how to solve for this and what actions we need to take.
8	Yes	When designers need to break the DS, they put their designs into a custom components file and tracking the number of use cases.	Honestly not much. It’s self service and people don’t really remember to do it. There also isn’t anyone on the team taking ownership of monitoring it and identifying candidates for promotion into the DS.
9	No	So far, I haven't reached that point yet as I'm just starting out in design systems.	N/A
10	Yes	As far as I know, mostly by word-of-mouth or from meetings with Design Ambassadors. Not sure where this info is stored.	Helps us learn where we might need to support our products better by making new components/patterns, updating documentation, etc. The information is helpful, but gathering it is hard since people seem like they don't want us to know when/if they deviate.
11	Kind of.	We meet regularly with teams that are building deviations so we have an internal knowledge of who is doing what. It is not documented, it is more of an organic knowledge-base so many of the deviations get unnoticed because we have way too many teams to track.	Many of the deviations are based on taste and visual appearance from a designer. Many times the deviation happens because a designer doesn't know how to use Figma component properties. A big portion of such deviations could be avoided if someone would enforce the design system because such deviations don't have a good reason such as data, research or best practice to back up the need for it.
12	No	We’re currently in the process of auditing our live site to map all the components, colors, and typography being used. This foundational work is essential before we can meaningfully track deviations. While we’re not actively tracking deviations yet, it’s definitely on our roadmap. We plan to build an automation tool that can surface usage patterns and flag inconsistencies across our ecosystem. The goal is to eventually have a reliable way to monitor adherence and inform refactoring or education efforts where needed.	One thing we’ve learned by not yet gathering these metrics is how easy it is for design system drift to go unnoticed. Without visibility, it’s difficult to have productive conversations with product teams about consistency or technical debt. It’s also made it clear that intuition alone isn't enough to prioritize refactoring—quantifiable data would give us the leverage to advocate for systemic improvements. This gap has helped us recognize the need for instrumentation not just as a nice-to-have, but as a critical tool for design system governance.
13	No	We're still relatively new and are having difficulty tracking current usage in a meaningful way	We know that the DS doesn't serve all goals, but are not sure where to allow deviation and ownership of the deviation.
14	No	We find the analytics in Figma to be very lacking, and we don't have approval for additional paid plugins at this time.	We are currently retrofitting an existing system to be more scalable and the lack of clarity on which components are used where, when people are detaching or creating new version, etc., is definitely an issue.
15	No	well, we do look into which components are being broken in Figma. but not tracking the deviations, not following up on why/what was needed.	missing the opp of gathering enhancement needs (other than intake form - which is work for DS customers) tacking the figma detachments helps us understand which components should be prioritized for revisiting...like if they are broken constantly, something is wrong.
16	No	we are still tring to figure out the best way to do it	jury is still out for my org, this weeks question is super interesting
17	No	We heads down on rebuilding components and prepping for a major release for the system (restarting). Our team's band-width and relationship with designers "as the system police" is probably holding us back from doing this. We are reframing this mentality with the next release of the system.	I've learned it's going to be important to track this going forward to encourage designers to bend the system and use it how they need to in order to influence the system moving forward.
18	No	We still consider ourselves in the early stages and need to tackle tracking in general	N/A
19	No	I'm curious what constitutes a deviation? Is it extending the system in some unexpected way, adoption, using something instead of the system, or some other decision/outcome? We track and measure what we can about system use, but being that our codebase is rather large and covers several languages it would be a challenge to look at other metrics.	—
20	Yes	We track if teams are using other colors/fonts other than our tokens and we track CSS-overrides on components (not to police but rather to understand if we missed something).	We learn when/where the system is lacking but the lack hasn’t been significant enough to adjust it, hehe. So basically we have tracking we do not use.
21	Yes	We're currently tracking deviations by sitting in on design reviews with teams. We had explored using listeners on the Figma components but found that the overhead for our small team would be much larger than the value of the output.	Through partnership with design teams we have been able to offer education about how to use existing components and work through design challenges from the system perspective that avoids requests for new components. We are finding it challenging to quantify this metric right now though and am interested to hear how others are approaching it!
22	This isn't exactly a "not applicable", more of an "I'm not sure". Because of the structure of the system at this point, the directive is to just use things out of the box and future enhancements to the system will come based upon adoption. So the answer is sort of yes but sort of no, but I'm not close enough to be able to give the strongest of answers.	I think there would be a ton of benefit in taking some sort of an approach like what I'd assume will be outlined in this deep dive. I'd love to get some of my coworkers to this session to see if there are things we could learn.	A big issue with regards to pushback of "what's in the system" and "what the system provides" is teams that are performance based and the story they come with is that the design and deviations that they put forward are performing better and use that for justification for why they're not going to implement what's documented. The issue comes because this is a team that generates a ton of money and the design system doesn't; it gets a pass, but the people in charge of the system (not me) aren't happy about it. There are questions with regards to research and how those tests are set up (they seem to stack the deck a bit), but I feel like that's not unique to this team versus across the whole org. Everyone seems to do it.
23	Yes	Tracking ‘snowflakes’ that we asked product teams to register with us.	A lot of snowflakes and some are hidden - eg they look like design system components but are duplicates, often made before the design system was fully in place
24	Yes	Figma analytics. We also aim to track styling deviations via code scanning.	Themes are seen, such as educating how to use a component in Figma, an enhancement is needed to support a new use case, a minor bug, etc. Getting code deviation is harder. We have code scanning and plan to get the details on styling customizations through the code chunks, hoping to identify themes for needed customization like layout or color variations, etc.
25	Yes	I do this very minimally in that I track what products are using other design systems besides ours. I also track 3rd party tool usage for data table usage as our data table is solid but we aren't a data table only team so some teams use tools like ag-grid. I'd love to be able to track deviations from our design system in more granular ways like patterns or even usage guidelines that aren't being adhered to. The main things holding us back from tracking these deviations are capacity and tooling. We also have 100s of tools that user our design system so trying to dig into those across the enterprise is time-consuming and manual without some good tooling in place. We are hoping to be able to leverage AI in doing this.	The most basic metric we start with is adoption. If we don't have a good idea of who is using our system and who isn't, then we don't know our users very well and can't serve their needs. Being able to track those products that are deviating from using our system altogether provides me, as product manager, a list of products to talk to so that I can find out more - why they chose another tool, what might be lacking from our design system, what they need in order to convert to our system, etc. If I can track things at a more granular level, then I can hopefully also identify patterns and use cases where we might need to develop new things or change what we offer to meet our user's needs. If everyone takes a pattern and deviates from it in the same way, then we probably need to evaluate our offering to change to how it's being implemented.
26	Yes	Figma library analytics One on one and group meetings	What is negotiable in changing/updating/removing/rebuilding. What is completely not working for our designers.
27	Yes	Detach analytics in Figma	Our DS hasn't been in use long enough and it's gone through changes such that we haven't been able to learn much yet from this metric.
28	No	We are just getting started with our design system and are in transition.	In the past, this was a good way to know if the main needed a branch into a product line!
29	No	We haven't had the capacity to do this in a meaningful manner. Yes, Figma measures detachments, but that's never been a meaningful metric for us.	We'd like to, but just don't have the resources to do it.
30	No	Another piece of data that we would have to maintain and track.	Understand where design system teams need to consider local systems, or graduating something into the system.
31	No	We're not tracking deviations (yet). I just released the beta versions of the design system libraries in Figma, and we're still in the process of migrating from a style guide sans libraries in Sketch.	I hope to learn about areas where the design system may be able to provide value by increasing flexibility/variants, or areas where we could standardize designs and reduce complexity by using the system as-is.
32	Yes	It needs to be done separately by platform. In Figma, we take a look at the component detaches. For web, we see usages of a "box" component which basically acts as a wrapper.	Simply observing the deviations is not enough. It is important to talk to users of the DS to understand the need or reason behind deviating. In some cases, this leads to adding new functionalities or investment in educational materials/sessions.
33	No	We don't have access to the production repos to be able to gather the data and we've yet to find a solid analytics platform which offers the features we would like for the price.	It's definitely something we would like to track. It can feel like wasted effort when we're improving a component which we don't know is actually used. It would be good to be able to prioritise our work better.
34	No	Too much work! We're talking about how might we do this, like pre-pending core components from custom components, but the dev team told me just yesterday that this would be hard to govern. I don't believe that yet.	The system is basically out of control, leading to an inconsistent product with each designer, now about 20, putting their mark on the product in a different way.
35	No	n/a	We're just launching our pilot this week, so we've yet to start formally gathering ANY data.
36	No	Haha, deviation feels like something that just happens In order to track deviation, we'd need to be able to quantify what qualifies as a deviation vs a variant or context modification. We have split our core library from the work that product teams need to deliver in order to meet deadlines and innovate. We then have processes to evaluate the "deviations" in order to fold updates into core.	Not gathering metrics have proven that we should be ;)
37	No	We are held back by a lack of official review system.	We've recently researched and trialed a new content QA process that potentially has overlap with the design system tracking idea. It could be that the review opportunity we are lacking could be wrapped into this process. There is definitely opportunity for it to be a secondary outcome.
38	No	Head count, shifting priorities, shifting stakeholder influence	Less confidence that we are serving our users and achieving our goals
39	No	Not at that point yet.	N/A
40	No	The design organization is aligned with the design system team and mission. If a deviation shouldn't be there, over time it is resolved. If it is a useful deviation, in time it is escalated to our team for official inclusion in the system.	While we don't formally track the deviations, we are aware of key deviations and it helps inform our planning if there are deviations we want to adopt across the system .
41	No	We haven't prioritized it	We're always frustrated when a dev team deviates by writing custom code, and then expects us to support their code.
42	No	We have no idea how to do this correctly. We could start a shared storybook but then we need to get teams to adopt it in their workflow or we need to actively go after code and designs to collect it in there. We tried a figma file for designers but it wasnt used.	The need for seeing “community” components and patterns is often shared by our subscribers, but they often don’t want to put the time in to make their work visible. Obviously the DS team needs to find a way to facilitate it and make it easy to do.
43	Yes	on the Figma side with plugins and REST API scripts that analyze designs for component usage and on the code side for a similar tool for LOBs code.	it’s key to gather the metrics but it’s even more important that components are aligned between Figma and code in: name (should be identical to avoid confusion, a lot of it), variants should also be easy to map to code “variants”. Making sure to not over complicate Figma components to match how code works as that can hinder adoption.
44	Yes	Not with a great rigidity. Thought Figma analytics for component detachment, Design review to see the compliance of the flow and mockups with the design system before their devellopment.	With deviation you can learn many things : - the component doesn't fit the need - there's a need of a new component - users don't know how to use the given component so they need support (tranings, office hours, ....)
45	No	No time. It'd be really difficult to automatically capture and categorise data on anything that's not the DS, especially since we expect a good 20-40% of tailor-made design to happen in every feature of a product. Finding things that look like the DS in all this noise, just to figure out why the DS wasn't used, feels innefficient. Instead, we encourage our users to report to us when a DS component didnt work or when they had a new need. We gather a lot of qualitative data through weekly design review crits, done by designers in a group setting (including DS designers).	ø
46	No	N/a	Even small inconsistencies can confuse designers and developers. We’ve learned that if our Figma file is up to date but doesn’t match Storybook or the documentation site, we get a lot of questions from teams trying to use the design system.
47	Yes	It's currently quite manual through audits, 'by-chance' encounters and it's definitely not a efficient way to do this. In an ideal world, the tracking would be auto-mated, self detect the errors and share a report back to the respective DS User to make necessary changes.	That if it's manual it can almost entirely consume all of DS Effort.
48	No	Competing priorities	There’s a lot of fear from our system’s users that tracking deviation from the ds would reflect poorly on them, even if it was for innovative/experimentation/explorative reasons. Deviation is expected (and perfectly healthy), but I’m learning that ds evolution could take a hit based on that fear.
49	Yes	I have a few methods I use to track deviations from the system. Unfortunately, none of them are automated and I'd love to discover a way to do this better and catch deviations earlier. One of the biggest challenges of design system management is keeping track of designs and iterations across multiple product teams. Outside of automated tooling, I use regular weekly design reviews and critiques, and perform a weekly review of component and variable usage analytics. However, the usage analytics is time and labor intensive and unreliable.	Most often I see deviations appear in two areas: a new feature or experience where requirements don't fit the exact current capabilities of the system, or, a gap in knowledge, understanding, or discovery of the system. In both cases it would be tremendously helpful to be aware of the gaps and deviations early and use them as a learning opportunity for both the design system and the feature or product teams. Approaching these gaps and deviations from the system with curiosity teaches me more about what the teams and system need with the goal of balancing governance with flexibility.
50	Yes	Through audits and communication with product teams. Feels a bit manual but works-ish.	We've learned where teams need flexibility, enhancement, new components, snowflakes, and generally how to adapt and be supportive (because this is adoption) rather than system police.
51	Yes	In a past role, I wrote a small script that leveraged tools like `react-scanner` to count up violations. This script would run in a nightly CI job, and send its measurements to Datadog, where I'd created a dashboard that showed violations. This way had some flexibility. At its most basic, it was just counting the number of components that had `Deprecated` in their name (which we wanted because it signaled to other developers the status of the component). We'd basically make all non-DS-approved components as Deprecated, like if a team had their own `Modal` component for a while. Since the CI → Datadog connection was already established, we also used it for tracking ESLint violations. And with ESLint, you can describe much more complex violations. As an example, I had also written an ESLint rule that disallowed a Button with the text "Click Here". It's part of the design system, but not exactly in the components-props way we usually think about.	Execs / managers love this type of data because it makes it so much easier for them to sell this work: it's just, number go down. It makes celebrating wins much easier, and keeping motivation high. Visibility is so powerful! The caveat I'd add is that... even though you can measure everything, not everything is worth measuring if it's not otherwise worth your time. Rarely are you lucky enough to get value out of "just keeping an eye on this number". So I'd also try to pick and choose what you measure and keep a high signal to noise ratio.
52	Yes	The closest I've gotten is two things: tracking requests, which in some ways will represent local customizations/diversions from the core library. if we can't support them right away. The other example is a recipe/emergent design library (across design and code) where people can share/reuse things that haven't yet graduated to the global level.	Maybe not something learned, but the reason I'm so interested in this topic is maybe just keeping track of diversions is enough for us to be okay with them. It adds some psychological safety that they will be considered/included in the future.
53	Not currently doing it, as I'm in between roles. But in my past role we started tracking.	We previously did audits, then tried to choose one component or icon to unify. Many things made it difficult, but two main things stand out. The amount of time it took to do due to the sheer number of deviations, the "inability" to change some things at the time as many pages had elements absolutely positioned so we "couldn't" unify things like form-field height (Yup, I know). And secondly, there was a popular opinion that the most "common" elements were the "right" one (we've always done it this way).	That you should think in terms of design systems from the very beginning, because the longer you put it off the harder it becomes. It's like the saying 'the best time to plant a tree was ten years ago; the second-best time is today.'
54	Yes	Omlet, honeycomb	What is needed in our system. Common issues people face when using components.
55	No	Time and resources are holding us back as we're a team of 2 and the speed of the project is quite fast. I'm chasing the other designer's deviations, so it's just 1 person to keep track of but there should be a better process	If we were more designers, it would be really hard to not make mistakes if we don't track deviations
56	No	Don't have the time, access, or resources to get the data.	That we're in a Catch 22 if we can't get metrics but they want them anyways.
57	No	It's actively top of mind, but working on a process to allow this much transparency between segments to allow to this exchange to be shared white not slowing down. RFC's are being worked on to handle this	Deviations can sometimes be coined as good friction rather than bad friction in my system. Will be super important as the system matures, and the culture of the org matures. We learn what's needed and can redirect early if necessary.
58	No	We're not. Still working to fully codify the foundation that would give us a clear position from which we could deviate.	The designers, and developers, want to maintain consistency. They don't want to necessarily reinvent the wheel, but they don't have the resources to avoid deviation or don't understand the implications of deviating.
59	Yes	Product audits, detachments, hard coded styles	Heavy lift due to limited automation, and capacity. A lot of noise in too many files risks measuring the irrelevant. Vague governance contributes to this problem. Appetite for "speed to market" is another contributore to this problem.
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100