Update: concurrency_limiter in streamlit-extras is now available for this purpose.

When you are developing a Streamlit app that includes a compute- or resource-intensive task, such as machine learning inference using a GPU, you may want to ensure that such a piece of the code only runs in a single thread (session) at a time, even when multiple users are accessing the app at the same time.

In such a case, using a lock is a solution. Streamlit creates a separate thread for each user session, so you can use a lock shared among the threads to control the execution of the compute-intensive code.

(For Gradio users: queue is a well-known solution for this problem in the case of Gradio.)

Here is an example of how to use a lock in a Streamlit app. st.cache_resource is used to create a global lock that is shared among the threads, and the lock is used to ensure that compute_intensive_task() is executed by only one thread at a time (if compute_intensive_task() is defined in another module, see the last section).

import streamlit as st
import threading


def compute_intensive_task():
    " Your compute-intensive code here "


@st.cache_resource
def get_global_lock():
    return threading.Lock()


global_lock = get_global_lock()

with st.spinner("Running a compute-intensive task"):
    with global_lock:
        st.write("Task started")
        compute_intensive_task()

st.write("Task completed")

Note that, with this lock-based approach, the concurrency of the locked task is limited to 1 per lock, while the queue and worker pattern can be used to control the concurrency to any number.

Practically, it would be a good idea to add a button to manually trigger the compute-intensive task in such cases. Without it, the task would be executed every time a new user accesses the app or the app is reloaded, which is not efficient.

import streamlit as st
import threading


def compute_intensive_task():
    " Your compute-intensive code here "


@st.cache_resource
def get_global_lock():
    return threading.Lock()


global_lock = get_global_lock()

if st.button("Run a compute-intensive task"):
    with st.spinner("Running a compute-intensive task"):
        with global_lock:
            st.write("Task started")
            compute_intensive_task()

    st.write("Task completed")py

If you want something like a “concurrency group”, you can create multiple locks and use them in the same way as below. To do so, get_global_lock() is changed to take a key argument so it returns different locks for different keys, as the st.cache_resource decorator controls the cache with the func arguments (and the func’s source code).

import streamlit as st
import threading


@st.cache_resource
def get_global_lock(key):
    return threading.Lock()


global_lock_A = get_global_lock("A")

with st.spinner("Running a compute-intensive task A"):
    with global_lock_A:
        st.write("Task A started")
        compute_intensive_task_A()

global_lock_B = get_global_lock("B")

with st.spinner("Running a compute-intensive task B"):
    with global_lock_B:
        st.write("Task B started")
        compute_intensive_task_B()

st.write("Task completed")

If the task is defined in another module and the Streamlit app script imports it, you can define the lock object in the module scope and simply use it in the function. This lock object will be shared among the threads. This is because, while Streamlit runs the main app script in a separate namespace in a different thread for each user session, the imported modules are loaded only once and shared among the threads like normal Python module usage.

# app.py (main Streamlit app script)
import streamlit as st

from foo import compute_intensive_task


with st.spinner("Running a compute-intensive task"):
    compute_intensive_task()

st.write("Task completed")

# foo.py (module containing the compute-intensive task)
import threading


lock = threading.Lock()


def compute_intensive_task():
    with lock:
        " Your compute-intensive code here "

Originally published at https://www.whitphx.info.

Different from academia

Unlike academic conferences and journals, it is common to submit the same talk proposal to multiple PyCon held in different countries/places simultaneously.

For other tech conferences, I have neither participated in nor submitted a talk proposal, so I don’t know the situation. You should check the conference’s website or ask the organizer or experienced people around you.

My case

In 2022, I presented a talk titled “Real-time browser-ready computer vision apps with Streamlit” at EuroPython 2022 and PyCon APAC 2022. This time, I first submitted the talk proposal to PyCon US 2022, then before the result came out, I submitted the same talk proposal to EuroPython and PyCon APAC as well. Because it was my first time to submit a talk proposal to PyCon, I was not sure if it was OK to submit the same talk proposal to multiple PyCon, so I asked the EuroPython and PyCon APAC organizers via e-mails. Then they said it was OK.

After I entered the PyCon sphere, I found that it is actually common. Many people do this, and it is sometimes a good strategy for them to make travel to many countries! (See my related post👉 Conf-driven Traveling✈️).

I wrote this post because I couldn’t find any information clearly answering this question on the Internet.

In 2023, I actually submitted the same talk proposal to multiple PyCon and it was accepted by PyCon DE & PyData Berlin 2023, PyCon LT, PyCon TW, and PyCon APAC🎉

Sign up for a conference, then plan the trip

If you are a tech person such as a programmer, and if you like to go somewhere but you always hesitate planning trips, why not sign up for a programming conference first? You will get a huge driver to travel there. No need to choose where to go. Just search a conference that will be held soon, or whose CfP is open if you want to attend it as a speaker. You will have excellent experiences wherever you go.

My case

I often take part in PyCons (Python conferences).

Last year, I visited Dublin for EuroPython 2022 (this article) and traveled UK and Germany after that.

At this very moment when I’m writing this, I am traveling European cities starting from Berlin for PyCon DE & PyData Berlin 2023 and it will end in Vilnius where I will attend PyCon LT 2023. I’m visiting several cities in 4 weeks between the conferences, sometimes exploring the cities and sometimes working remotely.

Unlike academic conferences, there are normally no restrictions on posting a proposal about the same topic to multiple Python conferences. So I have submitted the proposal to other PyCons, then I will have more chances to travel if it is accepted!

Bonuses🎉

There are some bonuses if you take this strategy.

• You will have local friends at the conference.
  Many conferences have social events where attendees can meet each other. You can also talk to speakers after their presentations, chat with other attendees at lunch or coffee break, or attend sponsors’ booths. If you would attend the conference as a (beginner) speaker, joining a mentorship program or speaker workshops is also a good chance to meet the community members.
  At the first place, building the community is one main goal of the conference of course. For travelers, in addition, making local friends is a good way to get to know the city and get involved into there. Going local restaurants or bars with them is a wonderful experience.
• Some conferences provide Financial Aid Programs. It will cover (some amount of) your travel expenses, accommodation, and/or a ticket. If you find a conference to go, check if it has the program.
• In most conferences, the ticket includes lunch and coffee break. You don’t have to worry about where to eat. You can focus on the conference and enjoy the lunch with other attendees. In addition, if you are a speaker, most conferences will provide a free ticket.

Wrap up

What you have to do is simple.

1. Search a conference that will be held soon, or whose CfP is open.
2. Buy a ticket or submit a proposal.

Then do all the remaining stuff automatically, such as booking the flight and the hotel!

Control your app’s color scheme and visual accessibility

One great feature of Streamlit is theming, so you can control your app’s color scheme. But creating a new color scheme can be hard. And making it visually accessible can be even harder.

So I made a color theme editor app with four color parameters of the Streamlit themes which general-purpose color editors/generators don’t even consider: primaryColor, backgroundColor, secondaryBackgroundColor, and textColor. I also followed the color contrast guidelines from the Web Content Accessibility Guidelines (WCAG) 2.0 standard for the app’s visual accessibility.

In this post, I’ll show you how to build this app in eight steps:

1. Define preset colors
2. If set, load the theme config
3. Manage the edited color parameters in the session state
4. Set colors programmatically
5. Define a composed widget of a color picker and a slider
6. Use WCAG contrast table
7. Show generated configs
8. Apply the edited theme to the app

👉 You can assess the app here and the source code here. This article is based on the revision [7213dbb]( 7213dbb).

5 key components of the color theme editor app

The app consists of five components to help you create and test accessible color themes:

• “Generate a random color scheme” button. Click this button to update the color theme with randomly selected colors that maintain high contrast between foreground and background.
• Color pickers. Use these components to adjust colors and their luminance parameters. Use the luminance slider to adjust contrast while preserving the color’s aesthetic.
• WCAG contrast ratio table. Display a 2x2 matrix of contrast ratios for the selected colors and the corresponding WCAG 2.0 level (AA or AAA) for each ratio to make your color theme accessible.
• Config generator. Generate content for the .streamlit/config.toml config file and a shell command with color theme arguments. Just copy/paste it into your project to apply the edited theme.
• “Apply theme on this page” checkbox. Check this box to apply the color theme you are editing to the whole application, letting you preview the colors in a real Streamlit app and synchronize the changes.

Step 1. Define preset colors

The app has preset colors that I picked from the Streamlit source code below:

• Base theme
• Light theme
• Dark theme

preset_colors: list[tuple[str, ThemeColor]] = [
    ("Default light", ThemeColor(
            primaryColor="#ff4b4b",
            backgroundColor="#ffffff",
            secondaryBackgroundColor="#f0f2f6",
            textColor="#31333F",
        )),
    ("Default dark", ThemeColor(
            primaryColor="#ff4b4b",
            backgroundColor="#0e1117",
            secondaryBackgroundColor="#262730",
            textColor="#fafafa",
    ))
]

Step 2. If set, load the theme config

This is a bit of a hack. You can access the global config object via st._config to get the theme config from it and to use it as a preset color:

@st.cache_resource
def get_config_theme_color():
    config_theme_primaryColor = st._config.get_option('theme.primaryColor')
    config_theme_backgroundColor = st._config.get_option('theme.backgroundColor')
    config_theme_secondaryBackgroundColor = st._config.get_option('theme.secondaryBackgroundColor')
    config_theme_textColor = st._config.get_option('theme.textColor')
    if config_theme_primaryColor and config_theme_backgroundColor and config_theme_secondaryBackgroundColor and config_theme_textColor:
        return ThemeColor(
            primaryColor=config_theme_primaryColor,
            backgroundColor=config_theme_backgroundColor,
            secondaryBackgroundColor=config_theme_secondaryBackgroundColor,
            textColor=config_theme_textColor,
        )

return None

theme_from_initial_config = util.get_config_theme_color()
if theme_from_initial_config:
    preset_colors.append(("From the config", theme_from_initial_config))

Step 3. Manage the edited color parameters in the session state

Store the RGB color values in the session state using the same keys as the color picker widgets. This lets you programmatically set the color picker values. It’s useful when updating them with new values from the random color generator. These values are synced with the color picker widgets, so their values must be compatible with the color picker’s #RRGGBB format.

Aside from managing RGB values, you also maintain the HSL values in the session state to control the slider widgets. Using the HSL format makes it easier to edit colors in a more intuitive way. To synchronize these two formats, create a utility function named set_color and use it every time you update the color values:

def sync_rgb_to_hls(key: str):
    # HLS states are necessary for the HLS sliders.
    rgb = util.parse_hex(st.session_state[key])
    hls = colorsys.rgb_to_hls(rgb[0], rgb[1], rgb[2])
    st.session_state[f"{key}H"] = round(hls[0] * 360)
    st.session_state[f"{key}L"] = round(hls[1] * 100)
    st.session_state[f"{key}S"] = round(hls[2] * 100)


def set_color(key: str, color: str):
    st.session_state[key] = color
    sync_rgb_to_hls(key)


if 'preset_color' not in st.session_state or 'backgroundColor' not in st.session_state or 'secondaryBackgroundColor' not in st.session_state or 'textColor' not in st.session_state:
    set_color('primaryColor', default_color.primaryColor)
    set_color('backgroundColor', default_color.backgroundColor)
    set_color('secondaryBackgroundColor', default_color.secondaryBackgroundColor)
    set_color('textColor', default_color.textColor)

Step 4. Set colors programmatically

To set the currently edited colors from the selectbox widget with the preset colors, use set_color():

def on_preset_color_selected():
    _, color = preset_colors[st.session_state.preset_color]
    set_color('primaryColor', color.primaryColor)
    set_color('backgroundColor', color.backgroundColor)
    set_color('secondaryBackgroundColor', color.secondaryBackgroundColor)
    set_color('textColor', color.textColor)


st.selectbox("Preset colors", key="preset_color", options=range(len(preset_colors)), format_func=lambda idx: preset_colors[idx][0], on_change=on_preset_color_selected)

You can also implement a random color generator button:

if st.button("🎨 Generate a random color scheme 🎲"):
    primary_color, text_color, basic_background, secondary_background = util.generate_color_scheme()
    set_color('primaryColor', primary_color)
    set_color('backgroundColor', basic_background)
    set_color('secondaryBackgroundColor', secondary_background)
    set_color('textColor', text_color)

To implement util.generate_color_scheme, refer to this code. It generates colors with the constraint that the luminance (the "L" in the HSL format) parameters of the colors have enough difference.

Step 5. Define a composed widget of a color picker and a slider

This app provides a pair of a color picker and a slider to control the color’s luminance parameter (to adjust the color contrast while maintaining its original appearance). There are four pairs of components. Each defined by a function called color_picker to make it reusable.

One trick here is to set the on_change callback of each component to synchronize the RGB data and the HSL data in the session state:

def color_picker(label: str, key: str, default_color: str) -> None:
    col1, col2 = st.columns([1, 3])
    with col1:
        color = st.color_picker(label, key=key, on_change=sync_rgb_to_hls, kwargs={"key": key})
    with col2:
        r,g,b = util.parse_hex(default_color)
        h,l,s = colorsys.rgb_to_hls(r,g,b)
        if f"{key}H" not in st.session_state:
            st.session_state[f"{key}H"] = round(h * 360)

        st.slider(f"L for {label}", key=f"{key}L", min_value=0, max_value=100, value=round(l * 100), format="%d%%", label_visibility="collapsed", on_change=sync_hls_to_rgb, kwargs={"key": key})

        if f"{key}S" not in st.session_state:
            st.session_state[f"{key}S"] = round(s * 100)

    return color

primary_color = color_picker('Primary color', key="primaryColor", default_color=default_color.primaryColor)
text_color = color_picker('Text color', key="textColor", default_color=default_color.textColor)
background_color = color_picker('Background color', key="backgroundColor", default_color=default_color.backgroundColor)
secondary_background_color = color_picker('Secondary background color', key="secondaryBackgroundColor", default_color=default_color.secondaryBackgroundColor)

Step 6. Use WCAG contrast table

This table layout is created using stacked st.column(). The content of each cell is encapsulated within a reusable helper function, such as synced_color_picker or fragments.contrast_summary:

col1, col2, col3 = st.columns(3)
with col2:
    synced_color_picker("Background color", value=background_color, key="backgroundColor")
with col3:
    synced_color_picker("Secondary background color", value=secondary_background_color, key="secondaryBackgroundColor")

col1, col2, col3 = st.columns(3)
with col1:
    synced_color_picker("Primary color", value=primary_color, key="primaryColor")
with col2:
    fragments.contrast_summary("Primary/Background", primary_color, background_color)
with col3:
    fragments.contrast_summary("Primary/Secondary background", primary_color, secondary_background_color)

col1, col2, col3 = st.columns(3)
with col1:
    synced_color_picker("Text color", value=text_color, key="textColor")
with col2:
    fragments.contrast_summary("Text/Background", text_color, background_color)
with col3:
    fragments.contrast_summary("Text/Secondary background", text_color, secondary_background_color)

synced_color_picker() is just a color picker component, but its value is synchronized with the relevant color picker component that appeared above. Its value is also managed in the session state with the value argument and the on_change callback:

def synced_color_picker(label: str, value: str, key: str):
    def on_change():
        st.session_state[key] = st.session_state[key + "2"]
        sync_rgb_to_hls(key)
    st.color_picker(label, value=value, key=key + "2", on_change=on_change)

fragments.contrast_summary() renders the WCAG contrast info. See this code for its implementation.

Step 7. Show generated configs

The app shows these ready-to-use code snippets and is done with st.code():

st.subheader("Config file (`.streamlit/config.toml`)")
st.code(f"""
[theme]
primaryColor="{primary_color}"
backgroundColor="{background_color}"
secondaryBackgroundColor="{secondary_background_color}"
textColor="{text_color}"
""", language="toml")

st.subheader("Command line argument")
st.code(f"""
streamlit run app.py \\
    --theme.primaryColor="{primary_color}" \\
    --theme.backgroundColor="{background_color}" \\
    --theme.secondaryBackgroundColor="{secondary_background_color}" \\
    --theme.textColor="{text_color}"
""")

Step 8. Apply the edited theme to the app

For reviewing purposes, this app can apply the currently edited theme to itself. You can do it with the st._config object, as shown below:

if st.checkbox("Apply theme to this page"):
    st.info("Select 'Custom Theme' in the settings dialog to see the effect")

    def reconcile_theme_config():
        keys = ['primaryColor', 'backgroundColor', 'secondaryBackgroundColor', 'textColor']
        has_changed = False
        for key in keys:
            if st._config.get_option(f'theme.{key}') != st.session_state[key]:
                st._config.set_option(f'theme.{key}', st.session_state[key])
                has_changed = True
        if has_changed:
            st.experimental_rerun()

    reconcile_theme_config()

    fragments.sample_components("body")
    with st.sidebar:
        fragments.sample_components("sidebar")

Use st._config.set_option() to update the configuration values. Reload the app for the changes to take effect by using st.experimental_rerun(). It was inspired by jrieke/streamlit-theme-generator (here is the code).

Wrapping up

The Streamlit color theme editor app offers a simple and effective solution for creating visually appealing and accessible color themes for your apps. Thanks to the WCAG 2.0 standard and the real-time preview, your themes will now be attractive and accessible to all users.

Happy app-building! 🧑‍💻

Accessible color themes for Streamlit apps was originally published in Streamlit on Medium, where people are continuing the conversation by highlighting and responding to this story.

Streamlit apps can run completely on web browsers without the server-side Python runtime

TL;DR

• I created a WebAssembly (Wasm) port of Streamlit, ”stlite“.
• You can try it out and share your apps on ”stlite sharing”, the online code editor + app sharing platform for stlite.
• The Wasm-based runtime provides many benefits such as offline capability, data privacy, and scalability.
• In addition to stlite sharing, you can also host your Streamlit/stlite apps on your web site or create offline desktop applications. So you can create multi-platform apps only with Python coding.

Streamlit: a great Python web app framework

Streamlit is a web app framework with which we can create interactive web apps only with Python coding. It is especially famous and popular for data apps development as it well supports many data widgets cooperating the DS/ML Python ecosystem, while it also covers a wide range of general purpose web development.

When you run a Streamlit app, the Streamlit framework starts a web server and it serves a frontend app that is a single page application (SPA) whose contents are dynamically constructed according to your app script written in Python. Then the frontend app continuously communicates with the web server and triggers the Python code executions on the server upon some events to get some results that dynamically update the frontend view. Due to this unique server-client architecture, we can construct interactive web apps only by writing the logics in the server-side Python code.

stlite: client-side Streamlit powered by Pyodide

There had been an epoch-making invent in the Python world — Pyodide. It is a CPython runtime compiled for WebAssembly (Wasm) that runs on web browsers (and NodeJS).

So I customized Streamlit to run on Pyodide runtime and released it as ”stlite“!

On stlite, the entry point is a JavaScript file that mounts the Streamlit frontend SPA into the HTML page, loads the Pyodide environment, and launches the Streamlit Python server in the Pyodide environment on the web browser.

With this architecture, thanks to Pyodide, the Python runtime no longer exists on the server side since it runs on the web browser. The web server is only for serving the static files such as HTML, JS, and CSS.

Pros and Cons

As it runs completely on the browsers, stlite has some benefits that the original Streamlit does not have.

• Offline capability: As even the Streamlit “server” runs on the browser, all the components resides on the client side, so once all the resources are loaded from the web server, the app can run offline.
• Data privacy: Since the entire app runs on your browser, even when you do “upload” some files from the file uploader on the page, these files are NEVER sent to any remote servers and only processed within your machine.
  This feature is sometimes beneficial especially in the applications of data science, machine learning, or statistics where Streamlit is widely used, as these often have strict privacy rules.
• Scalability: The main workload such as machine learning computation written in Python moves from the server to each browser, so the system becomes scalable.
• Multi-platform (web, desktop, mobile): As it runs on web browsers, it can also be an installable app (PWA), and can be bundled into a desktop app with Electron. Building mobile apps is also on the read map.
  As a result, you can create interactive apps only in Python with Streamlit and bundle them for each platform.
• Online editing experience: I developed the online editor & real-time preview service for Streamlit apps based on stlite — stlite sharing that we will see below soon.
  Precisely, this is not purely because of WebAssembly, but the Wasm-based architecture made it easier to create such a service which could not have existed before.

On the other hand, stlite and Pyodide has some disadvantages as a tradeoff.

• Some packages are not available: Some C extension packages such as TensorFlow cannot be installed because C extensions must be compiled for the Pyodide runtime specifically. For more details, read the Pyodide articles such as this one.
• Large initial payload: A large amount of resources will be downloaded when the user opens the app because Pyodide loads the whole Python runtime and the standard libraries, and stlite also downloads the necessary wheel files including the streamlit package.
• Network restriction: For the security reasons, accessing remote resources from the stlite applications are restricted by the browser, e.g. CORS.
• The source code is open: Note that all the source code and hosted data are sent to the client side, so they are visible to the users. You must not put any secrets on the source code of the stlite apps.

(This section is highly inspired by the blog post about Shinylive.)

Online code editor + app sharing platform: stlite sharing

stlite sharing is an online code editor + app sharing platform for stlite. It is the easiest and fastest way to try out stlite.

The following screenshot is the online editing & app preview mode of stlite sharing. You can try it out at https://edit.share.stlite.net/ .

When you create/edit/delete the files on the left pane, those changes are applied to the virtual file system on top of which stlite runs, then the Streamlit server detects the file changes and recommends you to hot-reload the application, as shown in the screenshot below. (The file change detection and hot-reloading is a part of the Streamlit core features. If you are not familiar with it, read the official document.)

The file changes are saved to the file system when the “Save” button is clicked. You can also add or delete the files on the file tabs.

Shareable URLs

On stlite sharing, you can get the shareable URL of the current app displayed at top right. There is also the “Open App” link next to it, through which you can open the URL.

By navigating to the URL or clicking the link, the app opens in the sharing mode as below, where only the Streamlit app opens and there are not the editor, the sharing link, and any other widgets.

Its URL is https://share.stlite.net/ with a long hash like #!ChBz....

The host https://share.stlite.net/ is for the app sharing mode of stlite sharing, while the editor mode we have seen above is at https://edit.share.stlite.net/.

In the long hash part of the URL like #!ChBz..., all the source code and data edited on the editor are encoded and embedded, so you can share the app only by copying this URL.

Another good point of this approach is, since all the code and data are embedded into the URL hash, these data are never uploaded and saved to the remote server.

The encoded URL hash is also loadable on the editor mode, so if you open the editor mode URL, https://edit.share.stlite.net/ with the hash, the app data will be restored on the editor.

Note that some SNS and URL shortening services cut off the long URLs, so you should take care of it when sharing the URL on such platforms. For the details, read this part of the Shinylive article.

This URL mechanism is inspired by Shinylive.

Adding files

By clicking the ”+” button at the file tabs area, you can create a new file.

You can also upload files from your local env (the files are never “uploaded” to any remote servers. They are just sent to the virtual file system on your browser.) Though this uploader, you can also add binary files.

Files in directories and Multipage apps support

You can edit the file name on the tab, and importantly, by inserting the file path delimiter, /, you can create files in directories.

With this feature, stlite sharing supports the Streamlit’s Multipage apps. In short, by creating a Python file in the pages/ directory (pages/*.py), you can add a new page in the app. For the details, read the official document about Multipage apps.

Installing packages

When you need to install some packages, use the special “requirements” tab.

The package names written in the editor on this tab will be passed to the package installer micropip.install() internally (on the Pyodide runtime, micropip is used as a package manager) when clicking the “Save” button. Write one package name per line.

This is a special tab in that the file content is not saved into the file system. It is used only for specifying the package names passed to micropip.install().

Please note again that some C extension packages cannot be installed. The following screenshot is the sample of such a case, where installing a C extension failed.

Samples

You can find some samples on the side menu, so check out these! I hope some inspire you.

For example, the “Streamlit Hello” demo runs the official Streamlit demo on stlite while it is available through the streamlit hello command with the original Streamlit.

With a recent release of Pyodide 0.21, many C-extensions became available including OpenCV. stlite sharing includes the samples using OpenCV and scikit-image with real-time video streaming. Even these samples run on web browsers and nothing are sent to any remote servers!

Yuichiro on Twitter: "Client-side real-time image processing!Write the app in #Python with @Streamlit + OpenCV, scikit-image, etc.,Run it on the browser with #WebAssembly.↓Try it out on stlite sharing!https://t.co/7GH6onBgrfEverything works on the browser. No uploads! pic.twitter.com/6xUlHBBfgU / Twitter"

Client-side real-time image processing!Write the app in #Python with @Streamlit + OpenCV, scikit-image, etc.,Run it on the browser with #WebAssembly.↓Try it out on stlite sharing!https://t.co/7GH6onBgrfEverything works on the browser. No uploads! pic.twitter.com/6xUlHBBfgU

Loading apps from GitHub or Gist

Instead of the encoded URL hash explained above, you can host your Python script somewhere and pass its URL to stlite sharing via the URL hash as below.

https://share.stlite.net/#https://...py

If the URL is of GitHub or Gist, you can pass the preview page URL instead of the raw file URL. For example, https://share.stlite.net/#https://github.com/napoles-uach/test/blob/main/app.py works.

When you need to install some packages, use the following format instead.

https://share.stlite.net/#url=<Script URL>?req=<Package1>&req=<Package2>

The example of this format installing opencv-python and matplotlib follows: https://share.stlite.net/#url=https://github.com/whitphx/stlite-sample/blob/main/opencv-image-processing.py&req=opencv-python&req=matplotlib
There is also the Gist version: https://share.stlite.net/#url=https://gist.github.com/whitphx/f23b7b2bbda19cd421121bd72ebf2101&req=opencv-python&req=matplotlib

Host your Streamlit app on your web site

stlite also supports self-hosting the apps.

All you have to do is write and host a single HTML file.

It would be something like below, where you can embed the Python source code into the JS code as a string literal.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, shrink-to-fit=no"
    />
    <title>stlite app</title>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/@stlite/mountable@0.15.0/build/stlite.css"
    />
  </head>
  <body>
    <div id="root"></div>
    <script src="https://cdn.jsdelivr.net/npm/@stlite/mountable@0.15.0/build/stlite.js"></script>
    <script>
      stlite.mount(
        `
import streamlit as st
name = st.text_input('Your name')
st.write("Hello,", name or "world")
`,
        document.getElementById("root")
      );
    </script>
  </body>
</html>

If you need to install some packages, specify the entrypoint file name, and/or use multiple files, use the following more flexible API.

stlite.mount(
  {
    requirements: ["matplotlib"], // Packages to install
    entrypoint: "streamlit_app.py", // The target file of the `streamlit run` command
    files: {
      "streamlit_app.py": `
import streamlit as st
import matplotlib.pyplot as plt
import numpy as np
size = st.slider("Sample size", 100, 1000)
arr = np.random.normal(1, 1, size=size)
fig, ax = plt.subplots()
ax.hist(arr, bins=20)
st.pyplot(fig)
`,
    },
  },
  document.getElementById("root")
);

After writing the HTML file, host it wherever you like, such as GitHub Pages. In your local env, you can test it with python -m http.server 8000.

This YouTube video may guide you to set it up on GitHub Pages (although the stlite API used in the video is out-dated now.)

Please read the README for the details about self-hosting apps!

Bundle your Streamlit app as a desktop app

The stlite apps run on web browsers, so it is also possible to bundle them into desktop apps with Electron.

Charly Wargnier 🥑 on Twitter: ".@whitphx, the brilliant mind behind `stlite` is getting close to having '@streamlit app -> desktop app' working! 🎈🤯(via @blackaryz, cc: @electronjs)🔗https://t.co/RDHoVp3rhv pic.twitter.com/HMqMyH70eG / Twitter"

@whitphx, the brilliant mind behind `stlite` is getting close to having '@streamlit app -> desktop app' working! 🎈🤯(via @blackaryz, cc: @electronjs)🔗https://t.co/RDHoVp3rhv pic.twitter.com/HMqMyH70eG

To create your own desktop app with stlite, follow this instruction! -> https://github.com/whitphx/stlite/tree/main/packages/desktop-cli#readme

There is a sample app repository and its distributable files too.

• A sample app repository
• Distributable files (the macOS app file is not signed, so the security alert may be shown)

Tell us your story!

When you create some apps with stlite, please share it!

If it’s on stlite sharing, all you have to do is copy and paste the URL 👍

These are good places to share your apps, samples, or case studies!

• stlite GitHub Discussions
• Streamlit community forum

Streamlit is a Python framework with which developers can quickly build web apps without frontend coding. On top of it, developers can make real-time video/audio processing apps that receive video/audio streams from users’ media devices, only with ~10 lines of code in the case of the simplest example.

Since such apps are web-based, they can be deployed to the cloud, shared with users easily, and have modern and user-friendly UIs.

This tech stack is useful for creating demos and prototyping ideas of video/audio apps such as human or object detection, style transfer, image filters, speech recognition, video chat apps, and more.

You can see more examples at the examples section below.

NOTE: These sample apps are hosted on the public cloud ( Streamlit Cloud), and the video and audio streams are transmitted to and processed at the cloud server. While those data are only processed on memory and not saved to any storage, however, if you are concerned, please do not use them. As for the following contents in this article, we can execute all of them on our local. In addition, you can try the examples above on your local by following the instructions at the examples section below.

NOTE: I had a presentation about this topic at EuroPython 2022 titled as “Real-time browser-ready computer vision apps with Streamlit.” The talk video is available below:

UPDATE: This article has been updated on 2022/09/02 using the newly introduced API of streamlit-webrtc that has been available since v0.40.0.

The advantages of web-based apps

We have been typically using OpenCV to build real-time demo apps of image or video processing. Some of you (especially developers or researchers in such fields) may have seen the following code or similar many times.

import cv2

cap = cv2.VideoCapture(0)

while True:
    ret, frame = cap.read()

    img = cv2.Canny(frame, 100, 200)  # Some image processing

    cv2.imshow('frame', img)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Compared to the GUI apps like above using cv2.VideoCapture and cv2.imshow that run on local environments, web-based apps have some advantages like below.

Easy to share and run:

• If we deploy the apps on the cloud, we can share the apps with our users simply by sending the URLs.
• The users can use the apps only by accessing them through web browsers. It does not require any set-ups or external dependencies.

Usable on smartphones:

• Because all the users need is web browsers, the users can use the apps on their smartphones. It’s convenient if we can show demos on such portable devices.

User-friendly UIs:

• Developers can use text inputs, sliders, or other web-based components to accept user inputs or show data. Such web-based UIs are more friendly for users than desktop GUIs in recent days.

Tutorial

We will create a simple web-based real-time video processing app with ~10 or 20 LoC. Please try this tutorial in an environment where a webcam and a microphone are available.

You can check the final result of this tutorial in this repository. The deployed online demo is here🎈

In this tutorial, we will write code in app.py. Please create an empty app.py at first.

$ touch app.py

Install necessary packages

Next, we have to install the packages necessary for this tutorial.

$ pip install -U streamlit streamlit-webrtc opencv-python-headless

• streamlit: The Streamlit main package.
• streamlit-webrtc: A custom component of Streamlit which deals with real-time video and audio streams.
• opencv-python-headless: OpenCV. We choose the headless version here because we will construct the UI with Streamlit.

First contact with Streamlit

NOTE: Please skip this section if you have experience in Streamlit.

First of all, launch the Streamlit with the command below. Please run the command in the same directory to the app.py.

$ streamlit run app.py

After a while, the Streamlit server process will boot up. Then access http://localhost:8501 to see the page like below (or it will automatically open in the browser by default). The screenshot here is in dark-mode, and if you are using light-mode, it looks different.

At this moment, there is no content on the web page because app.py is empty. We will add lines of code in the app.py for the Streamlit app.

Open the app.py with your editor and write the code below.

import streamlit as st

st.title("My first Streamlit app")
st.write("Hello, world")

When you save the file, Streamlit will detect the file change and shows the “Rerun” and “Always rerun” buttons on the top right of the screen.

Click the “Rerun” button. Then the web page is reloaded, and the page content would be like below. The web page content is generated based on the app.py code.

If you had clicked the “Always rerun” button, the page would automatically be reloaded every time the file changed.

Note that you have to reload the page like above in the following instructions where you update the app.py.

Now we have walked through the basic development flow of Streamlit apps. You write Python code with Streamlit components like st.title() and st.write() and pass it to the streamlit run command, then Streamlit generates the corresponding frontend contents on the web page.

In the next section, we will see how to develop a real-time video processing app on top of Streamlit. Apart from that, Streamlit itself covers more use cases such as machine learning, data science, or more general purposes. For such use cases, please see the official Streamlit tutorial for example.

Introduce the real-time video/audio streaming component

Update the app.py as below.

import streamlit as st
from streamlit_webrtc import webrtc_streamer

st.title("My first Streamlit app")
st.write("Hello, world")

webrtc_streamer(key="example")

We have added a single line with webrtc_streamer(). The web app would be like the screenshot below.

At the first trial, it may take some time to compile the package so that the page keeps showing the “running” message for a while after clicking the “Rerun” button. In such a case, wait for the process to finish.

Click the “START” button to start the video and audio streaming. You may be asked for permission to access the webcam and microphone at the first trial. Grant permission in that case.

The webrtc_streamer(key="example") above is a Streamlit component which deals with video and audio real-time I/O through web browsers. The key argument is a unique ID in the script to identify the component instance. We have set it as "example" here, but you can use any string for it. The component in this example only receives video and audio from the client-side webcam and microphone and outputs the raw streams. It's the most basic version of the component. We are going to enhance its functionality by adding other options in the following sections.

Development of a real-time video processing application

Update the app.py as follows.

import streamlit as st
from streamlit_webrtc import webrtc_streamer
import av
import cv2

st.title("My first Streamlit app")
st.write("Hello, world")


def callback(frame):
    img = frame.to_ndarray(format="bgr24")

    img = cv2.cvtColor(cv2.Canny(img, 100, 200), cv2.COLOR_GRAY2BGR)

    return av.VideoFrame.from_ndarray(img, format="bgr24")


webrtc_streamer(key="example", video_frame_callback=callback)

Try it out by clicking the “START” button like the previous section. With this new example, you can find that an image filter is applied to the video stream.

We have defined a callback that receives an input frame and returns an output frame. We also put image processing (edge detection in this example) code inside the callback. As a result, we have injected the image processing code into the real-time video app through the callback.

Detailed explanations about the code follow.

• webrtc_streamer() can take a function object through the video_frame_callback argument as a callback.
• The callback receives and returns input and output image frames. These are instances of the VideoFrame class from PyAV. The PyAV library is a Python binding of ffmpeg, which provides video and audio capabilities. It is installed as a dependency of streamlit-webrtc.
• The argument of the callback is an image frame in the input video stream sourced from the webcam. It can be converted into a NumPy array with frame.to_ndarray().
• The returned value from the callback is displayed on the screen. In the sample above, a new VideoFrame object to be returned is generated from a NumPy array, with av.VideoFrame.from_ndarray(img, format="bgr24").
• Any code can be put inside the callback. In the example above, we have used an edge detection filter cv2.Canny(img, 100, 200) (and a grayscale converter cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)) as an example.

Now, we have created a browser-ready real-time video processing app! We used a simple Canny edge detector in this example, and you can replace it with any image processing code in your original app.

If we use object detection or style transfer for that part, the app would be like the screenshots at the beginning of this article.

Receive user inputs

Update the app.py as below.

import streamlit as st
from streamlit_webrtc import webrtc_streamer
import av
import cv2

st.title("My first Streamlit app")
st.write("Hello, world")

threshold1 = st.slider("Threshold1", min_value=0, max_value=1000, step=1, value=100)
threshold2 = st.slider("Threshold2", min_value=0, max_value=1000, step=1, value=200)


def callback(frame):
    img = frame.to_ndarray(format="bgr24")

    img = cv2.cvtColor(cv2.Canny(img, threshold1, threshold2), cv2.COLOR_GRAY2BGR)

    return av.VideoFrame.from_ndarray(img, format="bgr24")


webrtc_streamer(key="example", video_frame_callback=callback)

Then click the “START” button. You will find that there are 2 sliders in this example. You can modify the parameters of cv2.Canny() through the sliders, even during execution in real time.

With this update,

• We added threshold1 and threshold2 variables.
• We added two slider components with st.slider() and assigned their values to these variables. st.slider() is a built-in component of Streamlit. Its official API reference is https://docs.streamlit.io/library/api-reference/widgets/st.slider.
• We then passed these variables to cv2.Canny() inside the callback as its parameters.

Now we have interactive inputs to control the real-time video filter!

Execution model of the callback and an important notice about it

Unlike OpenCV, streamlit-webrtc requires callbacks to process image and audio frames. This callback-based design is one major difference between OpenCV GUI and streamlit-webrtc, and there are a few things you have to be aware of about it.

Please note that the callback is executed in a forked thread different from the main thread where the Streamlit app code runs. It makes some restrictions as below.

• The global keyword does not work as expected inside the callback.
• Streamlit methods such as st.write() cannot be used inside the callback.
• Communications between inside and outside the callback must be thread-safe.

Deploy the app to the cloud

We are going to make the web app available to everyone by deploying it to the cloud.

Configure WebRTC

To deploy the app to the cloud, we have to add rtc_configuration parameter to the webrtc_streamer().

webrtc_streamer(
    key="example",
    video_frame_callback=callback,
    rtc_configuration={  # Add this line
        "iceServers": [{"urls": ["stun:stun.l.google.com:19302"]}]
    }
)

This configuration is necessary to establish the media streaming connection when the server is on a remote host.

streamlit_webrtc uses WebRTC for its video and audio streaming. It has to access a "STUN server" in the global network for the remote peers (precisely, peers over the NATs) to establish WebRTC connections. While we don't look at the details about STUN servers in this article, please google it with keywords such as STUN, TURN, or NAT traversal if interested.

We configured the code to use a free STUN server provided by Google in the example above. You can also use any other available STUN servers.

The value of the rtc_configuration argument will be passed to the RTCPeerConnection constructor on the frontend.

HTTPS

We have to serve web apps on remote hosts via HTTPS to use webcams or microphones.

Not only the webrtc_streamer() component we used here but also any frontend apps that access the client-side webcams or microphones use MediaDevices.getUserMedia() API. This API does not work in an "insecure context."

The document says

A secure context is, in short, a page loaded using HTTPS or the file:/// URL scheme, or a page loaded from localhost.

MediaDevices.getUserMedia() - Privacy and security

As a result, we need HTTPS to serve web apps on remote hosts which access the client-side webcams or microphones.

Streamlit Cloud

I recommend Streamlit Cloud for Streamlit app hosting. You can deploy the apps from GitHub repositories with a few clicks, and it automatically serves the apps via HTTPS. And Streamlit Cloud seems to provide better runtime than Heroku free-tier, while Streamlit Cloud provides a large deployment capacity for free.

Please refer to the official document for its usage.

I actually deployed the app we have seen in this article on Streamlit Cloud: https://share.streamlit.io/whitphx/streamlit-webrtc-article-tutorial-sample/main/app.py .

Its GitHub repository is https://github.com/whitphx/streamlit-webrtc-article-tutorial-sample .

Note that requirements.txt has been added to install the necessary dependencies ( streamlit-webrtc and opencv-python-headless) in the Streamlit Cloud environment: https://github.com/whitphx/streamlit-webrtc-article-tutorial-sample/blob/main/requirements.txt

Notice

As written above, the video and audio streams sourced from the client devices are transmitted to and processed at the server.

So, this library is not scalable and depends on network connectivity. You may think of it mainly for prototyping or demo purpose.

You also have to consider hosting the apps in local networks if there are concerns about transmitting media to the remote cloud server.

Examples

This section is a copy of the sample list at https://github.com/whitphx/streamlit-webrtc.

Showcase including following examples and more

⚡️Repository, 🎈Online demo

• Object detection (This is the sample app a screenshot of which is at the beginning of this article)
• OpenCV filter
• Uni-directional video streaming
• Audio processing

You can try out this sample app using the following commands on your local env.

$ pip install streamlit-webrtc opencv-python-headless matplotlib pydub
$ streamlit run https://raw.githubusercontent.com/whitphx/streamlit-webrtc-example/main/app.py

Real-time Speech-to-Text

⚡️Repository, 🎈Online demo

It converts your voice into text in real time. This app is self-contained; it does not depend on any external API.

Real-time video style transfer

⚡️Repository, 🎈Online demo

It applies a wide variety of style transfer filters to real-time video streams.

Video chat

⚡️Repository (Online demo not available)

You can create video chat apps with ~100 lines of Python code.

Tokyo 2020 Pictogram

⚡️Repository: 🎈Online demo

MediaPipe is used for pose estimation.

Yuichiro on Twitter: "Pictogram app on web browsers.After forking the original code which uses OpenCV, it takes only 2 hours to add the web UI supporting real time video streams with Streamlit and streamlit-webrtc.https://t.co/VRpB27enlI https://t.co/MQAWJVqkiM / Twitter"

Pictogram app on web browsers.After forking the original code which uses OpenCV, it takes only 2 hours to add the web UI supporting real time video streams with Streamlit and streamlit-webrtc.https://t.co/VRpB27enlI https://t.co/MQAWJVqkiM

What about audio?

You can deal with audio streams in a similar way as video. If you define a callback function and pass it to the audio_frame_callback argument, the callback will be executed with audio frames. In the case of audio, the input argument and the returned value of the callback are instances of the AudioFrame class.

Please see the source code of a sample app changing the audio gain or the Speech-to-Text app in the examples above.

Originally published at https://www.whitphx.info.

Developing Web-Based Real-Time Video/Audio Processing Apps Quickly with Streamlit was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing the WebRTC component for real-time media streams

Real-time video processing is one of the most important applications when developing various computer vision or machine learning models. It’s useful because it allows users to quickly verify what their models can do with handy video input from their own devices, such as webcams or smartphones.

But it also presents a challenge to those of us using Streamlit, since Streamlit doesn’t natively support real-time video processing well yet through its own capabilities.

I created streamlit-webrtc, a component that enables Streamlit to handle real-time media streams over a network to solve this problem. In this in-depth tutorial, I’ll also briefly introduce you to WebRTC (check out my article here for more in-depth info on WebRTC). If you want to jump right to playing with the component here is a sample app.

Ready?

Let’s dive in.

(This tutorial requires Python >= 3.6 and a webcam.)

The problem with existing approaches

Streamlit is actively used by many developers and researchers to prototype apps backed with computer vision and machine learning models, but it can’t yet natively support real-time video processing.

One existing approach to achieve real-time video processing with Streamlit is to use OpenCV to capture video streams. However, this only works when the Python process can access the video source — in other words, only when the camera is connected to the same host the app is running on.

Due to this limitation, there have always been problems with deploying the app to remote hosts and using it with video streams from local webcams. cv2.VideoCapture(0) consumes a video stream from the first (indexed as 0) locally connected device, and when the app is hosted on a remote server, the video source is a camera device connected to the server - not a local webcam.

How WebRTC resolves this issue

WebRTC (Web Real-Time Communication) enables web servers and clients, including web browsers, to send and receive video, audio, and arbitrary data streams over the network with low latency.

It is now supported by major browsers like Chrome, Firefox, and Safari, and its specs are open and standardized. Browser-based real-time video chat apps like Google Meet are common examples of WebRTC usage.

WebRTC extends Streamlit’s powerful capabilities to transmit video, audio, and arbitrary data streams between frontend and backend processes, like browser JavaScript and server-side Python.

The WebRTC basics

The following tutorial uses knowledge about WebRTC concepts such as “Signaling”, “Offer”, and “Answer”. The below figure provides a simple summary of how to establish a WebRTC connection.

• WebRTC has a preparation phase called “Signaling”, during which the peers exchange data called “offers” and “answers” in order to gather necessary information to establish the connection.
• Developers choose an arbitrary method for Signaling, such as the HTTP req/res mechanism.

If you want to know more about these concepts, read this article.

Just as in the article linked above, this tutorial will use aiortc, a Python library for WebRTC, and an example from the aiortc repository as the basis for our sample project.

The basics of Streamlit’s execution model

To read further, you should know about the development of Streamlit bi-directional custom components and about Streamlit’s execution model. You can learn about it here.

Here is a short summary:

• Upon each execution, the Python script is executed from top to bottom.
• Each execution of the Python script renders the frontend view, sending data from Python to JS as arguments to the component.
• The frontend triggers the next execution via Streamlit.setComponentValue(), sending data from JS to Python as a component value.

Integrate aiortc into a Streamlit component

In this section, to understand how to integrate a WebRTC implementation into a Streamlit custom component, we will create a minimal version of streamlit-webrtc called tiny-streamlit-webrtc, as a hands-on tutorial.

The source code of tiny-streamlit-webrtc is hosted on GitHub. Throughout this tutorial, we will refer to this repository and review each intermediate commit step-by-step to reach the final version.

It is recommended for you to clone the repository:

$ git clone https://github.com/whitphx/tiny-streamlit-webrtc.git
$ cd tiny-streamlit-webrtc

With the below command, you can check out the specific revision referenced in each section in order to see the entire codebase and to actually try running it.

$ git checkout <revision>

Install dependencies

Install the necessary packages. Note that this tutorial does not work with the latest version of aiortc ( 1.1.1) and 1.0.0 must be used.

$ pip install streamlit opencv-python
$ pip install aiortc==1.0.0

Setting up the project

As usual, we start with the official template of a bi-directional component. The reference tiny-streamlit-webrtc implementation is based on the revision 4b90f52.

After copying the template files, complete the rest of the setup, including the steps below.

• Rename “my_component" to "tiny_streamlit_webrtc".
• Run npm install in tiny_streamlit_webrtc/frontend.
• Remove the existent code, comments, and docstrings.
• Add necessary files such as .gitignore

Check out what this section does, with code version f6daf28.

Rolling out the first frontend implementation

Let’s start writing code.

First, we will simply copy and paste some lines of code from index.html and client.js in the aiortc example into our React component, but with some fixes.

e3f70e4is the actual edit, and you can try this version by checking out the commit, as explained above.

$ git checkout e3f70e4

The view contains only a <video /> element with autoPlay and playsInline props, as it is in the original index.html, and a button element to start the WebRTC session. The start button's onClick handler is bound to the start() method, which is copied from client.js and slightly modified to remove some lines unnecessary for this tutorial and adjust to the React class-based component style. We will do the same for negotiate() and createPeerConnection().

Let’s run this component in the usual manner for Streamlit custom component development.

$ cd tiny_streamlit_webrtc/frontend/
$ npm start

$ streamlit run tiny_streamlit_webrtc/__init__.py

After opening the app with a web browser, open the developer tools, and click the “Start” button. You can see the offer is generated and printed in the console as below.

This is printed via this line. Please follow the steps leading up to it. This code is equivalent to the code in the original example before sending the offer to the Python server. Yes, this case is different from the original example. How can we send the offer to the Python process?

(You also see your webcam become active since navigator.mediaDevices.getUserMedia() requests its use.)

Send offer from JS to Python

streamlit-webrtc makes use of Streamlit.setComponentValue() for this purpose. We will learn about it in this section.

7b7dd2dis the next update. Use git checkout 7b7dd2d to check out it.

With this change, the offer is sent from the frontend to the server as a component value.

const offerJson = offer.toJSON()
Streamlit.setComponentValue({
  offerJson,
})

The offer can be read on the server-side as below.

component_value = _component_func(key=key, default=None)
if component_value:
    offer_json = component_value["offerJson"]

Let’s run this version and confirm the offer is displayed after clicking the “Start” button, which means the offer is received by the Python process and shown with st.write() here.

Server-side implementation with asyncio

Now the offer is received on the server-side, so let’s implement the code to process it. Just as we did with the frontend, let’s copy and paste from the example server.py to our streamlit_webrtc/__init__.py, like this, which is copied from offer() coroutine in the example server.py.

Note that a video transformer is temporarily omitted from the track event listener like this to focus on the WebRTC part for now. It now just passes through the input track to the output.

However, as you can see, this code contains async and await and does not work in a function. So, we have to wrap this part in a coroutine like this.

Please run this version: a6f7cc0 and confirm the answer is displayed following the offer from here. That means the server-side pc object has processed the offer and generated the answer.

What we have to do next is send it back to the frontend.

Send back the answer from Python to JS

To do this, streamlit-webrtc simply relies on Streamlit's data sending mechanism from Python to JavaScript as below.

_component_func(key=key, answer=answer)

However, one problem arises. We’ve already called component_value = _component_func(...) and obtained the offer from it. After that, we generated the answer. So, how can we set the argument to the already called _component_func() again?

Simply calling the second _component_func() as below does not work, because in the Streamlit app, different _component_func() calls are recognized as different instances of the component.

component_value = _component_func()
offer = component_value["offer"]
answer = generate_answer(offer)  # Pseudo code
_component_func(answer=answer)  # This does not work!

To resolve this problem, we have to introduce a hack: SessionState and st.experimental_rerun(). With these tools, we can rerun the script to call a _component_func() in the same line again and hold a variable over the runs to feed it to the _component_func() in the second and later executions.

SessionState has been discussed in this forum topic and the source is available on this page in Gist.

st.experimental_rerun() seems, as its name implies, to be an experimental API and not documented yet. It has been discussed in this GitHub issue and can now be used.

Please see this version of the server-side code, where SessionState and st.experimental_rerun() are used to feed the generated answer to the component.

This illustrates how it works.

Another important thing here is that the key argument is no longer optional but must be explicitly provided like this. As the answer is fed as an argument to _component_func() and its value changes over the runs, key is necessary as a stable identifier of the component instance.

If key is None, Streamlit identifies the component instance based on arguments other than key, so Streamlit cannot trace the identity of the component instance over the runs as the answer changes.

Note that this if-clause is added to invoke st.experimental_rerun() only the first time the server-side process gets the offer from the frontend. This may also be achieved by resetting the component value on the frontend once the offer is passed to Python.

With this version: aa2ab49, you can see the answer is provided as a field of the args prop like this on the frontend. Let's confirm it with the browser's devtools.

Implement processAnswer()

Now we have the answer on the frontend. Let’s implement the rest of the frontend code like this.

This code is copied from the part following receiving the answer in the example client.js and fixed to adjust to ours.

Introduce a thread running over script executions

It seems we have done all things we have to do, but no video appears when you click the “Start” button with this version: 7fbf0eb.

The problem resides on the server-side. The server-side WebRTC code from aiortc runs on an event loop, which is implicitly started with asyncio.run() here now. An event loop is created on which aiortc functions rely throughout one Streamlit script execution. But this event loop will be trashed in the next script execution and the aiortc can no longer keep working.

To resolve this problem, we will fork a thread and create an event loop inside it to run aiortc functions. And the thread object is stored in the SessionState to be maintained over the multiple Streamlit script executions.

See this version of the code: 093f81b. This webrtc_worker() function is forked as a thread here. Inside this thread, a new event loop is created and the process_offer() coroutine is running on it - which was invoked by asyncio.run() in the previous revisions of this code. With this change, queue.Queue is introduced to get the answer object in the main thread, which is now generated in the forked thread.

There is one drawback of forking a thread — the streamlit run command does not stop when you hit Ctrl+c. This is because the forked thread remains even after the main thread is terminated.

To forcefully terminate the process, send it SIGKILL as below.

$ ps aux | grep python | grep streamlit # Find the process ID whitphx 19118 11.2 0.6 4759304 99928 s003 S+ 5:27PM 0:02.06 /path/to/venv/bin/python3.8 /path/to/venv/bin/streamlit run tiny_streamlit_webrtc/__init__.py
$ kill -9 19118 # Send SIGKILL to the process specified with the ID

To fix it, the daemon option of the forked thread is set to True like this. With this flag, the script stops correctly when necessary.

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left.
“Thread Objects” (Python.org)

Component height adjustment

Let’s try out the current version: fc48060. Now, WebRTC works and the video appears with this component! However, the displayed video is cropped and the lower part of it is hidden like below.

To fix it, we have to call Streamlit.setFrameHeight() when the size of <video /> element changes. Although it is automatically called when the props are updated, the element resize is not associated with props updates but with starting video streaming.

Now attach onCanPlay event handler on the <video /> element and call Streamlit.setFrameHeight() from it like this. (While using ResizeObserver may be the right way to observe DOM element resizes, we use the onCanPlay event here as a substitute, for simplicity's sake.)

Cool! Now it works correctly. 🎉 1a57a97 is this version.

Now all the core parts for WebRTC are complete. We’ll implement the rest in the following sections.

Implementing your own video filter

First, let’s try to implement some video filters. 3ba703d is an example with a simple edge extractor, copied from the sample code of aiortc.

Implement a stop button

Refer to the aiortc example to create a stop button to gracefully stop the stream. is the current version.

The execution model of streamlit-webrtc

We have followed the steps to develop a minimal Streamlit component utilizing WebRTC to stream video.

As we’ve seen in this component, we chose a design in which the computer vision code is running in a callback in the forked thread, triggered by new frame arrivals from the input stream, independent of Streamlit’s script execution timings. It looks a little bit weird the first time you see it, but it’s necessary and natural when dealing with real-time streams.

Let’s see it from a more abstract view. When processing frames coming from real-time streams, the streams are additional event sources other than user interactions through the frontend view. In normal Streamlit apps, all the events triggering Python script executions are only sourced from the frontend and they are nicely encapsulated by Streamlit.

With its execution model, then, developers can write the apps in a clean world where there are no callbacks and no (or little) side effects. In turn, if we want to handle the streams with good performance, we have to explicitly handle the events sourced from the streams like frame generations, which breaks the elegant encapsulation, causing callbacks and events to appear in the script.

What tiny-streamlit-webrtc lacks

Though we’ve created a small subset of streamlit-webrtc, tiny-streamlit-webrtc, it still lacks many important features streamlit-webrtc has. Here we will review some of them.

Parameters input from Streamlit components

One of the biggest benefits of using Streamlit is interactive controls such as sliders and radio buttons. With computer vision and machine learning models, these controls are very useful to change the parameters during execution.

Because the computer vision code is running in the forked thread with this component, we have to pass the values obtained from Streamlit widgets to the CV code over the threads. But it is not difficult, like here in the streamlit-webrtc sample.

With tiny-streamlit-webrtc, you can do this by adding a public property to VideoTransformTrack and read and write it from each thread, just like the sample code linked above. Please try it if you are interested, and be careful about thread safety when you pass complex values.

Frame drops

We’ve used edge extraction as an example in the tutorial. However, if you replace it with more computationally expensive filters like deep neural networks, you will see the displayed video slows down. You can test it simply by putting time.sleep(1) in VideoTransformTrack.recv().

This is because VideoTransformTrack.recv() processes all the input frames one by one - if it delays, generating the output frames is also delayed.

To solve this problem, VideoTransformTrack.recv() has to drop some input frames and pick the latest one each time it runs. In streamlit-webrtc, it's done here when async_transform option is set as True.

Extendability

In tiny-streamlit-webrtc, the video transformation is hard-coded inside VideoTransformTrack.recv(), but of course, this is bad design as a library. To be reusable, it should expose an injectable interface through which developers can implement arbitrary kinds of video transformation, encapsulating details such as VideoTransformTrack.recv() and WebRTC-related code.

With streamlit-webrtc, developers can implement their own video transformations by creating a class extending VideoTransformerBase class like this and this.

Key takeaways

Streamlit is a nifty framework with a useful library, but it doesn’t handle real-time video processing well on its own.

WebRTC makes Streamlit even more awesome by enabling server-side processes and clients to send and receive data streams over the network with low latency.

Have an amazing project in mind to use WebRTC for? Share it with us in the comments or message me.

Credits

Reviewed by Yu Tachibana (@z_reactor)

Originally published at https://blog.streamlit.io on February 12, 2021.

Developing a streamlit-webrtc component for real-time video processing was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.