We all feel the same way about being able to control things with your mind: it’s super cool. Elon Musk’s neurotechnology company, Neuralink, posted a video a few months back of a monkey playing pong and controlling his paddle just by thinking. That’s simply mind-blowing. It looks like researchers are making some pretty great strides in this field, and I can’t wait to see what kind of fun things we’ll be able to do with brain-computer interfaces like Neuralink.

I think that one of the main benefits of neurotechnology will be making things more accessible. Having mind controlled wheelchairs and robotic arms can be life changing for some people. In this article, I’m going to introduce you to the P300 wave that is emitted by our brains, and how you can use it to create amateur “mind-controlled” apps.

What is the P300 wave?

This is the Wikipedia definition of the P300 wave:

The P300 (P3) wave is an event-related potential (ERP) component elicited in the process of decision making. It is considered to be an endogenous potential, as its occurrence links not to the physical attributes of a stimulus, but to a person’s reaction to it. More specifically, the P300 is thought to reflect processes involved in stimulus evaluation or categorization.

It is usually elicited using the oddball paradigm, in which low-probability target items are mixed with high-probability non-target (or “standard”) items. When recorded by electroencephalography (EEG), it surfaces as a positive deflection in voltage with a latency (delay between stimulus and response) of roughly 250 to 500 ms.

I’ll break this down, starting with the oddball paradigm. This is basically a method where you repeatedly present someone with something they are not looking for, and there is a low chance for you to present them with an “oddball”, which is the thing that they are looking for. When a person sees this “oddball”, their brain releases a P300 wave.

Let’s take a look at a quick example. Imagine you have a bag of bouncy balls. 4 red balls and 1 green ball. Now, without looking inside the bag, you pick a ball and look at it’s colour. You are aiming to get the green ball. If you get a red ball, you put it back into the bag, shake the bag, and try to pick another ball. The first time, you pull out a red ball. Since that’s not the colour you want, you put it back in the bag, and your brain does not release a P300 wave. But, when you finally end up picking the green ball, your brain will release it.

Using the P300 wave for mind-controlled apps

Now that we have an understanding of the P300 wave, we can move on to applying it to create a mind-controlled application. I’ll be describing a high level, step-by-step process for designing a mind-controlled virtual keyboard for people who are unable to use physical keyboards.

Step 1: Using an EEG headset to detect the P300 wave

An EEG (electroencephalography) headset is something you wear around your head to monitor and record the electrical activity of your brain. One of the most common EEG headsets that can be used for this is the Muse Headband.

Once you have your EEG headset, you need to create a machine learning model that is able to classify a P300 wave from the EEG signals that you receive from the electrodes on the headset. You can collect the necessary training and test data by performing your own version of the oddball paradigm on yourself and using the headset’s SDK.

Once you have enough data, it’s time to create your classifier. The difficulty of this can range from 0: using logistic regression, to 100: using a covariance matrix and a minimum distance to mean (MDM) classifier (as described in this paper).

Step 2: Creating a socket server

Once you have your machine learning model for classifying the P300 wave, you need to set up your backend infrastructure. This is probably the easiest part in making the mind-controlled keyboard.

Make a simple server using your favourite programming language that has an endpoint for a socket connection and uses your machine learning model. Make sure you can access the EEG data stream from your headset as well for real-time classification. I’d recommend using Python, Flask, SocketIO, and PyLSL. Your socket endpoint should emit an event whenever your model detects a P300 wave from the headset.

Step 3: Creating the virtual keyboard

This is the last step to complete in order to create the mind-controlled keyboard. You will be creating a front end app to display the keyboard, and using the oddball paradigm to control it.

Since we want to emulate typing on the computer without using a keyboard, the tech stack required for this part would be something like Electron, RobotJS, React and SocketIO.

The keyboard and oddball paradigm will work like this:

There will be a grid of keys consisting of the letters, numbers and other symbols that are on a physical keyboard. This will be our virtual keyboard. You can place the keys in any order; follow the QWERTY layout, ABC layout or make your own.

There is a two-step process of typing a symbol. First the user will have to pick the column that the symbol is in, and then they will pick the row to pinpoint the symbol. This is where the oddball paradigm is used. Every column is randomly highlighted, and if the target symbol is in that column, the user will release a P300 wave to choose the column. The same goes for the rows. After a column is selected, every row in that column is randomly highlighted until the user releases a P300 wave to select that row, effectively selecting their target symbol.

In terms of how this will work on the tech side, Electron and React will be used to create a desktop app to to display the actual keyboard and the highlighting events. With the backend server running and the headset enabled, the desktop app will connect to it using a socket to get real time updates on whether a P300 wave was released or not. Finally, RobotJS will be used to actually emulate the typing.

Conclusion

That’s it, you’ve made a mind-controlled keyboard! This is definitely not the fastest way to type, nor is it extremely user friendly. However, I do think that this serves as a great example of the things that are to come in the world of neurotechnology. It is still in the early phases of research and discovery, but I hope that this inspired some of you to contribute to this field, and maybe create some life changing technology in the future.