What started as a small experiment turned into a surprisingly fun project involving Python, Selenium, REST APIs, and a small virtualized environment running on an old laptop. I was able to explore where browser automation can fit in a smart home, and more importantly, why certain architectural decisions matter when building automations that interact with the web.

Why Automate?

I regularly order groceries online and often buy the same items every week (think fresh produce). But the repetitive flow of:

• Searching for the product
• Opening the product page
• Clicking “Add to cart”

…quickly becomes tedious when done repeatedly. Multiply that across multiple items and you lose a surprising amount of time, and attention every week.

My initial idea was to use NFC tags. Each tag would correspond to an item in my kitchen and tapping it with my phone would add the item to my cart. It sounded seamless, as most smart phones have an in built NFC reader, but in practice it wasn’t:

• If I rearranged my kitchen, I would have to move all the NFC tags.
• I needed my phone in hand every time I was in the kitchen.
• It added friction instead of removing it as if I wanted to add multiple items, I would have to navigate to wherever the NFC tags were located.

I wanted a more seamless interaction: a single button on my kitchen tablet (or phone) that instantly adds items to my cart. No typing, no searching… just tapping. And this brings us to my chosen approach.

Automate How?

My automation has three parts:

1. A Python REST server (running on an Ubuntu VM)
2. Selenium WebDriver to perform the browser actions
3. Home Assistant (HA) to orchestrate the flow and notify me

And the automation flow:

Tap button in HA → HA makes REST call → Python opens webpage → Selenium adds item to cart → Python returns success/failure → HA notifies me

The whole process finishes in about 10 seconds, compared to ~30 seconds if I did it manually (possibly more) and I’m not tied to my screen during those 10 seconds.

I wanted a clean separation between my smart home and the automation logic. Instead of embedding Python or browser logic inside Home Assistant, the REST API:

• Decouples the front-end trigger (HA) from the automation logic
• Allows scalability as any item can be added just by passing the product URL
• Keeps HA lightweight as no heavy browser automation is carried out on it

To accomplish this, I spun up both Home Assistant OS and Ubuntu in Proxmox VMs. This allows me to keep the browser automation isolated from the HA automation. If something crashes, only that VM is affected. Anytime there has to be a change made to either the HA automation, or the server it is possible without bringing down the full service. This also makes any enhancement on either part of the flow more seamless as each tool is designed to be responsible for a single task. The added benefit of Proxmox VMs make it easy to snapshot, backup or clone and even scale if needed. And VMs also mean I do not need to purchase multiple expensive hardware for each of my OS and merely an old laptop is sufficient for a reliable home server.

Many grocery sites don’t expose public APIs for adding items to a cart, and selenium mimics real user actions. It is also resilient against any UI changes, and would be able to work on almost any site if I wanted to expand this across various platforms. This makes it an ideal choice for this particular use case.

My early version hard-coded each automation with every item having its own complete end to end flow with its own API request. It worked, but was not scalable. To fix this, I built a single reusable HA script that accepts parameters like the product URL, allowing me to add unlimited items without duplicating code. This also means, enhancements to the automation flow is simpler as I only need to modify it at one place.

I also included timeouts and error handling which will be sent back to me as a notification so I can quickly identify what the issue is. This enables the automation to always have an end state and not be left waiting for a response from the API endpoint. I also get a notification, if the item was successfully added to cart and clicking on it opens the app on my mobile for quick verification (I don’t use this often because I trust my automation but it is a nice feature to have)!

Enhancing the Automation

One improvement I’m exploring is the ability to add an entire preset list of groceries with a single button tap instead of one for each item. This would be perfect for fresh essentials like fruits, vegetables and milk which I purchase weekly without fail. I’d also consider allowing the system to automatically add the weekly purchases without my input or even integrating it with voice assistants for hands-free additions.

I am also going to try to do some data analysis to see how frequently I use up other regular items such as rice and oils and possibly automate the addition based on the last date added to cart. This way, it would be like the smart home knows what I want, and allows me to reduce all the mundane tasks (and honestly this is my vision of a smart home, not just having devices connected to the internet and using our phones merely as a remote control).

For now, though, I already have the most important feature working… One tap is all it takes, my smart home buys for me!

Millimeter wave (mmWave) sensors such as Aqara’s FP2 presence sensors are one of the coolest applications of mmWave available to the consumer market at this time. But before we dive into its applications, we need to understand the underlying technology behind the sensor.

What are mmWaves

mmWaves get their names from the fact that their wavelengths are between 1–10 millimeters, which also makes them ideal for sensing very small changes in movements. Similar to infrared sensors, mmWave sensors work by transmitting electromagnetic waves. These waves are transmitted with increasing frequency and the reflected signals are processed. Not too dissimilar to ultrasonic sensors which emit sound waves and detect objects’ presence based on the reflection of said sound waves. However, contrary to ultrasonic waves, which are tailored for the transmission of sound waves through one type of medium (air), mmWave sensors tend to be more accurate as they are not optimized for a single transmission medium and these waves can be relied upon regardless of the environmental conditions [1]. The changes in signal from mmWaves can also be translated to accurate measurements of distance, velocity, and angle of an object [2].

So what about motion sensors? Typically motion sensors use Passive Infrared (PIR) signals. Instead of transmitting signals, they detect changes in the surrounding thermal environment. In a nutshell, when a warm body passes the sensor, the change in IR in the surroundings is what a PIR sensor detects [3]. So while they are good at catching large-scale movements, they may not be as accurate for smaller range of motions (I always get frustrated when I have to wave my hands vigorously before motion sensor enabled taps detect me or when a motion sensor light times out). mmWave sensors can be so accurate to the extent of detecting one’s breathing.

Applications of mmWave Sensors

mmWave sensors are currently being used in a whole host of applications. One area of huge promise is in automated parking. It can also assist in detecting rear seat occupancy so you don’t accidentally leave your kid locked in the car. In fact, Tesla has also filed to use mmWave sensors in its cars [4]. But I am more interested in the smart home applications of these sensors — which also make their potential accessible to the average consumer!

Such sensors can continue detecting you when you are sitting still on a couch, unlike a motion sensor. This means you can now safely set home automations to light fittings purely based on if someone is in the room or not. Additionally, one would not even need any additional fall detection hardware or tools as the high accuracy of identifying subtle movements can be beneficial in detecting falls! The accuracy of the sensors ensures they can detect multiple people and even which part of a room one is in (zone detection). Consequently, you can even have automations based on how many or where someone is in the room (imagine having the television turned on automatically if you are sitting on the couch). If you have several smaller lights in the room, you can turn on the lights individually based on where you are automatically as you traverse the room — how cool is that!

Future of mmWave Sensors

This technology has the potential to change the way we interact with gadgets and digital interfaces as shown by Soli, a gesture detection technology based on mmWaves developed by Google [5]. Such accurate detection could give rise to interactions with your television consoles, computers, or other devices without the need for a remote — just your hands and gestures. Google’s vision of gesture detection and how it can enhance human-computer interactions is portrayed here. Just like how Bluetooth and USB are standards, if gesture-based interactions have a specified standard, we could even see a unified way of interaction between devices, giving rise to an overall better user experience. While these sensors are fitted in cars to assist with automated parking, just imagine if we use them to control in-car entertainment systems hands-free. This could result in a reduction in distraction for drivers, making for a safer experience.

Another area in which these sensors could be deployed would be in retail and event settings to manage crowds, or even identify popular zones. By detecting precise movements or counting individuals, organizations can identify how and where customer interactions occur, leading to a data-driven approach in deciding what works and what doesn’t in terms of store layout, product displays, and the likes.

The ability to track micromovements makes the technology extremely viable in healthcare settings — not just for fall detection but they can be a potential substitute for wearables or contact devices that are currently used in tracking respiration or heart rate, aiding the comfort of patients [6]. With even better data processing and modelling, in the near future they could even be used to help detect certain diseases or medical conditions [7]. As mmWave radar technology becomes more integrated into healthcare systems, its applications will expand to enable real-time, remote, and highly accurate patient monitoring, potentially reducing the burden on healthcare systems and improving patient outcomes. With advancements in AI and data analytics, mmWave technology will also become more precise, comfortable, accessible, and actionable.

As the technology continues to advance, it’s exciting to think about how mmWave sensors will evolve. Their ability to detect fine movements with remarkable precision opens up new possibilities from device interactions to seamless home automation and improved vehicle safety features. Its integration with complex data models and analytics will only enhance its capabilities, making it an essential tool. With companies like Tesla, Google, and others investing in mmWave’s vast potential, we may soon see even more innovative and ubiquitous uses for this technology in everyday life as we are just beginning to tap into its full range of possibilities.

GenAI models churn out an output based on statistical models, analysing the input elements and identifying potential patterns and the likely output [1]. For instance, a text based genAI model like chatGPT is trained on an insane amount of text based datasets, trying to figure what the next likely character or word will be; while an image generating platform would be trained on images (pixels), and it will attempt to identify what shade the next pixel should be when generating an output. The outputs generated are kind of in a blackbox even to the engineers who developed such models. Even if the training data is 100% accurate, the data may not cover everything under the sun, so it ends up hypothesising and making a best guess effort in generating a response. This results in the very first problem, hallucinations.

Since these models attempt to merely predict the next best correct output, they do not possess the capability to understand the context of the question asked by the human user. There is no logical reasoning applied to questions asked. Subsequently, the response may not be the most accurate, and worst yet, may contain falsehoods. In general hallucinations can include a variety of inaccuracies, from contradictions, to falsehood, to even nonsensical answers. One popular hallucination making the rounds now can be seen on reddit posts where chatGPT consistently miscounts the number of letter ‘r’ in the word ‘strawberry’. You can view more examples of hallucinations over here. Overly relying on the responses as the absolute truth will be detrimental. Take for example a search I did recently where half of the suggestions were not even 2-syllables even though I explicitly asked for them and ChatGPT acknowledged the request.

That said, genAI models are continuously being improved upon [2], and the occurrence of hallucinations are likely to reduce as models get better over time. But for as long as models are not 100% accurate, we must always verify our sources, and do a cross-reference before accepting genAI responses as the ultimate truth.

Secondly, genAI models can be susceptible to bias. The models are only as good as the data they were trained on. If the data is inherently bias, it could be carried forward to the resulting output. Take for example, Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) — where Black people had a 44.9% chance of being incorrectly identified to be at risk of recidivism, while for white individuals, it was 23.5%. This means the algorithm was almost 2 times more likely to criminalize Black people incorrectly, potentially perpetuating a racial bias [3]. While COMPAS is not a genAI tool, the same limitation applies. Attempting to understand these models is another issue with genAI as their models tend to be complex, and convoluted [4]. And even if we unveil and understand these models better, there tends to be a greater risk involved — giving rise to the AI transparency paradox [5]. Exposing such information could result in vulnerability attacks.

There are also plenty of other associated (ethical) problems such as deepfakes [6] or copyright issues with genAI image models [7], but that would make for a lengthy article. The problem with genAI does not just stop with the models themselves. They also pose a risk to us. Over-reliance on such tools could result in stifling our creativity and critical thinking as we do not think as much, potentially under stimulating our brains neither do we verify sources, and ensure the response is factually correct. This could lead to a wider and rapid spread of misinformation.

But all hope is not lost — genAI can still be useful if used right. These tools can help in easing and speeding up our work, and assist in learning, provided they are not abused. They should complement human efforts. This can be done if we are conscious of their limitations and bias. We should take an active role in verifying genAI tools’ responses and exercise ethical awareness and responsibility. Maybe then, we will be able to overcome the risks associated with such tools.