Like any good system, we at Practo always have a database with a slave attached to it, and thanks to AWS RDS, we don’t have to worry about the data consistency across the master and slave database. This article aims to explain how to run a Query on the Slave Database with minimal code changes.

Before I dive into explaining the code, the application that we made this for is a Flask application with SQLAlchemy (https://www.sqlalchemy.org) as an ORM.

SQLAlchemy is a great library, that allows developers leverage the full power and flexibility of SQL. One of the most important features of SQLAlchemy is the Query and the Session API. In short, the Queryclass is a SQL construction object, where you can generate a SELECT query programmatically. The Query object is also attached to the Session, which acts like an agent that helps you communicate with your database.

The Flask application we work on has a convention where all the Model Objects return Query Objects instead of actual data. This is useful for two main reason — firstly, this allows you to add filters in the later stages of the code, and, you can lazy load data, since, the Query object acts as a generator. In simple terms, the query is only executed when you request for data.

Although you could just execute the query string on the slave database, it would not provide us with lazy loading and options like adding filtering in the later stages of the code. So we needed to code something where we can still return a Query object, except instead of running the query on the Master Database, it should run it on the Slave Database. For this, we made use of Python decorators.

So lets go through the code:

1. Create Session Object attached to the Slave Database:

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
slave_db_engine = create_engine(<URI for Slave Database>)
slave_session = sessionmaker(bind=slave_db_engine)()

2. Create the Decorator:

from functools import wraps

from app import slave_session


def convert_to_slave_query(query_function):

    @wraps(query_function)
    def create_slave_query(*args, **kwargs):
        return slave_session.query(query_function(*args,
                                   **kwargs).subquery)
    return create_slave_query

3. Use it in existing code (in this case, get_rick_characters):

from util import convert_to_slave_query

@convert_to_slave_query
def get_rick_characters():
    return Rick.query()

You can now easily use this decorator as a plugin and run queries on your slave database without having to go and edit existing code.

Generally, slave databases are used to run read-only queries. To avoid data manipulation to run on the slave database, you can use the following code:

DATA_MODIFICATION_LITERALS = [
    'update',
    'delete',
    'create',
    'copy',
    'insert',
    'drop',
    'alter'
]

def _check_query_read_only(query):
    query_literals = str(query).split(' ')
    intersection = [
        literal.lower() for literal in query_literals
        if literal in DATA_MODIFICATION_LITERALS
    ]
    return len(intersection) == 0

You can then change convert_to_slave_query decorator to:

from app import slave_session

DATA_MODIFICATION_LITERALS = [
    'update',
    'delete',
    'create',
    'copy',
    'insert',
    'drop',
    'alter'
]

def _check_query_read_only(query):
    query_literals = str(query).split(' ')
    intersection = [
        literal.lower() for literal in query_literals
        if literal in DATA_MODIFICATION_LITERALS
    ]
    return len(intersection) == 0

def convert_to_slave_query(query_function):
    @wraps(query_function)
    def create_slave_query(*args, **kwargs):
        read_query = slave_session.query(query_function(*args,
                                   **kwargs).subquery)
        if not _check_query_read_only(read_query):
            raise Exception("Invalid Operation on Slave Database")
        return slave_session.query()
   return create_slave_query

By doing so, you can be sure that only read queries run on your slave instance.

Although, you can now run queries on one slave database, this code does not allow you to run the query on multiple slave database, choosing the database using the Round Robin algorithm.

One more thing to keep in mind is that there is always some Replication Lag between Master and Slave instances of the Database, so it would be a bad idea to use this on any task that relies on latest data.

What’s Next?

Right now, the above code snippet supports reads only from one slave database. In future, this should be updated so that it supports n different slave database reads. Things like ReplicaLag, network proximity etc. can be used to select the correct database to run the query on.

Redirecting queries to Slave Database was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you are finding a doctor, booking an appointment or consulting a doctor online on Practo, a lot of work is done asynchronously using Job queue systems (a.k.a workers). This is a critical part of our architecture and it plays an important role in making our systems scalable and reliable.

We use AWS SQS as the message queue and we process on a scale of about 800 million jobs per month. These workers are run as a Kubernetes Deployment with every worker running as a separate pod. This helps to deploy, scale, monitor and troubleshoot each worker process independently.

This article focuses only on the “scaling part” of these worker deployments.

Implementation 1 — Using Custom Metrics and HPA

When we started with Kubernetes we used HPA with its Custom Metric Support and Prometheus to scale the workers. This is also the default recommended solution from the community for scaling worker pods.

Horizontal Pod Autoscaler(HPA) is a controller which scales Kubernetes Deployment typically based on CPU metric and supports scaling Kubernetes Deployment based on any custom metric. The only requirement is the user needs to make sure the custom metric is available for consumption by the Kubernetes process.

We exported the SQS Cloudwatch metrics to Prometheus by making the Prometheus Cloudwatch Exporter compatible for this use case. Here is the code for it — https://github.com/practo/cloudwatch_exporter

This was our first implementation of scaling our background workloads. We ran this for about a year or so. Based on the feedback from the various service teams and our experience, the following were the problems with this implementation:

1. Slow scale-ups: A lot of services required scaling to be near real-time. If there is a surge in the messages in the queue then the workers should scale instantly. But this was not happening because the Cloudwatch metrics are only available at 5-minute intervals for SQS. Solving for this required a combination of long-polling, SQS metrics API and Cloudwatch metrics.
2. Scaling down was causing issues: HPA does not allow to scale up and down based on different metrics. But it can be different based on the use case. For example, we were scaling up on SQS metric — ApproximateNumberOfMessagesVisible but we could not scale down based on the same metric because for high throughput workers when they were consuming the queue very fast this metric was always zero.
3. Non-idle workers were scaling down: Since it was a target-based scaling when the Queue length metric dropped it started to scale down the workers which were not idle.
4. Costs were not optimized: Since metrics were polled from Cloudwatch at fixed intervals and not intelligently as required, this incurred a significant AWS cost towards Cloudwatch. This can be optimized by fetching only the relevant metrics only when we require them.
5. Using custom metrics was not as straightforward as running a few kubectl commands and required a good effort from the user to make the whole setup work. Also, HPA was requiring us to decouple the metrics from the scaler which means we cannot derive the scaling metrics using the currently running pods already known to the scaler.

So to solve the above problems, we wrote Worker Pod Autoscaler.

Implementation 2 — Worker Pod Autoscaler

practo/k8s-worker-pod-autoscaler

Scaling workloads based on queue length is a fairly common use case. Therefore we decided to build a generic, performant and very easy to use open-source autoscaler for workers using Kubernetes CRDs. Following were the primary goals of this project:

1. Superfast scale-ups: Near real-time scaling of workers. As soon as a job comes in the queue the containers should scale if needed.
2. Solve the scale down problems for the workers: The solution should not scale down workers that are processing a job. Scale down metric should not be restricted to the same metric as scale-up, it should be based on the queue provider.
3. Built-in Queue Exporters: Kubernetes HPA with custom metrics requires org effort in exporting custom metric and storing them to be made accessible to the Kubernetes API. This makes sense for Kubernetes to support several use cases. But the current HPA implementation is restrictive to use for queue-based scaling. Also, built-in help in focussing the user worry about the application and frees the user from maintaining metric exporters. The project should build generic metric exporters (pollers) with every queue provider integration.
4. Support for scaling up from and down to zero pods to optimize costs.
5. Ease of Use: The whole setup of worker autoscaling should start working by running a few kubectl commands.
6. Generic Platform Independent Autoscaler: The autoscaler should extend to support any message queuing service and should work in any cloud or on-premise Kubernetes environment.
7. Open Source: The solution should be completely open-sourced for a bigger community to use, contribute and make it better.

Now, let’s dig into how Worker Pod Autoscaler aka WPA works!

Worker Pod Autoscaler

When the WPA controller pod starts it install the WPA custom resource in Kubernetes if it is not present and it starts the following go-routines:

• WPA Goroutine: WPA controller go-routine is the control loop which scales the deployment to the required number of replicas based on the queue specified for the deployment.
• Poller Goroutine(s): There is a poller manager who starts and stops go-routines (poller) for each WPA resource. Each poller polls the queue provider metrics for the exact metric keeping the API calls to minimum. The sync intervals here can be configured based on use cases. The concurrency here is one of the reasons for near real-time scaling.

The algorithm for deciding the desired replicas can be read here.

Explaining WPA working with a simple example.

Install WPA

Worker pod autoscaler can be easily installed using hack/install.sh. Please follow this for detailed information.

Example

Create a deployment and a WPA resource to manage the scaling of it.

kubectl create -f artifacts/example-deployment.yaml
kubectl create -f artifacts/example-wpa.yaml

WPA will start scaling example-deployment from 0 to 30 if the messages in the queue are greater than 150. The workers will scale down to 0 when there are no jobs in the queue.

That’s it! Two kubectl commands and your worker deployment starts scaling!

We have been running WPA in production from the last 4 months. We have got great feedback from our developers in Practo. It is now in v0.2.2 release.

Future Work

SQS is the only queue provider supported at present. But it can easily be extended to any other queue provider. This interface needs to be implemented and a simple poller like this needs to be written down to bring in a new queue provider support! Pull requests are welcome :)

Thanks to the Kubernetes team for making CRDs and sample controller. Please feel free to open issues in Github if you have any questions or concerns. We hope WPA is of use to you!

Launching Worker Pod Autoscaler was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWS Lambda Functions, Google Cloud Functions, Azure Functions are some of the serverless hosting options provided by the cloud providers. Some CDNs provide this option too, for example, Workers on CloudFlare. The added advantage of hosting your code on CDN is latency reduction, since the user is served by the CDN server closest to the user. That being said, serverless architecture might not be ideal for every use-case of yours. For your long running processes or applications with continuous traffic, cost would increase and so, you are better off spinning off a server to host your process or app.

In the micro-services and serverless era, if your infrastructure revolves around AWS, Lambda functions are one of the most popular and easiest options to host your micro-services or backend processes . Through Lambda functions, AWS enables you to host your code, without you managing the infrastructure (like an EC2). Lambda functions can be triggered by hooks like API gateway, S3 events or one of the many CloudWatch events available.

If your use case is solved by a single Lambda function, setting up the entire flow is a piece of cake. If not, your multiple Lambdas will have to be glued together. Let’s look at a hypothetical use case of automating the task of validating the key-pair attached to any new EC2 instance launched in your account and then notifying a concerned person, if it is an outdated/invalid key-pair. For the sake of separation of concerns, let’s have 2 Lambda functions — one to validate the key-pair and the second to notify. How we typically would implement this, is as follows.

1. Create a Cloudwatch rule to receive events of EC2 creations.
2. Create a Lambda function, let’s call it the-validator, and add it as a trigger to the rule created in Step 1.
3. If the key-pair is determined to be invalid, the-validator would push a job to an SQS queue (or an SNS topic or any other trigger for lambda).
4. The second Lambda function, let’s call it the-notifier, would be added as a trigger to the SQS/SNS of Step 3 and it would notify whenever there is a new event.

Though the above setup seems straight forward, the lambda-SQS-lambda pipeline would increase as and when we add more concerns to the flow. Step functions provide a graceful way to glue your lambdas together. Step functions is to lambdas, what conjunctions is to grammar. A Step function constitutes a state machine which holds the definition of how the Lambdas are inter-connected. Below is the state machine which would create a step function to handle the use case discussed above.

{
  "StartAt": "the-validator",
  "States": {
    "the-validator": {
      "Type": "Task",
      "Next": "the-notifier",
      "Resource": "arn-of-the-validator-lambda"
    },
    "the-notifier": {
      "Type": "Task",
      "End": true,
      "Resource": "arn-of-the-notifier-lambda"
    }
}

The above state machine and the lambdas are the only resources required to implement the discussed use case. This state machine is added as the trigger to the CloudWatch rule for EC2 creation events, which means, when a new EC2 is created, this state machine is executed. State machines are built to pass on the output of one step to the next. In our case, the output of the-validator would be passed to the-notifier, with no extra configurations to be done.

The above state machine can be extended to add more states when the concerns increase, eliminating the need to create complex workflows with SQS, SNS or any other glue in picture. Let’s say, a 3rd Lambda is added to our above discussed use case. This new Lambda, let’s call it the-db-handler, would save the invalid key-pair and the EC2 details in persistent storage. Here is our updated state machine.

{
  "StartAt": "the-validator",
  "States": {
    "the-validator": {
      "Type": "Task",
      "Next": "the-notifier",
      "Resource": "arn-of-the-validator-lambda"
    },
    "the-notifier": {
      "Type": "Task",
      "Next": "the-db-handler",
      "Resource": "arn-of-the-notifier-lambda"
    },
    "the-db-handler": {
      "Type": "Task",
      "End": true,
      "Resource": "arn-of-the-db-handler-lambda"
    }
}

The new Lambda and addition of the state to the state machine are the only changes required to update the flow making it very straightforward to add more functionality to your system. the-notifier, which was the end state of the state machine earlier, now specifies that the-db-handler is the next state and the-db-handler is the new end state.

Creating a workflow for your lambdas is one of the basic use cases of step functions. It also has a lot more useful features like conditional states, wait states, parallel states execution, etc. Though step functions isn’t a technical marvel, it does make the serverless experience better and is worth a try for your serverless workflows.

Serverless flows with Step Functions was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why migrate?

Web-pack is our static module bundler for modern JavaScript applications and has become a core part of how we bundle them.

🐌 Slow Build: The Main Reason.

We have a fairly complex app with lot of entry points, also each entry point can have multiple output files considering for i18n translations. Also, we have neglected our build process for some time now. Result? Our build was slow!

🏎How to Speed Up the Build?

1. Legato (Web-Pack 4) is claimed to be 98% faster. Migrating to Web-Pack 4 can itself can help you loose those extra seconds. 😅
2. Migrating exercise will also help you deeply understand the build process of your existing code base. It will give you ability to refactor and further optimise your web-pack build. At the very least, it will help you make a todo list of things you can improve on in your build.

After migrating to Web-Pack 4 and some easy refactoring, the build time is reduced by 60%.

From 159 seconds to 60 seconds! 🏎🏎🏎

How to Start Migrating to Web-pack 4?

• Read the change-log, as to know what has changed and what’s new. When migrating to a new major release, this should be the first thing to look for in any open source project.🕵️‍🕵️‍
• Know your existing web-pack build process inside-out, find out what entry, output, loaders, plugins are and how it is currently being used.

Before migrating, try to refactor/optimise your build code.

• Instead of one big webpack.config.js, it is better to separate all the functions used to say build-utils. All the constants in build constants.
• Separate the dev config from the prod config.
• If you have multiple apps (multiple entry points) in your codebase, in development one would ideally work only on one app at a time. So, make sure your dev build is only building that one app instead of all.
  You can write a simple function which return only one filtered entry point in development. Less stuff to build, faster your build! 🏎🏎🏎
• If you are doing i18n by generating a output file for each locale you have to translate. While developing, make sure you are not generating more than one output file. Less stuff to build, faster your build! 🏎🏎🏎

After you are satisfied with refactor and optimisations

• Add latest web-pack to your project. Currently it is 4.16.0.

yarn add webpack@latest --dev

• Add web-pack CLI.

yarn add webpack-cli --dev

• Provide Mode to web-pack.

Possible values for mode are: none, development or production(default). Providing the mode configuration option tells web-pack to use its built-in optimisations accordingly.

You will have to pass mode in npm script or add it in webpack.config.js

webpack --mode=production

OR

//webpack.config.js

 module.exports = { 
   mode: 'production', 
   ...
 };

Most probably you would have used CommonChunks plugin. ✋✋

• CommonChunks will have to be replaced to use the new optimization.splitChunks.

Now, the most important step!

• Set process.traceDeprecation to true this will help you to find which packages are using deprecated API

node --trace-deprecation ./node_modules/.bin/webpack

This can also be set from DefinePlugin

new webpack.DefinePlugin({
'process.traceDeprecation': true,
})

Most likely, the Node packages still using the old API will throw the below error.

DeprecationWarning: Tapable.plugin is deprecated. Use new API on `.hooks` instead

You can simply upgrade the package one by one

yarn add <package>@latest --dev

99% of the times upgrading will solve your error. It did for me! 😅

In that 1%, where just upgrading package to the latest version does not solve the problem, you would have to fix the issue yourself or raise an issue in the open-source project and wait till some-one else fixes it. 😥

If you liked reading this, don’t forget to clap. 👏👏

You can also follow me on on twitter @karanjariwala47 for java-script or react updates.

Thank-you!

Migrating to WebPack 4 for Beginners was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Redux Saga is a library that aims to make application side effects (i.e. asynchronous things like data fetching and impure things like accessing the browser cache) easier to manage, more efficient to execute, simple to test, and better at handling failures.

— Redux Saga Docs

Today, we use Redux-Saga heavily in our enterprise application. Most of the logic and all of the side-effects are handled by our Sagas. Our Components and Reducers are mostly dumb.

We treat the root Saga like a separate thread in our application. Any action that can lead to a side effect is handled by our saga thread.

From a testing perspective, the main advantage of this architecture is that all our side effects and logic conditions are in one place i.e Sagas. Testing Sagas alone would give us much greater confidence than testing reducers or components. So, testing Sagas is an imperative for us.

There are two ways to test sagas.

Stepping through each yield in Saga one by one

With this approach, you will step through yield effects one by one. Typically, in the exact order by asserting the effect objects at each step. This is like unit testing your sagas. Though straight forward, this will make your tests brittle. As your tests are tightly coupled with the implementation and also the order of effects. Any refactoring or restructuring will most likely break your tests.

So, we decided to go with the second option.

Testing not how a saga is implemented but only the effects that you’re interested in.

Here you will run the saga to the end, mocking select and call effects and asserting the put effects you are interested in.

This is more towards integration testing. As you are not running your saga effects in isolation one after the other, this type of test should give you more confidence.

The main benefit, however, is your tests are not Brittle. They are not tightly coupled with the implementation. Example: If you re-order your effects or you add one unrelated put effect for a corner case, it should not break your test.

While both of the above testing methods can be written natively, there exist several libraries to make both methods easier. For testing each generator yield step-by-step there is redux-saga-test and redux-saga-testing for option one. redux-saga-tester, redux-saga-test-plan and redux-saga-test-engine for option two. You can also unit test with test plan and tester.

— Redux Saga Docs

We decided to go with Redux-saga-test-plan.

Let’s test a basic saga worker which is written for the requirements below.

If user is not logged in, he should be redirected to login.

If user is logged in and does not have a right to open the Modal, he should be denied access.

If user is logged in and has the right to open the Modal, Modal opens.

Our TriggerModal Saga worker when triggered by an action selects the UserId and the ModalRight from the Redux store using the select effect.

If UserId is not falsy, it means user is logged in.

The ModalRight is true, if he has the action right and false if he does not.

We have to test our saga worker in three different redux states where:

1. User is not logged in

2. He is logged in but does not have the right

3. He is logged in and has the right.

Our three Mocked states look like these.

We intend to test the behaviour not the implementation.

For the above three states we have three cases.

For all the three cases, we run the saga and pass the corresponding redux states. We then mock the select effects and assert what the put effects should be.

We are only asserting what comes out of saga in a particular state of application there by testing only the behaviour and not the implementation.

Here, even if you change the order of select effects in the saga or add a new independent effect it will not necessarily break the Test.

Also after .run. you could do .then and assert with even more control on values of effect objects.

For example, in each of the above cases we know only one put should be dispatched. We can assert the same using .then as shown below

Read More about testing sagas using redux-saga-test-plan here

If you liked reading this, don’t forget to click 👏 and add claps for the post. You can also follow me on on twitter @karanjariwala47 for java-script or react updates.
Thank-you!

Testing Redux-Sagas with a plan. was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

It takes a lot of efforts and time to build and launch an app. It takes even more efforts to make users install the app. If users are leaving your app, you would immediately want to know why.

The main motive behind tracking uninstall rate is to figure out what made users delete the app. Uninstall tracking can also help us identify users who uninstalled the app and retarget them to install the app again or even to ask the for feedback. This tutorial will help you track uninstalls and find events performed before the uninstall happened using Firebase.

Firebase automatically tracks ‘app_remove’ events when an application package is removed or “uninstalled” from an Android device. But the problem with Firebase console is that it only shows the number of events. It does not share the user’s demography or events performed before the uninstall happened.

To find users demography and events performed before uninstall, we need to export all analytics event from firebase to BigQuery. Linking BigQuery to Firebase app lets us access our raw, unsampled event data along with all of the parameters and user properties. To transmit Firebase data to BigQuery, you must upgrade your project to the Blaze plan. We’ll talk more about costs later in the costs section.

Setup

First of all, we’ll have to integrate firebase with our Android project. Follow the instructions mentioned here to add firebase to your Android Project. Add Fireabase analytics to your project and start tracking events which you would like to track before the uninstall happens.

Do not forget to set user properties in analytics events. It’ll help you identify users who uninstalled the app and retarget them.

Step 1:

Upgrade your Firebase to Blaze

Step 2:

Link BigQuery to Firebase. To link BigQuery to your Firebase project:

• Sign in to Firebase
• Click the Settings icon and select Project Settings
• On the Project Settings page, click the Integrations tab
• On the BigQuery card, click Link
• Once linking is complete, find the BigQuery card in integration tab, click on Manage
• Under Firebase analytics flow, turn on the flow to export data of your app

It will take some time to export start exporting data from firebase to BigQuery. Once your data is loaded into BigQuery, you can run SQL-like queries over it using BigQuery. Here is the sample query which we ran to analyze the events before the uninstall happened. For more information, see Querying Data in the BigQuery documentation. If you have added user properties in your analytics events, you’ll be able to identify users who had uninstalled the app and will be able to retarget them

Step 3:

Query raw analytics evens in BigQuery. Here is the sample query to identify users who uninstalled the app on 2018/12/03

There’s a lot more stuff you can do by querying your raw data. Here is another sample query to find the last two events performed by the users before uninstalling the app.

Cost

As we discussed, to transmit Firebase data to BigQuery, you must upgrade your project to the Blaze plan. Blaze plan is a postpaid plan, you’ll only pay for what you use. No fixed monthly charges either. For more information, see Firebase pricing.

In addition, storing and querying data in BigQuery incurs a fee. For more information, see BigQuery pricing.

To reduce data storage costs even further as daily event tables accumulate, delete older tables or move them to Cloud Storage.

How much does it cost us?

Well, less than $1.00 per month💸💸. We log around 70 events, 3 user properties and store only last 6 months data and we do not use any other paid feature of Firebase Blaze plan

It takes time to build an app that everyone likes. Until then, we have to keep a watch on users’ actions within our app and discover the problem which led to uninstalls and of course fix that problem.

Track Android uninstalls for less than $1 a month was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

On a merry Saturday evening you suddenly come down with a fever. The shivering is incessant and nausea is building up too. Things could have been better but they are not bad either, until someone brings up the Swine Flu outbreak in the city. You start panicking. A cab is booked for the nearest hospital but you are stuck in the infamous traffic.

Anxiety is through the roof so you search the internet about your symptoms. Your heart sinks when internet informs you of the possibility of a terminal disease. You search again and find Practo’s online consultation service. 3 clicks and 2 minutes later you are talking to an actual Doctor, who tells you that your fever is fairly ordinary (terminal only if a bear eats you). You reach the hospital with far less anxiety and a lot more reason.

The above is not a thriller script. It is just a dramatized version of a real-life account from a real Practo user.

Now you probably have 2 questions

FIRST: What is Practo?

Practo is a noble idea that took shape in 2008, in a college dorm in India, when one of the two founders couldn’t share his father’s medical reports with a doctor in the US.

Ever since then, Practo has devised numerous solutions for doctors and patients to make their lives easier. Following are just a few of their many offerings:

1. An online doctor appointment booking platform
2. Online consultation through text, call or video chat
3. Diagnostic tests with home sample collections facility
4. A slew of software for clinics, hospitals, and multi-chain clinics

[Visit practo.com for a complete rundown on all of her products and services.]

SECOND QUESTION: How do you consult online?

Practo offers an online doctor consultation service where patients can talk to specialists across many specialties.

It’s the next best thing to visiting a doctor physically.

Less than 2-min response time | In-chat report sharing | 100% secure

“How do you ensure 2-min responses?”

The first step of initiating an online consultation requires patients to elaborate on their health concern. Our automation system predicts the problem area and the related speciality. Later they proceed to pay the marginal consultation fee, after which our algorithm kicks in. It searches our database for the doctors who meet certain minimum criteria.

e.g. instant online availability, feedback received from previous consultations, general response time, experience in the field etc.

Next, we send notifications to these doctors on our Provider app about the potential consultation. These notifications are sent to different but relevant group of doctors until one accepts the request. In approximately 97–98% of the cases, one of the doctors in the group accepts the notification and directly connects with the patient.

Since the doctors accept the consultation directly, it implies that they can reply to the user instantly and in most of the cases, within a minute. In case the doctor is unable to reply to the patient’s messages, our system sends prompt reminders through repeated notifications.

Thousands of people use this feature to find solutions to nagging health problems. So, we work day and night to make it even more life-like. Recently, we have added video chat to the text so that the patients can literally see a doctor. (The video and audio calls in the text are free and the user don’t have to pay any extra amount).

Practo was ground up by some of the best people in the industry. And, its online consultation service has seen a hockey-stick growth in the last 2 years, and the credit for that goes to the Product, Engineering and Operations team working together day and night to give the best UX/UI to the anxious consumer (patient).

Want to give it a try? Just hit the following link:

Ask A Doctor Online And Get Answers 24X7 | Online Doctor Consultation | Practo

Curious to understand the nuances of the feature? If you are in Bangalore and around JP Nagar, just ping me. I will be happy to indulge in a techie chat.

Disclaimer: The algorithm is a copyright of Practo and completely confidential. Excuse the spy- level discretion employed in the article. However, if you have more ideas and would like to use them to make the world better and healthier, join us!

Special Note: Thanks to Mahima Singh for taking time out and almost co-writing the article.

Connecting you to The best Doctor in 60 seconds: Demystifying Practo Consult was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: In this post, I will be stressing on the word decentralised due to the fact that most of the teams/projects use centralised solution for key distribution and we will look into why centralised solution can create a havoc!

Lets start with the few reasonable assumptions

• We all know what is inter-service communication since almost everyone has adopted or read about service oriented architecture and amazing benefits it brings to a system.
• We have encapsulated micro-services (Ideal scenario) which communicates with each other over some protocol (Such as HTTP, RPC, WebSockets, MQTT, and AMQP).
• We use a secure channel (such as HTTPS) for communication.

Now lets reiterate the problem at hand and the probable solutions.

Though the communication within the channel is secured i.e. HTTPS channel, if anyone is aware of the micro-service resource URL then a simple POST request can alter the state of the system or a simple get could leak the data i.e. core to an organisation and we just can’t afford that.

The above problem can be solved in many different ways and lets concentrate on the prominent ones.

• Filter traffic to the service
• Pre-establish shared secret between micro services.
• Centralised key store service/database where every other service is connected to fetch the latest secret for encrypting the data.
• Decentralised key establishment between micro services with TTL.

Now lets evaluate each of the solutions and try to come up with the best possible way for secured inter-service communication.

Filter traffic to the service

This can be achieved using an IP whitelist or by deploying the application in a private cloud where external access is restricted etc.

This is the first level of defense for every application and creates a wall between trusted network and untrusted internet. But, this approach falls short if different micro-services have different levels of security and importance. If one micro-service is compromised, it should not bring down the whole system along with it.

Pre-establish shared secret between micro services

Key shared over a file is not safe for long (Mainly due to human involved while key establishment), so we need to rotate these keys with new ones often. Whenever new keys are established that may requires a deployment and/or a planned downtime which isn’t a bad solution but it’s just inefficient and resource (human and the rest) intensive over time and we need to stop relying on human presence for key establishment.

Centralised key store service

After looking into the pros and the cons of above mentioned solutions, the first viable solution that comes to our mind is to build a key exchange service where every other service communicates with to fetch a shared key between any two services.

Meanwhile, we have to make sure that the new important service is secure by making this service inaccessible from the outside world while also adding additional security features such as key expiry.

Well, this is a solution! but is this a good one?

We are creating a centralised system where every service is dependent on it and all the time, which means devastating single point of failure and it doesn’t stop there, we are keeping all our shared secrets/keys at one place which is really not a great solution.

Decentralised key establishment

What does this mean? In simple terms each service pair should be capable of establishing a secret and rotate them whenever it must without human intervention or downtime. Now lets work towards how to achieve the same.

Lets decentralise! “key establishment”

Now that we have understood that decentralised shared key exchange is a good problem to solve, lets Google “shared key exchange” (Well bing it or use any of your fav search engine).

The first thing you will notice is one algorithm that is mentioned over and over again i.e. Diffie Hellman algorithm

Fact: No it is not created by Diffie or Hellman, it’s theorised by computer scientist and one of public key encryption inventor Ralph C. Merkle and named after two famous cryptographers Whitfield Diffie and Martin Hellman.

You can find a simple and easy to understand explanation for Diffie Hellman here. I am reiterating the first and important sentence below for quick reference.

In simple words, Diffie Hellman is a way of generating a shared secret between two people/system in such a way that the secret can’t be seen by observing the communication.

Important distinction: You’re not sharing information during the key exchange, you’re creating a key together.

There is a always a catch!

Once we decide to use any new approach for a problem at hand, we tend to wonder what can be the drawbacks of this approach? And yes, there are drawbacks of using DH as it takes lot of resources (i.e. time an important factor and computing power though computing power is a worry only for small devices in IOT) to create 2048 bit key pair (since 1024 bit keys is not safe anymore and Java 6 open source version doesn’t support DH > 1024).

As always, we have a good alternative — ECDH.

ECDH is better performant than DH and you can find a lot of great resources about it. The philosophy/design behind the DH and ECDH are the same but they achieve results using different algorithms. Elliptic curve cryptography in short ECC has provided a great means to improve the performance of many Cryptographic algorithms and one such improvement is ECDH over DH.

For instance 2048 bits DH key generation in Python 2.7 takes more than 30 seconds and the same takes around 10 seconds on NodeJS while the same level of Security can be achieved by ECDH using a curve NIST P384 which takes less than 10ms to generate public private key value pairs across programming languages.

(hardware used: Mac Pro 2015 with i5 processor and 8 gigs of RAM).

Hey! most of the programming languages already supports ECDH.

Yes, A major problem is resolved by the programming languages by providing good cryptographic modules so that you don’t have to worry about implementing DH yourselves. But not all programming languages follow the same interface for e.g. NodeJS crypto methods encodes and expects Public and Private keys in 'latin1', 'hex', or 'base64' formats whereas Python and Java works 'DER' or 'PEM' encoding schemes.

It doesn’t stop there, now we have to worry about storing this established secret since we can’t afford key exchange for every request.

Key establishment for each communication session is referred to as perfect forward secrecy for which we have DHE where E stands for ephemeral and being used by most of the your trusted tools such as TLS, SSH, ECDSA etc.

Since the applications are deployed in a horizontally scaled environment, we need to store the secret in some secure place and we need a way to fetch it for every request which is not solved by these Standard libraries. Hence we are defining a way to generate secrets, store and access the established secrets and expire with TTL (Time to Live) support across heterogenous setups and across multiple programming languages.

Well it’s not all over yet, we have one drawback with DH/ECDH which is man in the middle attack. Consider you are trying to establish a communication channel between your Service A and and an external Service B, while a prankster Service C poses as Service B in which case you have no way to differentiate the services in an untrusted environment. That is why we are limiting the discussion to Inter-service communication in this article, but there are ways to establish communication with untrusted parties but again over a trusted channel or interacting with trusted third party services.

ECDH, ECDSA, etc. ECC based DH algorithms are being used by your browser for SSL handshakes, OpenSSL and for most of the secured communication over the Internet.

Lets dig into the implementation!

Now we have arrived at conclusion that we will be using ECDH for key establishment. But are there any standards? Yes, there are standard curves created for this purpose and you can find these safe curves here.

1. Use ECDH key exchange for generating a temporary shared secret valid for a short duration and storing it in a central datastore that supports expiration.
2. Use Request-In-Request and not request-response mechanism during the ECDH key exchange for establishing the trust (via Domain validation) for two-way authentication.
3. At the end of the ECDH key exchange, services maintain a list of non-secret identifiers of the temporary shared secrets and the expiration timestamp against the service so that they can reuse an existing valid secret if available.

I get the Gist but I need more than that.

Step 1: Service A generates its ECDH asymmetric key pair and an UUID and store it in a key-value store preferably with support for TTL like DynamoDB or Redis for few seconds (Configurable depending on application preference) with the UUID as the key and the value comprising of the following:

• ECDH curve, Generated private key, Generated public key, Service Info (B in this case)

Step 2: Service A makes a direct HTTPS request to Service B with the following details:

• `POST <optional-prefix>/_handshake-1`
• `curve` — ECDH curve used
• `ecdh_pub_key` — Public key generated by service A
• `source service` — Identity of Service A
• `UUID` — Unique identifier for the request

Step 3: Service B generates its ECDH asymmetric key pair using the curve provided in the request.

Step 4: Service B makes a direct HTTPS request to Service A passing the following details:

• `POST /<optional-prefix>/_handshake-2`
• `pub_key` — Public key generated by service B
• `UUID` — Same UUID provided by Service A
• `source service` — Identity of Service B

Step 5: Service A retrieves the data stored initially based on the UUID using a strongly consistent read operation.

Step 6: Service A checks that the service passed in the second request matches the value stored in the datastore.

Step 7: Service A generates the temporary shared secret by combining the public key of service B and stores it in a data store preferably with TTL support for a pre specified duration with the UUID as the key (overwriting the previous entry that was valid for few seconds).

Step 8: Service A responds back to service B’s request with a success response

Step 9: On receiving a successful response, service B combines the private key of B and public key of A to generate the temporary shared secret and store it in its data store for a some duration (As per application requirement) with the uuid as the key and the value comprising of the following:

• Temporary shared secret, Service Info (Service A in this case)

Step 10: Service B records the uuid and the expiration of the shared secret against the service A for use in the future in its relational database with a composite index on (service, expiration) columns.

Note: In this approach, we are using relational database to store service, and expiration info against UUID and using Key value database with TTL support to keep the UUID with generated secret which removes single point of failure in terms of Data leak.

Step 11: Service B responds with success for the initial request

Step 12: On receiving a successful response, service A records the uuid and the expiration of the shared secret against the service B for use in the future in its relational database with a composite index on (service, expiration) columns.

Wait! What happens when i have multiple keys and how do i choose one?

1. Use non-secret identifiers (UUID) to communicate which temporary secret was used to sign for the actual communication.
2. And while checking the signature, if the shared secret is not found respond with a different error message than when the signature matching fails so that the client can retry after creating a new temporary shared secret.

And ta-da we have created the mechanism to establish key exchange in a decentralised way across services with a mechanism for storing and expiring shared secret key.

Hope you enjoyed this post. I am ending this article with a quote from Bruce Schneier:
“The fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security.”

Thanking great resources that helped me create this post

1. https://security.stackexchange.com/questions/45963/diffie-hellman-key-exchange-in-plain-english
2. https://technology.amis.nl/2017/07/04/ssltls-choose-cipher-suite/
3. https://www.internetsociety.org/blog/2015/10/diffie-hellman-key-exchange-problems-recommendations-for-stronger-encryption/
4. https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work
5. https://crypto.stackexchange.com/questions/12823/ecdsa-vs-ecies-vs-ecdh

Special thanks to Kishore Kumar for coming up with this great approach for solving key establishment problem for inter-service communication.

P.S: We are working on libraries across programming languages and we are hoping to make it open source as soon as possible.

Thank you!

If you liked this article, please hit the 👏 button to support it. This will help other Medium users find it. Share this on twitter to help out reach as many readers as possible.

Inter-service authentication and the need for decentralised shared key exchange was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This was somewhat similar to developers complaining about slower execution speed of Python (addressed in https://medium.com/practo-engineering/execute-python-code-at-the-speed-of-c-extending-python-93e081b53f04) and some others complaining that Python and other interpreted languages are not truly “concurrent” ( addressed in https://medium.com/practo-engineering/threading-vs-multiprocessing-in-python-7b57f224eadb).

Because of Python being open source and ultra flexible , there is almost nothing which you can achieve with other languages and not with Python.

Below snippet will highlight the issue of function overloading in Python.

So when you define two functions with same names, the last one only matters ( area(size) in above). Thus area(3) works but area(4,5) gives

TypeError: area() takes exactly 1 argument (2 given)

You can always print locals() symbol table which is a data structure maintained by a compiler containing all necessary information about the program. This stores like below:

{‘area’: <function area at 0x102231398>, ‘__builtins__’: <module ‘__builtin__’ (built-in)>,…}

In symbol table, key is function name here. Thus when another function with same name is encountered, in symbol table current key value pair is overridden. Hence, by default function overloading is not available.

But by using decorator design pattern in python, function overloading can be implemented. Below is code snippet to implement function overloading. Explanation will follow later.

In above, we have implemented decorator called overload. This is special case of decorators with arguments. These arguments to decorators are the key elements of function overloading since they specify the type and number of arguments.

Due to the decorator overload, call to area function resolves as below:

area (size ) resolves to overload(int)(area)(size)

and

area (length,breadth) resolves to overload(int, int)(area)(length,breadth)

Our main aim is to maintain a typemap dictionary. Here key is types which is a tuple of arguments to decorator and the value is function object.

self.typemap[types] = function

Let us see typemap in our cases.

overload(int)(area)(size)

results in :

{(<type 'int'>,): <function area at 0x102231938>} (actual)

{(int,): <function area at 0x102231938>}(for sake of explanation)

and

overload(int, int)(area)(length, breadth)

results in:

{(<type 'int'>, <type 'int'>): <function area at 0x1022318c0>} (actual)

{(int, int): <function area at 0x1022318c0>} (for sake of explanation)

Lets do deep dive to implementation of overload decorator function. From above, its straight to figure out that outer function accepts arguments and inner function accepts function object.

In above code, for area (length,breadth) which resolves to overload(int, int)(area)(length,breadth):

types = (int,int) and function refers to area function object.

From function object, we can get name by function.__name__

mm = MultiMethod(name)

MultiMethod(name) will now call class MultiMethod with function name (area in our case)

Thus self.name becomes “area” and self.typemap = {}

after this decorator calls mm.register(types, function)

types = (int, int) or (int,) for our cases

Note: I am using int for (<type ‘int’>) for the sake of easy explanation.

function refers to function object.

The register function will generate typemap now as below:

{(int,): <function area at 0x102231938> , (int, int): <function area at 0x1022318c0>}

The function decorator overload returns register (inner function object) which in turn returns mm object.

So what happens when actual function call happens:

Case1:

• area (3 ) resolves to overload(int)(area)(3)
• overload(int)(area) returns register which in turn returns mm.
• overload(int)(area)(3) thus calls mm(3)

Interesting since mm was an object of MultiMethod class. So calling mm(3) is like calling object with some arguments. But yes, in Python you can have callable objects if class defines __call__ special function.

Lets see the __call__ function of MultiMethod class in detail.

when object is called, __call__ method is invoked.

Thus mm(3) means args = (3,)

From the passed arguments tuple of the type of variables passed is generated using:

types = tuple(arg.__class__ for arg in args)

To be noted that __class__ on any object gives its class.

Example:

>>> a  = 34
>>> a.__class__
<type 'int'>
>>> a = 45.6
>>> a.__class__
<type 'float'>
>>> a = "test"
>>> a.__class__
<type 'str'>

Now the invocation is straightforward.

function = self.typemap.get(types)

since our typemap looks like

{(int,): <function area at 0x102231938> , (int, int): <function area at 0x1022318c0>}

• in case of mm(3) types = (int,)

thus function area at 0x102231938 will be invoked. (area(size))

Case2:

area(4,5)

• area (4,5 ) resolves to overload(int,int)(area)(4,5)
• overload(int,int)(area) returns register which in turn returns mm.
• overload(int,int)(area)(4,5) thus calls mm(4,5)

when object mm is called, __call__ method is invoked.

Thus mm(4,5) means args = (4,5)

and type becomes (int,int)

Thus <function area at 0x1022318c0> gets called (area(length, breadth)).

Hence, by above strategy we can implement function overloading in Python.

Finally, actual call happens through

function(*args)

Between, you can code it once and use everywhere since this is implemented as decorator. So once you have written the code , you never need to write it again, rather you can just use decorator as and when needed.

You can clone this git repo https://github.com/ppundir/overload and use it to implement decorator in your code without knowing the inside details also.

Steps:

1. git clone git@github.com:ppundir/overload.git
2. python setup.py install

usage:

If you liked this article, please hit the 👏 button to support it. This will help other Medium users find it. Share this on twitter to help out reach as many readers as possible.

Function Overloading in Python was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

The main idea for these tests is to build fast and fail fast. With our Practo mobile apps, we always aim to thrive with a data driven approach to optimise user solutions and these tests are the core of this approach now. We use Firebase for our A/B tests for both Android and iOS.

What did we need an A/B test for?

Before starting the experiment, the Practo app had four tabs —

1. Home tab — The home for all the transactions
2. Todo tab — List of all actionables and “todos” for the user
3. Read tab — A place to read about health queries and health articles
4. Profile tab — Everything about the user

Since the launch of Todo cards, we have a lot of faith in the product and the data has been equally corroborating. So we wanted to offer it to the users right on the first screen, like this —

We had an assumption that putting todo cards on home can give over the top success for the product, and equally happy users. So we decided to run an A/B test to concretise the decision.

How MVVM actually helped

As mentioned above, the main idea of A/B tests is to build fast and fail fast. Which means put the code there fast, fix the bugs fast, and make sure the features don’t break in any of the variants. For a fair test, you would not want any disparity of your feature support between your release and patch cycles.

For the brevity of this post, I will not try to explain some of the best practices that should be followed for writing MVVM, but I will advocate how it can accelerate both business and development.

For one of the variants, todo cards is a separate tab on our home screen. So we have a fragment, and a viewmodel for the todo tab on home, i.e., TodoHomeViewModel. The fragment is nothing but a container for a recycler view sugar coated with some cool UI. And the responsibility of the view model is to create a list of todo cards that the fragment can display. The viewmodel has a bunch of observables and livedata for databinding, and recycler adapter’s data. But the main sorcery happens within these functions -

The fragment subscribes to the Maybe returned by the viewmodel, and gets the todo cards from the server. The viewmodel then creates different viewmodels for different cards. The observable list is updated with these viewmodels. And the list is observed to spit the cards in the recyclerview. Everything is working. Unit tests are running. Life is good!

For convenience, the AppHomeViewModel(the viewmodel for home) is also structured in a similar way. There’s an observable list being observed by the app home fragment, and functions that put data into that list.

Now comes the second variant, where the todo cards are on the home tab and the todo fragment does not exist. So how can MVVM help here? With this implementation that we have here with a bunch of observables and a list being observed, putting the results at one place hardly appears to be a puzzle now. All we have to do now is to instantiate the TodoHomeViewModel in the AppHomeViewModel, make it fetch the cards from the server just the way todo fragment does, observe the response and append the cards to the list.

And the updated tests —

Everything is working. Unit tests are running. Life is good!

Most importantly, time taken to get the A/B experiment out on production = Time taken to instantiate the todo home viewmodel in the app home viewmodel + Time taken to observe the cards sent by the server + Time taken to append those cards to the list of segments. A big time saver for the business. A win-win for all.

And to put this all in picture —

Why MVVM

Well you can still live with copying and pasting the same code for two different variants of the app (I would like to argue against that though). Or put everything in huge methods and wrap everything in a lot of if’s and else’s and things would still work. But things can get, and will get complicated when you have more than 2 variants, or multiple tests running at the same time. The business logic will get lost in the bulk of control flow logic and code maintenance, scalability and debugging will go for a toss. And you will find yourself praying to stop any more variants or tests coming in your code. After adding a few more variants on top of that, it will start looking like jenga blocks about to fall.

MVVM framework helps in maintaining single responsibility principle and helps in writing easy communications between different components via observer objects, mediator objects or some other behaviour that suits the case.

To wrap things up, investing in MVVM gives us some cogent advantages:

1. Zero code duplication — No need to rewrite blocks of business logic again and again.
2. Separation of concern — Individual features, or view models have the responsibility of (only) their own experiments.
3. Maintainability and scalability — The code will look a lot less daunting and more breathable. Adding more experiments will not take much time. (Remember: build fast, fail fast)
4. Maintains testability — This is by far the favourite gains we have had from writing in MVVM involving multiple experiments. There are little to no changes in the unit test cases for each individual view models. Since, a lot of our experiments were UI-centric, our unit test suite remained almost the same. So along with building fast and failing fast, we could also maintain the sanctity of our code.

Writing or rewriting your code in MVVM might feel like a needless or a redundant task at first, but it really pays you in the future. It does not take much time getting used to, and the gains are surely worth it, to — Build fast, fail fast.

If you liked this article, please hit the 👏 button to support it. This will help other Medium users find it. Share this on Twitter to help out reach as many readers as possible.

How MVVM helped us run A/B tests quicker was originally published in Practo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.