Assuming that you scanned this and know a bit of the async stuff.

async

@route(“/users/info”)
async def userInfo():
    user = await fechUser() #(some http request to get users info)
    return process(user)

As we used async, fastapi will run the function for each request on the same single event thread.

As there is an await for a response, fastapi will use this time for serving the other requests. And those requests also wait for response from the external service and hence all of them are served concurrently.

So in this case as most of the time, a request to the route waits for a response from some external service and no single request is taking a long time doing something on the single thread they are running, its fine to use async as fastapi can serve other requests which come at the same time while some of them are waiting.

When we use await how fastapi puts the request in a backlog and serves another request while the response for the first request comes back from the external service depends on the framework implementation.

async with blocking code

@route(“/heavyTask”)
async def userInfo():
    time.sleep(10)

Similar to the above, all the requests use the same event thread for the function execution.But in this case, the code is sleeping for 100 seconds(or consider doing some heavy maths calculation).

Now until the first request completes (10 seconds), the other requests will be waiting and they will be served only when the first request is done.Not only for this route but any request to any other route also gets blocked.Why ? Because the event loop ie. the single thread which executes async functions and also schedules execution of all the requests is blocked on sleep.

Even though you used async, as your code has a blocking operation, all the requests that come after the current served request wait in queue and get blocked.Oops!

So just be sure that any operation down the line is not blocking.

sync

@route(“/heavyTask”)
def userInfo():
    time.sleep(10)

Now similar to above, here we have just changed from async to sync and hence the function userInfo for each request is executed in a separate thread pool and hence one of them doesn’t block the other.

Let's consider the below code

import time
from fastapi import FastAPI

app = FastAPI()

@app.get("/async/sleep")
async def async_sleep():
    time.sleep(100)
    return {"message": "slept for 100 secs"}

@app.get("/sync/sleep")
def sync_sleep():
    time.sleep(100)
    return {"message": "slept for 100 secs"}

@app.get("/async/hello")
async def async_hello():
    return {"message": "hello"}

@app.get("/sync/hello")
def sync_hello():
    return {"message": "hello"}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Test 1:

Requests:

1. http://localhost:8000/sync/sleep

2. http://localhost:8000/async/hello

3. http://localhost:8000/sync/hello

Access Log:

INFO: 172.17.0.1:45992 — “GET /async/hello HTTP/1.1” 200 OK
INFO: 172.17.0.1:45992 — “GET /sync/hello HTTP/1.1” 200 OK
INFO: 172.17.0.1:45968 — “GET /sync/sleep HTTP/1.1” 200 OK

Summary:

As you see, we have launched 1, 2, and 3 requests one after the other. The 1st request is for a route which sleeps for 10 seconds.

From the logs, you can see that we get responses for 2 and 3 instantly. We get a response for 1 after 10 seconds. (the logs don’t capture the timestamps and hope you get what I meant when you test it)

As we use def for the sync/sleep route, the request is run on a separate thread pool and other routes are not impacted.

Test 2:

Requests:

1. http://localhost:8000/async/sleep

2. http://localhost:8000/async/hello

3. http://localhost:8000/sync/hello

Access Log:

INFO: 172.17.0.1:46064 — “GET /async/sleep HTTP/1.1” 200 OK
INFO: 172.17.0.1:46066 — “GET /async/hello HTTP/1.1” 200 OK
INFO: 172.17.0.1:46116 — “GET /sync/hello HTTP/1.1” 200 OK

Summary:

The 1st request is for a route which sleeps for 10 seconds but we have used async for it.

From the logs, you can see that only once the 1st request is completed, the 2nd and 3rd are served and processed.

As we used async def for the async/sleep route, the request is run on the same event loop and hence blocks all the other async operations.

Even the 3rd request i.e. /sync/hello is also blocked because the event loop which schedules the execution of requests on the thread pool is blocked by the async sleep function!

So haha, there is no one to schedule the 3rd request on the thread pool though it does not run on the event loop thread.

Now, there are still like which I did not yet come to terms with.

1. How does sync + async work

2. What about def mutating the global state, does it create race conditions

3. What does this mean from here. The initial trigger point is a request itself and from which the route operation gets invoked and which might go down the route with one function calling another etc. Like we don’t explicitly invoke a utility function right?

Any other utility function that you call directly can be created with normal def or async def and FastAPI won't affect the way you call it.

This is in contrast to the functions that FastAPI calls for you: path operation functions and dependencies.

If your utility function is a normal function with def, it will be called directly (as you write it in your code), not in a threadpool, if the function is created with async def then you should await for that function when you call it in your code.

https://github.com/tiangolo/fastapi/issues/2619#issuecomment-762495981
https://github.com/tiangolo/fastapi/issues/603
https://news.ycombinator.com/item?id=25992078
https://gist.github.com/crackerplace/c853fe41d66045652a4421654098f0f8https://gist.github.com/lukin0110/0074ec5325224674010193bb95f8b835

My Use Case:

I have an instance with a certain configuration(x cores and y gb ram) and I want to test how much load an application deployed on this instance can take.

Application A: A standalone application with no external API interactions. So given a request it should respond. It's an ml app to be specific which does generate certain recommendations.

Expectation:

How much load the application can take?

or

How much load does an application take and serve within a reasonable time?

Here I would refer to load as the request per second(RPS) that the app can handle.

Assume you have a 1 core machine and the app is a CPU-intensive app such as some number crunching.

So if the app takes like say 1 sec to serve the request, the RPS is just 1req/s.

What if you have 2 cores, as the app was in Python and as Python is single-threaded, even if you have 2 cores, it doesn't help. So you need to spawn 1 more process to utilize the other core. So now multi-process thing is dependent on which framework you are using.

So now, we are serving 2 req/s.

Now let's say your app takes 2 seconds to serve a request and we have 2 cores.

Now your request rate is 1 req/s as 2 requests take 2 secs to complete in overall.

But we are missing a point here.

What happens if we increase the number of requests assume we are send 8 requests now at once.

So the overall time taken for all the requests to complete is 8 seconds..Some of the requests take more time waiting for CPU time as we only have 2 cores at a certain moment for computation. So in the worst case it takes 8 secs for the last two of the 8 requests.

So just increasing the load on the application for testing, will not help much as the system is already saturated and hence even if you increase the load there will not be much improvement in the request rate. And worse the latency starts to increase as requests have to wait for more time to get a chance of CPU time.

That brings up saturation. So load testing is also about trying to understand the saturation limits of the system beyond which it will not scale and deteriorates.

Req/sec and latency are 2 important metrics to keep in mind.Req/sec as an independent metric is not much useful. So ok your app can serve x req/s within y time? That is a bit more meaningful.

We were occasionally seeing the below errors in our app logs when some queries run on the Postgres replica.

“Canceling statement due to conflict with recovery”

Background

There are many reasons why conflicts can occur on a hot standby(replica) as described here.

Also from across the web.

A replication conflict occurs whenever the recovery process cannot apply WAL information from the primary server to the standby, because the change would disrupt query processing there. These conflicts cannot happen with queries on the primary server, but they happen on the streaming replication standby server because the primary server has limited knowledge about what is going on on the standby.

Which Conflict ?

Now in our case why is the conflict happening ? You check the stats by running the below query on your replica

select *from pg_stat_database_conflicts;

datid | datname | confl_tablespace | confl_lock | confl_snapshot | confl_bufferpin | confl_deadlock
— — — -+ — — — — — -+ — — — — — — — — — + — — — — — — + — — — — — — — — + — — — — — — — — -+ — — — — — — — —
16404 | myservice | 0 | 0 | 3 | 0 | 
016403 | rdsadmin | 0 | 0 | 0 | 0 | 
016401 | postgres | 0 | 0 | 0 | 0 | 
014372 | template0 | 0 | 0 | 0 | 0 | 
016402 | template1 | 0 | 0 | 0 | 0 | 0

So count 3 indicates the number of occurrences of conflicts of type confl_snapshot.
As per the docs, this means

Number of queries in this database that have been canceled due to old snapshots

Snapshot Conflicts

From docs, the above error occurs when

Application of a vacuum cleanup record from WAL conflicts with standby transactions whose snapshots can still “see” any of the rows to be removed.

If the replica is applying a wal received from the primary and that wal’s application on the replica includes removing dead rows, and if at the same time, a query(or queries) running on the replica has visibility of the same dead rows, then a snapshot conflict occurs.

Analogy

To over-simplify (yeah overly hence might not be accurate), in Postgres when a transaction starts it takes a snapshot (assume s1) and which is nothing but the state of the database when the transaction began. So assume at s1, the DB had 100 rows. This is on the replica.

So while t1 was still in progress, a transaction t2 on master deleted 5 rows and these changes are streamed in the wal file to the replica. So now as t1 is running the replica tries to apply the wal changes but as t1’s snapshot s1 was aware of those 5 rows which are being deleted now, a conflict arises as the replica assumes t1 might be needing those deleted rows.

Note, the transaction t1 in this case doesn’t even need those deleted rows, but that does not matter. Irrespective of the data being needed by the query, the query on the replica is cancelled due to the conflict. It's more about the snapshot time than the actual data being accessed by the query.

Continued in part 2 …

For a project whose scope is well defined and given the functional and non-functional requirements, we could first create the system design and then find many of the answers.

But still, sometimes there could be some aspects which still hamper our productivity.

• variable naming/database table naming

• ORM vs SQL

• SQL vs NoSQL

• java vs nodejs

• react vs angular

Below are some ways you could make progress and rapidly

• Comfortable

  Choose whatever you or your team is comfortable. For example, choose sql if your team is not comfortable with ORM’s and you are short on enough time.

• Previous Experience

  You could have had some experience with some aspect of a software component which could push you to use a different one.

• Less Number of Moving Parts

  When you are starting, the more components you introduce, the greater the complexity.
  Try to push your existing systems and reap the maximum benefit.
  Of course, postgres is not design to function as a queue, but that doesn’t mean it cannot be used. It could very well work for your scale and problem.

  You need to build a recommendation system for suggesting products to your customers. You could use sophisticated machine learning algorithms using a complex pipeline. But what if you could pull off an initial baseline using some SQL. This helps in having a baseline for evaluating an ML-based recommendation system should you choose to go down that line in the future.

• Conventions

  Choose a convention good or bad (choose the best practices) and stick to it. After some experience, if you feel a need for improvement, change the conventions. The less number of decision points, the faster you progress

  • folder structure

  • file naming (underscore/camelcase)

  • Generic Request validation, Services for Business logic etc.

• Experiment

  Sometimes you could have a decent time, try using something that you have been wanting to try. Like, use golang for building a cli tool used internally.
  This way, you will learn what its like to be on other side of the fence which you would have been wanting to.

Whatever you do, even if sometimes you move backwards, just remember that its improving your workflow and productivity.

Start building !

Premise: I wanted to launch multiple workers of the app as Python is single-threaded and also be able to have a common cache across.

We can use Uvicorn for launching multiple workers of FastAPI. But Uvicorn doesn’t support preload option that is we wanted to load the main app only once and still have multiple workers.
So I had to look at gunicorn and as gunicorn is a wsgi server, we had to use worker type as uvicorn and launch FastAPI.

We can use preload option of gunicorn so that the app loads only once with multiple workers for handling the load. Check this.

Okk ! kool.

But I wanted to have a common data structure (cache) across all the workers, so I instead went with the multiple threads option with just one worker.

Oops, but if we use any worker class other than gthread, gunicorn ignores it as in this case, I had to use uvicorn worker for asgi interface between guvicorn and fastapi.

From here

Threads is only meaningful with the threaded worker. Every other worker type ignores that setting and runs one thread per process.

So, I cannot use threads.

Also if you are using an async framework such as FastAPI, using threads is a bit orthogonal.

Ok, can I use multiple workers with the preload option and have a common data structure which is loaded in the app as a module-level variable.

Oops, but as per this a mutable cache is not possible between workers.

With or without the preload option you will end up with one background thread in each worker because when a process forks it forks all its threads. Whether the threads are created before the fork or after does not matter. In both cases the processes are independent once forked and do not share data structures. If you populate the data at module load time, that initial data will be visible to every worker. Future modifications will not be because they happen in separate processes. To share memory between processes (workers) you need to use a construct for explicitly sharing memory (/dev/shm, filesystem, network cache, db, etc).

You might be able to do the below, but you cannot change the data in itself as a common data structure.

So it's not possible to have a common mutable cache across workers at least in a straightforward way unless you employ other techniques.

Just to end this, let's see how the preload option itself works.I came across the below blog which explains it well.

https://www.joelsleppy.com/blog/gunicorn-application-preloading/

P.S: References

https://github.com/tiangolo/fastapi/issues/2425
https://levelup.gitconnected.com/supercharging-pythons-scalability-1eec2f501dd5
https://stackoverflow.com/questions/38425620/gunicorn-workers-and-threads