A few years ago I worked for this startup as a principal engineer. The engineering manager kept touting how he was from XYZ and that he ran Pythonista meetups and how vast his knowledge of Python was. We were building a security product and needed to scan hundreds of thousands of documents quickly so we built a fan out with coroutines. I came on board well into this effort to assist in adding another adapter to this other platform that worked similarly. After seeing all the coroutines, being pickled and stored in S3, so that nodes could “resume” if they crashed yet - not a single create_task was present. All of this awaiting, pickling, attempting to resume, check, stuff, report, pickle, happened synchronously.
When trying to point out the issue with the architecture and getting into a shouting match with Mr. Ego, I was let go.
Maybe I'm nitpicking a bit but the second concrete example shows different flow in the parent method compared to the first one. If the second example would be a modification of the first one like shown below the output would be the same in both cases.
Sorry if I've got this wrong, but wouldn't the first example behave the same way in Javascript as well? The function parent() "awaits" the completion of child(), so it wouldn't be possible to interleave the print statements.
Personally, I've never been able to make async work properly with Python. In Node.js I can schedule enough S3 ListBucket network requests in parallel to use 100% of my CPU core, by just mapping an array of prefixes into an array of ListBucket Promises. I can then do a Promise.all() and let them happen.
In Python there's asyncio vs threading, and I feel there's just too much to navigate to quickly get up and running. Do people just somehow learn this just when they need it? Is anyone having fun here?
I may be misunderstanding it, but the examples shown don't seem to be illustrative? It seems reasonably obvious that the prints will happen in this order in any language, because in example 1 we are explicitly (a)waiting for the child to finish, and in example 2 both of the parent prints are above the await. So I don't feel like either of these makes the point the author is trying to get across?
The article’s understanding of Java’s Virtual Threads is incorrect. Virtual threads have a preemptive concurrency model, not a co-operative one, so suspension points are not only at calls to things like Future.get()
This is gold.
Here’s a horror story for you.
A few years ago I worked for this startup as a principal engineer. The engineering manager kept touting how he was from XYZ and that he ran Pythonista meetups and how vast his knowledge of Python was. We were building a security product and needed to scan hundreds of thousands of documents quickly so we built a fan out with coroutines. I came on board well into this effort to assist in adding another adapter to this other platform that worked similarly. After seeing all the coroutines, being pickled and stored in S3, so that nodes could “resume” if they crashed yet - not a single create_task was present. All of this awaiting, pickling, attempting to resume, check, stuff, report, pickle, happened synchronously.
When trying to point out the issue with the architecture and getting into a shouting match with Mr. Ego, I was let go.
Maybe I'm nitpicking a bit but the second concrete example shows different flow in the parent method compared to the first one. If the second example would be a modification of the first one like shown below the output would be the same in both cases.
Sorry if I've got this wrong, but wouldn't the first example behave the same way in Javascript as well? The function parent() "awaits" the completion of child(), so it wouldn't be possible to interleave the print statements.
The example from this StackOverflow question might be a better demonstration: https://stackoverflow.com/q/69958618
Personally, I've never been able to make async work properly with Python. In Node.js I can schedule enough S3 ListBucket network requests in parallel to use 100% of my CPU core, by just mapping an array of prefixes into an array of ListBucket Promises. I can then do a Promise.all() and let them happen.
In Python there's asyncio vs threading, and I feel there's just too much to navigate to quickly get up and running. Do people just somehow learn this just when they need it? Is anyone having fun here?
I may be misunderstanding it, but the examples shown don't seem to be illustrative? It seems reasonably obvious that the prints will happen in this order in any language, because in example 1 we are explicitly (a)waiting for the child to finish, and in example 2 both of the parent prints are above the await. So I don't feel like either of these makes the point the author is trying to get across?
For the JS developers: similar useful behavior (and more!) can be implemented in JS using the wonderful Effection library.
https://effection-www.deno.dev/
but how would you systematically audit a large async codebase to find all the hidden interleave points, unnecessary locks?
The article’s understanding of Java’s Virtual Threads is incorrect. Virtual threads have a preemptive concurrency model, not a co-operative one, so suspension points are not only at calls to things like Future.get()
Just to note the it's the same in Rust with tokio::spawn (and alternatives).