Tomcat, Why just 200 default threads?
Springboot is a kind of defacto go-to framework when it comes to building applications with Java. It's in demand with almost all the good product-based company and demand continue to grow so, internally it uses tomcat as a server which defaults it to 200 threads. But why?
With great demand there comes great scalability issues.
Then why 200?
Why we defaulted ourselves to only 200 threads why not 1000, why not 10000. What's the cap over here, or is even 200 is kind of overkill. Let's see How it works under the hood.
Let's first get the basics right, what happens when an API call happens to Server.
It spawns a new thread, does some processing like validation and all, then in most cases, it will do a DB call, where it waits. ( Thread goes in waiting state)
Waits, because it an expensive operation, the server needs to call the DB API, have to run some query and return the result, it might be in millisecond but for server POV, it's huge. Now when the results came back from DB, the same thread returns the response and the process goes on.
Now, what happens when a new request came when the thread is in waiting state. Well, we will spawn a new thread. Simple…!!! And it goes on till the 200 limit is reached.
Now, if the 201st request came through, it will simply wait, till some thread is free to process the request. Here we hit a ceiling.
One could say, ok we will add 1000 threads whats the harm, it could accept 1000 connections and work fast?
The answer is, Each thread has a stack, which consumes RAM. Moreover, each Thread will actually allocate stuff on the heap to do its work, consuming again RAM, and the act of switching between threads (context switching) is quite heavy for the JVM/OS kernel.
This makes it hard to run a server with thousands of threads “smoothly”.
Ok, so we could increase the Ram and memory, it could work right?
Yes, it would, but what if we have a better solution to the problem, and there is something more that also limits the processing of number the threads.
That is the number of Cores. :-D
Like, even if you are having 10000 threads, if you are having 8 cores, it can only process 8 threads at a time max. Period (Note:- Waiting thread is not a processing thread)
What's the better solution here then, whats the whole point?
What if, we tweak the process a little. Like in spite of waiting for DB to return the response on the same waiting thread we say that, ok, DB whenever you are ready with the result tells us, and in the meanwhile, the thread which was supposed to wait could process the other incoming request.
That would be wonderful right, less number of threads, more throughput, no idle sitting thread.
And that solution exists, and that is basically reactive programming and SpringBoot does that by SpringWebFlux, which would dramatically increase the performance and reduce the server load. Everyone is happy… :-D
Conclusion:
Case 1:
Any Process intensive task with little IO operation (No waiting thread), like resizing image on the fly. Max number of thread = Max number of core+10(Just for safety).
Note:- This service should be an async service, not a customer-facing service, otherwise, it will get “Too many connections error”.
Case 2:
Having IO operations(i.e DB calls), more number of waiting threads = Depends upon the machine power you are having but better yet stick to 200 default, it will be handling 200 concurrent requests.
Case 3:
Reactive spring server thread count = Number of cores+ 10 extra(Just for safety).
As there will never be a waiting thread.
Claps please. :-), will love the feedback.
One can connect me on alpitanand20@gmail.com.
Or just google Alpit anand you can find me 😾
