Header blob picture Blob picture Polygon blob picture 1 Polygon blob picture 2 Footer blob picture

Tech hints: MongoDB memory allocation and cache management

Title picture of Blog Post

Blog post content

This article describes some details of memory allocation by MongoDB.

MongoDB uses memory mapped files for file management, so in this component, it relies a lot on the OS side. But let’s look at it based on Linux OS. MongoDB maps files into memory using standard mmap call from glibc.

How it works

So in a sense, all data and indexes are in memory, split by small memory pages. MongoDB has pointers to each page. But they are not in RAM yet.

Now when an actual query is made, MongoDB finds the page that contains the required data and reads that data. It reads it from memory in general, and it doesn’t care if the data is in RAM or not. It can’t know that it just sends the request to the address in memory.

The rest is up to the OS. It initializes those pages in RAM on their first usage. Such initialized pages are known as cache pages, their total size can be seen with the command ‘free’ (column ‘cached’). The problem is that OS expires them after some time. This expiration is inevitable no matter how much RAM is available. It is an OS-wide process based on recency of usage. There is a kswapd process that runs periodically and checks whether the page has been accessed since the previous run. If it has, it does nothing. If it hasn’t, it divides the age by two. When the age reaches zero, this page becomes a candidate for eviction. (Well, it never becomes zero, probably just close to it enough, I don’t know exact math behind it).

This mechanism, file mmap’ing, is used for every file in the system. Log files are mmap’ed as well, so the more logs, the more RAM used by cache pages.

There is some dependency on RAM usage: the higher is the ‘active’ RAM usage (see vmstat -s), the more frequently kswapd runs and the faster is the aging process. MongoDB reports the actual size of data in RAM and the maximal age of that data. So we could see that in case of memory management mongodb are pretty much rely on the OS side and this could cause some downside effects like cache expiration and purging of data from RAM just because OS think that data should be expired already.

Cassandra uses the same mmap thing BTW.

You will achieve much better result if will concentrate more on your product and team, instead of solving technology and infrastructure problems. Centaurea’s team has an awesome experience with MongoDB and we are ready to offer you high professional consulting or development services. We are looking forward to start working with you!

P.S. We are glad to receive feedback from you. If you have noticed any mistakes in the article or have suggestions about it please let us know.


Another blog posts

Avatar of blog post author
Ivan Karpey
Long awaited, secret startup catalyst now revealed
Title picture of another blog post
With the rise of cloud technologies in the tech world, we really get suited to everything “as a service”. Infrastructure, Platforms, Software And this will be even more in the future even in our regular life: car as a service, complex home equipment, some rare services… Well, pretty much everything we do not need constantly. And the main reasons for that is simple: it useful, simple and provide significant economic benefits. But I want to talk with you about applying this principle to business and more precisely to starting a new business, startup. How does this process mostly looking right now: Leader or a group have an idea and probably some money. Then they need to clarify an idea and trying to adopt the raw business idea to some kind of business model. Let’s assume it’s verified and it seems like there is a market for it and startup demonstrate reasonable market fit. illustration of an MVP Then it’s time for building an MVP which already require some professional skills which arise question on a team and hiring the right people with the right mentality and right skills. This is exceptionally hard even for a settled and successful business and 10 times harder for a startup. Except for those early members in many cases expected to get a share from a company. But just bring people together isn’t enough — it’s important to set up the proper process to ensure they collaborate effectively. Let’s hope founder was able to assemble an initial team and after, let’s say, the 3–12 month the company initially have an MVP which already working and hopefully do what’s expected. half of the software project never launch But data say, that for every started software product — it’s only 50% actually see it’s users. I’m not saying about plenty of users — just any early user. But let’s look at the things positively and assume our hypothetical team did it. Now it’s time for launching the product and getting some traction from users/clients (depending is it B2B or B2C market). In the majority of times at this stage founders resources already close to being exhausted so another significant problem arises — getting some funding from venture capital companies as a part of early-stage investment. Believe me or not but this is a significant problem which requires a huge amount of founder attention and focus. So if the team operate perfectly, the idea is working and founder have enough faith, tenacity, patience than the company getting some space on market and start really working, getting later stages of funding which provide more resources to operate successfully and without constantly thinking that tomorrow you will be dead. As you could see — it’s not a simple path at all and it normally takes at least several years from the initial idea to later stage funding. Several years of constant commitment and hard work for founders and a team. And you know what? Sadly to say that but 9 of 10 startups failed during this path. Statistically, there is 9 startups from 10 just fail For a whole bunch of reasons, which is definitely a subject of another article. Surely, even when startup fails — the Founder gets exceptionally valuable experience, but the price of it — several years of his or she’s life and a significant amount of money. Quite sad. But even if the startup succeeds and gets significant funding which could be considered as a valuable mid-path success, so at this time the founder already no longer owns 20–35 or even 50% of a company. Surely that’s not always the case but this situation is pretty general and, I would say, most common. So it’s pretty obvious that there is a lot of traps and pitfalls. However, people already collect some experience of how to making things better for founders and entrepreneurs. To allow an entrepreneur to concentrate on what’s really matter for the startup instead of solving routine problems day by day. Could it be different? Yes, for sure. And the answer to it — Startup as a Service. This is a business model designed and built to reduce Founders overwhelm of everything in startup and allowing he or she to focus on 2 keys area of every startup — fine-tuning a business model and funding. The rest is done by a skilled, settled team of professionals with experience in building and launching startups. With all skills used on demand. You need a skilled software architect to create a backbone of the future system — you get it for the exact amount of time it’s required. Need DevOps to deal with system deployment and orchestration — it’s already there. Need a skilled marketer to create a marketing strategy of a startup launch — you just need to ask. The goal of Startup as a Service vendor is to reinforce Founder with all professional and technical skills during building and launching product from a very raw idea till the late stage funding moment and then help with either transition expertise for an onsite team or collaborating as before if founder and investors happy with it. You could ask — how is this different from outsourcing development and outsourcing marketing? To fully answer it I will create a separate article, but the main difference — outsourcing companies are good at fulfilling requirements. In doing what you ask. In the majority of cases they do not Create, they just implementing the specification. That’s their experience. There is a huge difference — to Create and to follow. The big difference between actually innovate and create new and just follow other’s ideas But how often the startup have super detailed specification where every detail written to pieces, every button described and then nothing really changes when development goes? Well… never. That’s why 95% of startup founders are not happy with outsourcing development. Regarding outsourcing marketing — it’s a little bit different. Outsourcing marketing company just do not have a proper connection with a product. There are middlemen translating them ideas they need to promote, they don’t have raw data, they can’t influence the product, put their feedback. This all really decreasing their productivity and make them slowly working in the past, because of lags in communication are everywhere. In Startup-as-a-Service company people, in the opposite, trained to create new things in a very fast paced, ever-changing environment. In my experience, we react to client request in minutes with a new feature. Really, the very important client just submit the request which was easy to build and we write the code and deploy it to production in nearly 12 minutes. That’s what is customer centricity are =0) And there is a single product owner for design, development and marketing team. So all the pieces in one place to create, build and promote as much effective and efficient as possible. Zero communication lags, zero overheads on management and cross-department competitions. Just one goal — to build and launch a successful product. And Founder job is only to translate his vision to a team. The rest, details, exact features, implementation — it’s all will be completed by the team. proper Startup-as-a-Service vendor provide significant economic benefits in comparison to other variants And you know, I can’t say for every vendor (however I believe that should be the case for the majority), but as a Founder of Centaurea I could ensure that develop startup with us is just much more financially effective, majorly because of: on-demand nature of used resources; zero management overhead; skilled, trained and startup experienced professional team; already created and verified development process and delivery infrastructure; lack of a need to provide a company share for every early team member. So yes, I honestly believe that: Software as a Service business model is a long need catalyst for startup development. It could significantly simplify and speed up startup development, reduce costs and eventually grow the level of innovations. Because it’s Startups — those who are really innovating, not the corporations. If you have some questions you what to discuss — feel free to put them in comments or do not hesitate to contact me directly. I will honestly try to respond to every question, I promise. That’s just a very first piece of content regarding Startup-as-a-Service business model, startup development, entrepreneurship, and technology leadership. If you like it and want to know more subscribe to us in social networks or via our newsletter. Will be much appreciated for liking and sharing this content. Visit our website — www.centaurea.io to find out more about Startup-as-a-Service business model and how we build and launch startups. Thanks for reading. Stay tuned.