vCore-based Azure Cosmos DB for MongoDB vs MongoDB Atlas

Summary

This benchmark study provides a comprehensive comparison of Azure Cosmos DB for MongoDB (vCore) and MongoDB Atlas, with a focus on their performance in vector queries and how they can be leveraged for GenAI applications.

Our findings reveal that vCore-based Azure Cosmos DB for MongoDB significantly outperforms MongoDB Atlas in vector query performance. Specifically, Azure Cosmos DB achieves search requests per second that are 236%-1000% higher and lower latency (77%-93% lower) across different dimensional datasets. The benchmarking process compared vector queries under various operational conditions, using datasets of different dimensions (1536, 784, and 100) to evaluate the performance of both database solutions across multiple SKUs.

These findings suggest that Azure Cosmos DB for MongoDB (vCore) offers superior performance, flexibility, and scalability for enterprises looking to deploy AI-driven applications at scale. It is particularly well-suited for GenAI use cases that require high query throughput and low latency.

Generative AI and Vector Databases in the Enterprise

Generative AI (GenAI) represents a significant leap forward in artificial intelligence, allowing machines to create new content - such as text, images, music and videos - based on patterns learned from existing data. This capability is transforming numerous business domains by enabling innovative applications and improving operational efficiencies. For instance, in marketing, GenAI can generate personalized content for customers at scale, while in finance, it can predict market trends and optimize investment strategies. In healthcare, GenAI aids in drug discovery and personalized medicine by analyzing complex datasets to identify potential treatments and patient-specific solutions.

Central to the efficacy of GenAI applications is the use of vector databases, which store and manage high-dimensional vectors generated by machine learning models. These vectors encapsulate intricate patterns and relationships within the data, enabling sophisticated similarity searches and real-time data retrieval. In enterprises, vector databases are critical for applications such as recommendation systems, fraud detection, and semantic search. They enhance the ability to process and analyze unstructured data, such as text and images, thereby driving more accurate insights and decision-making across various business domains.

vCore-based Azure Cosmos DB for MongoDB vs. MongoDB Atlas

Azure Cosmos DB for MongoDB (vCore) and MongoDB Atlas are two prominent database solutions that support MongoDB workloads, yet they cater to different needs and preferences in the enterprise environment. Both are designed to provide scalable, high-performance database services, but they differ in their underlying infrastructure, feature sets, and integration capabilities.

Azure Cosmos DB for MongoDB (vCore) is a fully managed MongoDB-compatible, globally distributed database service provided by Microsoft Azure. It has open-sourced its engine, DocumentDB, under the fully permissible MIT license. Additionally, it offers native support for MongoDB workloads using the vCore-based purchasing model, which allows users to provision and scale resources independently for more predictable performance and cost management. Azure Cosmos DB ensures high availability and low-latency access through its global distribution, making it ideal for applications requiring robust scalability and geo-redundancy.

MongoDB Atlas, on the other hand, is a fully managed cloud database service offered by MongoDB Inc. It provides a rich set of features tailored specifically for MongoDB, including automated backups, real-time performance monitoring, and advanced security configurations. MongoDB Atlas supports deployment across multiple cloud providers, including AWS (Amazon Web Services), Google Cloud Platform, and Azure, offering flexibility in cloud strategy and reducing vendor lock-in. It also integrates seamlessly with MongoDB's ecosystem, providing tools for application development, data visualization, and machine learning.

While both Azure Cosmos DB for MongoDB and MongoDB Atlas support MongoDB workloads and aim to simplify database management, they differ in their approach to scalability, cloud integration, and feature offerings. Azure Cosmos DB is optimized for global distribution within the Azure ecosystem, whereas MongoDB Atlas focuses on providing a comprehensive, MongoDB-centric experience with multi-cloud flexibility and extensive toolsets for developers.

Benchmarking Methodology

To provide a comprehensive comparison between Azure Cosmos DB for MongoDB (vCore) and MongoDB Atlas, we have designed a methodology that evaluates their performance across different scenarios and benchmarks. Our approach includes two primary benchmarks: vector queries. Vector queries are essential for evaluating the performance in GenAI and similar applications where high-dimensional data is processed.

This methodology focuses on three distinct comparisons: comparison of M40, M50, M80 with Azure OpenAI (AOAI) embedding in 1536, 784 and 100 dimensional datasets and a comparison of the free-tier offerings. Each of these comparisons will assess the databases under various operational conditions to provide a clear understanding of their capabilities and performance characteristics. For the vector search benchmarking, the datasets used include 'arxiv-aoai-1536-euclidean-500k' with 500K documents and 1536 dimensions, 'fashion-mnist-784-euclidean' with 60K documents and 784 dimensions, and 'glove-100-angular' with 1.2M documents and 100 dimensions. The average document size and the distance types (l2 and cosine) are critical factors influencing performance, with KNN samples ranging from 5,000 to 10,000 and KNN neighbor numbers fixed at 100.

Comparison of M40, M50, and M80 in a 1536-Dimensional Dataset: The Most Likely Scenario evaluates the databases using high SKU (Stock Keeping Unit) configurations and data embeddings from Azure OpenAI. This scenario tests the performance of the databases when handling complex, high-dimensional data typical in advanced AI applications. The goal is to measure how well each database can manage and query embedded vectors, which are important for tasks such as similarity searches in AI-driven applications.

Comparison of M40, M50, and M80 in a 784-Dimensional Dataset: The Common Scenario is like the above but focuses on 784 instead of 1536-dimensional dataset.

Comparison of M40, M50, and M80 in a 100-Dimensional Dataset: The Least Likely Scenario focuses on high SKU configurations but with a simpler, low-dimensional dataset of just 100 dimensions typically used for a very specific use case. This scenario helps determine the efficiency and scalability of the databases when dealing with less complex data structures, providing insights into their versatility and adaptability to different data environments.

Free-Tier Comparison examines the databases under lower SKU configurations, which represent more cost-effective, resource-constrained scenarios. This comparison is essential for understanding how each database performs when resources are limited, making it relevant for smaller-scale applications or cost-sensitive operations.

For each of these scenarios, we benchmark the databases across three key metrics:

• Search RPS (Requests Per Second) vs. Recall: This metric evaluates the efficiency of the databases in handling search queries, measuring how many requests per second they can process while maintaining different levels of recall accuracy.
• P95 Latency vs. Recall: This benchmark assesses the 95th percentile latency, indicating the time taken for the database to process most search queries. This helps understand the responsiveness of the databases under various recall thresholds.
• Upload Time vs. Recall: This benchmark evaluates the time taken to upload data into the databases, which is critical for understanding the data ingestion performance.

Note for developers: We noticed that Azure Cosmos DB for MongoDB (vCore) allows more flexible configuration in HNSW. HNSW (Hierarchical Navigable Small World) is a data structure and algorithm used for approximate nearest neighbor search in high-dimensional spaces. It has gained popularity due to its efficiency in handling large-scale nearest neighbor search tasks, particularly in machine learning and information retrieval applications.

Above image shows the parameters passed to the "vCore" engine (Azure Cosmos DB for MongoDB (vCore)) and MongoDB Atlas separately. It seems like vCore allows more flexibility and MongoDB Atlas doesn't.

• efConstruction: Influences the structure of the index (the graph) built during the index construction phase. It determines how densely connected the nodes in the graph are.
• efSearch: Influences the search behavior during the query phase. It determines how many nodes are considered during the nearest neighbor search.
• M: Influences the structure of the graph during index construction by controlling the degree of connectivity of each node.
• numCandidates: Influences the search behavior during the query phase by determining how many nodes are considered as potential nearest neighbors.

By analyzing these metrics, we aim to provide a detailed comparison of Azure Cosmos DB for MongoDB (vCore) and MongoDB Atlas, highlighting their strengths and weaknesses in different operational contexts. This methodology will offer valuable insights for enterprises to make informed decisions based on their specific requirements and constraints.

Comparison on Vector Queries

Comparison of M40, M50, M80 with AOAI Embedding using 1536 Dimensions (Most likely scenario)

Comparison of M40, M50, M80 in a 1536-dimensional dataset for both Azure Cosmos DB for MongoDB (vCore) vs MongoDB Atlas is shown below. Every image below has two graphs comparing different metrics. The different types of machines are:

Search RPS vs Recall

The left side of image 1 shows the changes in recall along with the request per second for the searches done. Request Per Second (RPS) or Queries Per Second (QPS) is a measure of the number of queries that a system can handle in a second. It's a common metric used to evaluate the performance of databases, including vector databases.

On the other hand, Recall is a metric used to evaluate the quality of search results in vector databases. It measures what percentage of the returned results are objectively the closest to the query vectors. The set of vectors closest to the query vector are called the “ground truth” and are used as a baseline when measuring recall.

In the context of vector databases, these two metrics are often considered together because they reflect different aspects of the system's performance. QPS measures the system's speed, while Recall measures the system's accuracy. Balancing these two aspects is a key challenge in designing and optimizing vector databases.

For Queries Per Second (QPS), a higher value is better. This is because a higher QPS means the system can handle more queries in a second, indicating better performance and efficiency.

For Recall, a higher value is also better. A higher recall means that a larger percentage of the relevant results (the “ground truth”) are being returned by the system. This indicates that the system is more accurate in retrieving the vectors that are closest to the query vector.

So, in both cases, a higher value indicates better performance. However, it's important to note that there is often a trade-off between these two metrics in real-world systems. Optimizing for a higher QPS might result in a lower Recall, and vice versa. Therefore, the optimal balance between these two metrics depends on the specific requirements of your application.

Image1: Search RPS vs Recall and P95 Latency vs Recall

Conclusion: The result shows that Azure Cosmos DB for MongoDB (vCore) performs way better than MongoDB Atlas as it achieves higher recall for similar Search RPS but also has a much higher Search RPS than Mongo DB Atlas.

Note for developers: In image 1, we have only highlighted the results with recall between 0.8 to 1. That's because we are comparing apples to apples. We compare the Search RPS for similar recalls on both the databases. Why does Azure Cosmos DB for MongoDB (vCore) seem to have lower recalls in the shaded area? That's because it allows us to configure in a more flexible way and that leads to increased RPS for a lower recall. More flexibility is surely preferred by developers who want more control on their application and would like to make a more intelligent choice between recall and other important metrics. Currently, we do not see a way to get the same amount of flexibility with MongoDB Atlas. It occurs to us that some trade-offs have been made to achieve higher recall. As developers we find that quite confining.

P95 Latency vs Recall

P95 Latency is a measure of the time it takes for a system to respond to a query. Specifically, P95 (or the 95th percentile) indicates that 95% of the API requests are faster than this value, and only 5% of the requests are slower. It's a common metric used to evaluate the performance of databases, including vector databases. P95 Latency measures the system's speed, while Recall measures the system's accuracy.

For P95 Latency, a lower value is better. This is because a lower P95 Latency means the system can respond to more queries faster, indicating better performance and efficiency. For Recall, a higher value is better.

So, in this case, a lower P95 Latency and a higher Recall indicate better performance. Optimizing for a lower P95 Latency might result in a lower Recall, and vice versa. Therefore, the optimal balance between these two metrics depends on the specific requirements of your application.

The right-hand side graph of Image 1 shows P95 Latency vs Recall comparisons. In this case, Azure Cosmos DB for MongoDB (vCore) has lower latency than MongoDB Atlas for similar recall values and hence it outperforms.

Upload Time vs Recall

Upload Time refers to the amount of time it takes to upload the vectors to the vector database. This metric is important because it directly impacts the speed and efficiency of the data ingestion process. However, this is a one-time process and that's why it's not a very important metric, in general.

Image 2: Upload Time vs Recall

The Upload Time measures the system's data ingestion speed, while Recall measures the system's accuracy. For Upload Time, a lower value is better. This is because a lower Upload Time means the system can ingest data faster, indicating better performance and efficiency.

Conclusion: M40, M50, M80 with Embedding using 1536 Dimensions

Image 3: Search RPS comparison between Atlas and vCore for similar recalls on 1536 embeddings.

Mostly, Azure Cosmos DB for MongoDB (vCore) demonstrates much higher Search RPS for the same exact recalls or even higher recalls than MongoDB Atlas. A similar pattern is noticed in the case of P-95 latencies, too.

Search RPS:

• M-40 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 841%-1000% higher.
• M-50 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 236%-380% higher.
• M-80 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 787%-801% higher.

Image 4: P-95 latency comparison between Atlas and vCore for similar recalls on 1536 dimensions.

P-95 Latency:

• M-40 SKUs: Azure Cosmos DB for MongoDB (vCore) delivers lower latency (85%-93% lower) compared to MongoDB Atlas.
• M-50 SKUs: Azure Cosmos DB for MongoDB (vCore) delivers lower latency (77%-78% lower) compared to MongoDB Atlas.
• M-80 SKUs: Azure Cosmos DB for MongoDB (vCore) delivers lower latency (88%-92% lower) compared to MongoDB Atlas.

Despite MongoDB Atlas winning in upload time, which is a one-time effort, Azure Cosmos DB for MongoDB (vCore) has a clear overall victory.

Comparison of M40, M50, M80 with 784 Dimension (Common Scenario)

For 784 dimensions, Azure Cosmos DB for MongoDB (vCore) continues to outperform MongoDB Atlas for Search RPS and P95 latency for all the SKUs.

Image 5: Search RPS vs Recall and P95 Latency vs Recall

In terms of upload, MongoDB Atlas continues to outperform Azure Cosmos DB for MongoDB (vCore).

Image 6: Upload Time vs Recall

Conclusion: Comparison of M40, M50, M80 with 784 Dimension (Common Scenario)

Image 7: Search RPS comparison between Atlas and vCore for similar recalls on 784 dimensions.

Mostly, Azure Cosmos DB for MongoDB (vCore) demonstrates much higher Search RPS for the same exact recalls or even higher recalls than MongoDB Atlas. A similar pattern is noticed in the case of P-95 latencies, too.

Search RPS:

• M-40 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 46%-66% higher.
• M-50 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 36%-201% higher.
• M-80 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 47%-127% higher.

Image 8: P-95 latency comparison between Atlas and vCore for similar recalls on 784 dimensions.

P-95 Latency:

• M-40 SKUs: Azure Cosmos DB for MongoDB (vCore) delivers lower latency (39%-44%) compared to MongoDB Atlas.
• M-50 SKUs: Azure Cosmos DB for MongoDB (vCore) delivers lower latency (38%-41%) compared to MongoDB Atlas.
• M-80 SKUs: Azure Cosmos DB for MongoDB (vCore) delivers lower latency (18% to 65%) compared to MongoDB Atlas.

Comparison of M40, M50, M80 on Embeddings in a 100-dimensional dataset (Least Likely Scenario)

For 100 dimensions, Azure Cosmos DB for MongoDB (vCore) continues to outperform MongoDB Atlas for Search RPS for all the SKUs: M40, M50 and M80. In terms of P95 latency, Azure Cosmos DB for MongoDB (vCore) continues to outperform M40 and M50 and slightly underperforms M80.

Image 9: Search RPS vs Recall and P95 Latency vs Recall

In terms of upload, MongoDB Atlas continues to outperform Azure Cosmos DB for MongoDB (vCore).

Image 10: Upload Time vs Recall

Image 11: Search RPS comparison between Atlas and vCore for similar recalls on 100 dimensions.

Mostly, Azure Cosmos DB for MongoDB (vCore) demonstrates much higher Search RPS for the same exact recalls or even higher recalls than MongoDB (vCore). A similar pattern is noticed in the case of P-95 latencies, too.

Search RPS:

• M-40 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 160%-214% higher.
• M-50 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 435%-790% higher.
• M-80 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with search requests per second that are 86%-445% higher.

Image 12: P-95 latency comparison between Atlas and vCore for similar recalls on 784 dimensions.

P-95 Latency:

• M-40 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with lower latency (41%-57%) compared to MongoDB Atlas.
• M-50 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with lower latency (77%-89%) compared to MongoDB Atlas.
• M-80 SKUs: Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas with lower latency (42%-86%) compared to MongoDB Atlas.

Free-Tier Comparison:

For free-tier, Azure Cosmos DB for MongoDB (vCore) continues to outperform MongoDB Atlas for Search RPS as well as P95 Latency.

Image 13: Search RPS vs Recall and P95 Latency vs Recall

For upload time, Azure Cosmos DB for MongoDB (vCore) tends to outperform MongoDB Atlas significantly.

Image 14: Upload Time vs Recall

vCore-based Azure Cosmos DB for MongoDB Free Tier:

• The free tier lasts for the lifetime of your account.
• It boasts command and feature parity with a regular Azure Cosmos DB for MongoDB (vCore) account.
• It makes it easy for you to get started, develop, test your applications, or even run small production workloads for free.
• With Free Tier, you get a dedicated MongoDB cluster with 32-GB storage.
• Users can provision a single free DB server per supported Azure region for a given subscription.
• High availability, Azure Active Directory (Azure AD) and Diagnostic Logging are not supported.
• When you're ready to scale, you can seamlessly transition to any of the available paid tiers without any hassle. Your data, connection string, and firewall rules will stay unchanged throughout the upgrade process.

MongoDB Atlas Free Tier:

• Provides a small-scale development environment to host your data.
• Free clusters never expire and provide access to only a subset of Atlas features and functionality.
• You can deploy only one free cluster per Atlas project.
• The Free Tier runs the latest version of MongoDB.
• It provides support for datasets larger than 512 MB.
• All App Services in an Atlas project share a monthly free tier: 1 million requests or 500 hours of compute, or 10,000 hours of sync runtime (whichever occurs first), plus 10 GB of data transfer.

Free-Tier: Azure Cosmos DB for MongoDB (vCore) free-tier simply is an overall winner. It exceeds performance on all major metrics and can be used for small production workflows as opposed to MongoDB Atlas, which is more suitable for mostly development and testing workloads.

Note for developers: The vertical line representing perfect 100% recall pertains to the dataset, which is notably smaller compared to others, containing only 10,000 records due to the storage limitations of the 512MB MongoDB Atlas M0 cluster. Because of its smaller size, the task becomes relatively easier for both vCore and Atlas, as they can find all nearest neighbors with straightforward parameters. HNSW quickly achieves 100% recall due to the simplicity of the graph, and a similar effect is observed with IVF. Benchmark results also include datasets with 1.18 million records, presenting a more challenging task for evaluating the vCore free tier performance.

Conclusion

In this benchmark, Azure Cosmos DB for MongoDB (vCore) consistently outperforms MongoDB Atlas in vector query performance across various scenarios and dimensions.

For 1536 dimensions (most likely scenario), Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas in QPS vs Recall and P95 Latency vs Recall. These are the two most important factors and Microsoft has a clear lead. MongoDB Atlas leads in upload time. However, that's usually a one-time effort and that's why Azure Cosmos DB for MongoDB (vCore) is the overall winner.

For 784 dimensions (common scenario), Azure Cosmos DB for MongoDB (vCore) outperforms MongoDB Atlas in Search RPS vs Recall and P95 Latency vs Recall. These are the two most important factors and Microsoft has a clear lead. MongoDB Atlas leads in upload time. However, that's usually a one-time effort and that's why Azure Cosmos DB for MongoDB (vCore) is the overall winner.

For 100 dimensions (least likely scenario), Azure Cosmos DB for MongoDB (vCore) continues to outperform MongoDB Atlas for Search RPS for all the SKUs: M40, M50 and M80. In terms of P95 latency, Azure Cosmos DB for MongoDB (vCore) continues to outperform M40 and M50 and slightly underperforms M80. MongoDB leads in upload time. However, that's usually a one-time effort and that's why Azure Cosmos DB for MongoDB (vCore) is the overall winner.

In all three scenarios, Azure Cosmos DB for MongoDB (vCore) shows clear leadership in Search RPS and P95 Latency, with MongoDB Atlas only achieving better performance in upload time, which is a one-time metric. Therefore, Azure Cosmos DB for MongoDB (vCore) is the overall winner across all dimensions and scenarios.

Disclaimer

All benchmarking and testing reported in this document were conducted using specific hardware configurations and on a designated cloud environment. The results presented herein reflect conditions as of June 20, 2024. It is important to note that performance outcomes may vary if attempted on different hardware configurations, alternative cloud platforms, or at later dates due to factors such as updates in software versions, changes in hardware availability, or optimizations made by cloud service providers.

Furthermore, this report is provided for educational purposes only. Readers are encouraged to conduct their own due diligence and consider their specific requirements and circumstances when making decisions based on the information presented here.