Quicks:
- Learn mongo CLI: Click here
- LinkedIn Quiz Solutions: Click here
flash
is my own testing library, the code is @flash
.- From
jest
official docs for testing mongodb database (using officialmongodb
library): https://jestjs.io/docs/mongodb - Seems like a good read on efficient mongodb queries with mongoosejs: Click here
// schema:
password: {
type: String,
required: [false, 'Please Enter Your Password'],
minLength: [8, 'Password should be greater than 8 characters'],
select: false,
},
// password field is only retrieved when explicitly asked for like below
const user = await User.findOne({ email }).select('+password')
const mongoose = require('mongoose');
const userSchema = new mongoose.Schema({
username: {
type: String,
required: true
},
email: {
type: String,
required: true,
unique: true
}
});
const User = mongoose.model('User', userSchema);
// This is useful for simple validation on APIs
async function validateUser() {
const newUser = new User({
username: 'john_doe',
email: 'john.doe@email.com'
});
try {
await newUser.validate();
console.log('Data is valid!');
} catch (error) {
console.error(error.message);
}
}
validateUser();
Query: For e.g., we have a 10 classes in a school, then each class has students with defined roll numbers to them in ascending order. So, we need to fetch list of students in a particular order only, in other words we need to maintain their order/sequence.
Solution:
- Use a Sequence Field: Add a
sequence
field to your documents that represents the order or sequence of the documents. When you retrieve documents, you can use this field to sort them in the desired sequence. When retrieving documents, you can use .sort({ "sequence": 1 }) to get them in ascending order.{ "_id": ObjectId("documentId"), "name": "Document Name", "sequence": 1 // Other document fields }
- Maintain an External List in your class document for e.g.,
rollNumbers: [studentId1, studentId2, and so on...]
that represents the desired sequence. When you need to retrieve documents in a specific order, you can fetch the list of IDs and then retrieve the documents based on this list. This approach allows you to change the order without modifying the documents themselves. However, it requires additional management of the order list.
const doc of moviesModel
.find({ deleted: MovieDeletionStatus.NotDeleted })
.cursor({ maxTimeMS: 5000 })
) {
// doc
}
Note: Model.exists(filter)
returns { _id: id_of_first_matched_document } | null
.
async podcastUrlIsAvailable(podcastUrl: string): Promise<boolean> {
const podcastExists = await this.podcastsModel.exists({ rssUrl: podcastUrl });
return !podcastExists;
}
Source: Click here
Problem: When we run a below script with a simple console.log
, the app.close works pretty good but when we have some bulky db operations with db the app.close
seems to hang out for some time like 1-2 minutes thus we can fix that issue by actively closing the db connection as shown in below screenshot.
Short notes:
- Default is ascending order ie.,
sort: 1
- Providing
sort: -1
to get in descending order.
Link to saved playground by Author: Click here
Stackoverfow Answer: Click here
Source: Click here
Also, $and
is redundant and thus we can remove $and
and it would work same again:
Source: Stack Overflow Question: Click here
The latitude must be a number between -90 and 90 and the longitude between -180 and 180.
Specifies a rectangle for a geospatial $geoWithin query to return documents that are within the bounds of the rectangle, according to their point-based location data.
$box: Click here
{/* using mongocompass's query, worked awesome! */}
db.events.find(
{
location: {
$geoWithin: { $box: [ [topRightCoordinateLat, topRightCoordinateLng], [bottomLeftLat, bottomLeftLng]] }
}
}
)
const results = await this.eventModel.aggregate([
{
$geoNear: {
near: {
type: 'Point',
coordinates: [lattitude, longitude],
},
maxDistance: maxDistanceMetres,
distanceField: 'distance',
// get distances of each event in miles
distanceMultiplier: METRES_TO_MILES_MULTIPLIER,
},
},
// any other optional query (NOTE: `$geoNear` must be the first query in the pipeline array of queries
{ $match: query },
]);
Docs:
- GeoJSON Objects: Click here
- Geospatial Queries: Click here
MongoDB geospatial queries on GeoJSON objects calculate on a sphere; MongoDB uses the WGS84 reference system for geospatial queries on GeoJSON objects.
Syntax:
<field>: { type: <GeoJSON type> , coordinates: <coordinates> }
// Point
{ type: "Point", coordinates: [ 40, 5 ] }
// LineString
{ type: "LineString", coordinates: [ [ 40, 5 ], [ 41, 6 ] ] }
// Polygon
{
type: "Polygon",
coordinates: [ [ [ 0 , 0 ] , [ 3 , 6 ] , [ 6 , 1 ] , [ 0 , 0 ] ] ]
}
// Polygons with Multiple Rings
{
type : "Polygon",
coordinates : [
[ [ 0 , 0 ] , [ 3 , 6 ] , [ 6 , 1 ] , [ 0 , 0 ] ],
[ [ 2 , 2 ] , [ 3 , 3 ] , [ 4 , 2 ] , [ 2 , 2 ] ]
]
}
// MultiPoint
{
type: "MultiPoint",
coordinates: [
[ -73.9580, 40.8003 ],
[ -73.9498, 40.7968 ],
[ -73.9737, 40.7648 ],
[ -73.9814, 40.7681 ]
]
}
// MultiLineString
{
type: "MultiLineString",
coordinates: [
[ [ -73.96943, 40.78519 ], [ -73.96082, 40.78095 ] ],
[ [ -73.96415, 40.79229 ], [ -73.95544, 40.78854 ] ],
[ [ -73.97162, 40.78205 ], [ -73.96374, 40.77715 ] ],
[ [ -73.97880, 40.77247 ], [ -73.97036, 40.76811 ] ]
]
}
// MultiPolygon
{
type: "MultiPolygon",
coordinates: [
[ [ [ -73.958, 40.8003 ], [ -73.9498, 40.7968 ], [ -73.9737, 40.7648 ], [ -73.9814, 40.7681 ], [ -73.958, 40.8003 ] ] ],
[ [ [ -73.958, 40.8003 ], [ -73.9498, 40.7968 ], [ -73.9737, 40.7648 ], [ -73.958, 40.8003 ] ] ]
]
}
// GeometryCollection
{
type: "GeometryCollection",
geometries: [
{
type: "MultiPoint",
coordinates: [
[ -73.9580, 40.8003 ],
[ -73.9498, 40.7968 ],
[ -73.9737, 40.7648 ],
[ -73.9814, 40.7681 ]
]
},
{
type: "MultiLineString",
coordinates: [
[ [ -73.96943, 40.78519 ], [ -73.96082, 40.78095 ] ],
[ [ -73.96415, 40.79229 ], [ -73.95544, 40.78854 ] ],
[ [ -73.97162, 40.78205 ], [ -73.96374, 40.77715 ] ],
[ [ -73.97880, 40.77247 ], [ -73.97036, 40.76811 ] ]
]
}
]
}
Source (official mongoose docs):
-
- Schemas in TypeScript: Click here
-
- Statics and Methods in TypeScript: Click here (below screenshot)
// getting names of all models
const mongooseModelsNames = connection.modelNames();
// getting all models as an array
const mongooseModels = connection.modelNames().map((modelName) => connection.model(modelName));
(await connection.db.collections()).find((collection) => collection.collectionName === 'events').createIndex({ location: '2dsphere' });
- Source: Mongoose: Using GeoJSON: Click here
- Yet another amazing article: Geospatial Queries: Click here
- GeoJson: Click here
const citySchema = new mongoose.Schema({
name: String,
location: {
type: {
type: String,
enum: ['Point'],
required: true
},
coordinates: {
type: [Number],
required: true
}
}
});
// to add data use this format:
// loc: { type: "Point", coordinates: [ longitude, latitude ] },
- MongoDb Docs: findOneAndUpdate
- MongooseJs: How to Use findOneAndUpdate() in Mongoose
- Source: SO
// creating new document if already doesn't exist (note: we're passing {} as second coz we just want a new document with required fields (search fields in 1st argument)
await this.feedReplyLikeModel.findOneAndUpdate({ feedReplyId, userId }, {}, { upsert: true, new: true });
const doc = await Contact.findOneAndUpdate(
{ phone: request.phone},
{ status: request.status },
{ upsert: true, new: true }
);
# Comment from here: https://stackoverflow.com/a/7486950/10012446
# I don't think you need the {$set: ... } part here as its automatic form my reading
# ~Sahil: People use $set as well:
const doc = await Contact.findOneAndUpdate(
{ phone: request.phone},
{ $set: { status: request.status } },
{ upsert: true, new: true }
);
LEARN: ♥ ❣♥ ❣♥ Difference between upsert: true
and new: true
. Source: https://stackoverflow.com/a/44794886
They are completely different flags
- If upsert is true then if the row trying to be updated does not exist then a new row is inserted instead , if false then it does not do anything .
- If new is true then the modified document is returned after the update rather than the original , if false then the original document is returned
- Source: Mongoplayground
- Docs: $exists
Without $exists
:
# Collection
[{"key": 1,car: 10},{"key": 2}]
# Query
db.collection.aggregate([{$match: {car: {$ne: 20}}}])
# Output:
[
{"_id": ObjectId("5a934e000102030405000000"),"car": 10,"key": 1},
{"_id": ObjectId("5a934e000102030405000001"),"key": 2}
]
With $exists
:
# Collection
[{"key": 1,car: 10},{"key": 2}]
# Query
db.collection.aggregate([{$match: {car: {$exists: true, $ne: 20}}}])
# Output (notice that field which has no `car` key is not returned):
[
{"_id": ObjectId("5a934e000102030405000000"),"car": 10,"key": 1},
]
with mongoose you can do like: const ratingUsersCount = await this.movieUserStatusModel.count({ movieId, rating: { $exists: true, $ne: 0 } });
Source: Mongoplayground
Using $count
in aggregate: Docs
Similarly:
# Documents
[
{"_id": 400, movieId: 200},
{"_id": 500,movieId: 200},
{"_id": 600,movieId: 300}
]
# Query
db.collection.aggregate([
{"$match": {movieId: 200}},
{"$count": "totaItems"}
])
# Output
[{"totaItems": 2}]
Using $facet
: Click here
But this is a issue that I faced myself and seems there's no such fix for this werid behaviour: Click here
Drawback:
- We don't get created/updated document, so we need fo to make a
findOne
query explcitly - Do not rely on returned value i.e,
value.upsertedId
as this is only preset if document is created (not returned when its updated)
// its useful as it creates the record with necessary values if it already doesn't exist
await this.movieUserStatusModel.updateOne(
{ movieId, userId },
{ $set: { rating } },
{ upsert: true },
);
- You can create a shareable link via that button as well:
- To show output for individual stages
- We can use only these methods. You can view the entire docs by clicking on the Docs button on the mongo playground.
Source: Mongoplayground
Query code:
# collection documents
[
{"_id": 400,movieId: 200,rating: 1},
{"_id": 500,movieId: 200,rating: 6},
{"_id": 600,movieId: 300,rating: 5}
]
# query
db.collection.aggregate([
// We pass movie._id explicitly for which we want to compute $avg (otherwise it would be joining/lookup/populate for all movie documents which is too costly.
{$match: {movieId: 200}},
{$group: {_id: "movieId",average: {$avg: "$rating"}}}
])
Source:
- Mongoplayground (Template > Multiple Collections)
- $Avg
- $group
- $unwind - StackOverflow Awesome example (multiple nested array example)
# Template: Multiple Collections
db={
"movies": [
{"_id": 100},
{"_id": 200},
{"_id": 300}
],
"movieuserstatus": [
{"_id": 400,movieId: 200,rating: 1},
{"_id": 500,movieId: 200,rating: 6},
{"_id": 600,movieId: 300,rating: 5}
]
}
# query
db.movies.aggregate([
// We pass movie._id explicitly for which we want to compute $avg (otherwise it would be joining/lookup/populate for all movie documents which is too costly.
{"$match": {_id: 200}},
{$lookup: {from: "movieuserstatus",localField: "_id",foreignField: "movieId",as: "movieuserstatus"}},
{"$unwind": "$movieuserstatus"},
{$group: {_id: "$movieuserstatus.movieId",average: {$avg: "$movieuserstatus.rating"}}}
])
Cool!
Source: Click here
import mongoose from 'mongoose'
// TESTED: 6 Dec, 2023 (16.13.1), mongoose: ^7.6.3, mongodb (docker): mongo:4.0.9
// Drop entire database (which will drop all collections in one operation)
await mongoose.connection.dropDatabase();
// Delete all documents from the database
await Promise.all((await connection.db.collections()).map((collection) => collection.deleteMany({})));
Docs Mongodb: Click here
Source: Official Docs Mongodb: Click here
Source: Click here
// TODO: Remove this comment: [offset=5, limit=5<numberOfItems>] i.e., {$slice: [5, 5]} will bring array i.e., [5,9]
const [feedPost] = await this.feedPostModel.find(
{ _id: new mongoose.Types.ObjectId(id) },
{ likes: { $slice: [offset ?? 0, offset + limit] } },
);
Source: Click here, another similar here.
Recommened Read by Eric (especially @ ESR rule).
Read about it here: https://mongoosejs.com/docs/deprecations.html
Source: Click here (~ Credits: Eric)
#not_tested
You can Enable indexing for fields in backend server directly as well (otherwise we simply enable in the mongodb server In Compass
)
Transactions are new in MongoDB 4.0 and Mongoose 5.2.0. Transactions let you execute multiple operations in isolation and potentially undo all the operations if one of them fails. This guide will get you started using transactions with Mongoose.
Transactions in Mongoose DOCS: Click here
That means if you use wrong order then the query will not match the document:
- Docs -
$group (aggregation)
: Click here - Playground Link: mongoplayground.net
Playground Link: mongoplayground.net
# Documents:
[
{ "first_name": "Sahil" },
{ "first_name": "Mohit" },
{ "first_name": "Mohit" },
{ "first_name": "Mandy" },
{ "first_name": "Mandy" },
{ "first_name": "Mandy" },
]
# Query
db.collection.aggregate([
{ "$group": { "_id": "$first_name", "duplicates": { "$sum": 1 } } },
{ "$match": { "_id": { "$ne": null }, "duplicates": { "$gt": 1 } } },
{ "$sort": { "duplicates": -1 } },
{ "$project": { "first_name": "$_id", "_id": 0, duplicates: 1 } }
])
# PRO Tip: You can filter some data using a $match stage in the front of the aggregate query as well, for e.g, `{$match: {car: 30}}`
- Docs -
$group (aggregation)
: Click here - Playground Link: mongoplayground.net
Documents:
[
// good case: both users are deleted
{ "email": "Sahil", "deleted": true }, { "email": "Sahil", "deleted": true },
// good case: one user deleted, one active
{ "email": "Mohit", "deleted": true }, { "email": "Mohit", "deleted": false },
// *bad case*: one deleted user, two active user i.e, deleted=false
{ "email": "Mandy", "deleted": true }, { "email": "Mandy", "deleted": false }, { "email": "Mandy", "deleted": false },
]
# Query:
db.collection.aggregate([
// filter only deleted users
{ $match: { deleted: false, } },
// now we group by email
{ "$group": {
"_id": {
"email": "$email",
"deleted": "$deleted"// delete is *optional* here though
},
"duplicates": { "$sum": 1 }
}
},
{ "$match": { "_id": { "$ne": null }, "duplicates": { "$gt": 1 } } },
])
- Amazing - Mongodb Docs: Click here
- MongooseJs: Click here
- Stackoverflow Question: Click here
Q. (Please see ChatGPT's awesome explanations below as well) Why cursor and not batches (i.e, limit()
way)? Ans. Using the skip feature when iterating through results in batches can get slower over time for large data sets, when skip is given a high numeric value, so streaming with a cursor helps to get around that. ~Credits: Eric
From ChatGPT:
Solution: Click here
# notifications collection
mongoexport --collection=notifications --db=slasher_test --out=notifications.json -u=root -p=rootpassword --uri=mongodb://localhost:27017 --authenticationDatabase=admin
# users collection
mongoexport --collection=users --db=slasher_test --out=users-mocked.json -u=root -p=rootpassword --uri=mongodb://localhost:27017 --authenticationDatabase=admin
Solution of this problem: Click here
Stack overflow Anser: Click here
Docs: Click here
import { Connection } from 'mongoose';
export const dropCollections = async (connection: Connection) => {
await Promise.all(
(await connection.db.collections()).map((collection) => collection.deleteMany({})),
);
};
Source: Click here
// switch to database
use imdb;
// firstly we have to clean up potential remainders
db.dropDatabase();
// create admin user
db.createUser({
user: 'imdb',
pwd: 'simple',
roles: [{ role: 'dbOwner', db:'imdb'}]
});
This is for checking a point lies in given shape with given values of its edges and coordinates. (Not at all useful to check if a coordinate exists in 10kms range of another coordinate point on earth).
Sourc: Click here
Also, $geoWithin
works in similar way. Source: Click here
Even with this $near
operator mongodb doesn't support calculating distance b/w two coordinates. We would need to use some custom logic or some third party lib to get approx distance.
links for eric:
- Issue on Community (NOT RESOLVED): Click here
- Proposal on feeback forum (NOT RESOLVED): Click here
- Issue on Jira (NOT RESOLVED): Click here
Finding distance between two gps locations (point to point and not the actual travelling distance by roads)
Source: Click here
Also: In case you wanna calculate distance using Haversine formula: Stackoverflow Question
Docs: $geoNear: version4.0, ver6-latest
Docs: Geospatial Queries
I managed to calculate distance b/w two gps locations and returning the docs in nearest first order with just mongo's geospatial utils. ALSO: I tested with my nearby locations as well, it seemed to work for me.
Find this code in mongo-shell-with-watcher
folder, yikes! TESTED with my custom gps locations as well. Yikes!!
Source: Click here