Lessons

How to Query (Multiple Docs) Data from Mongo with Go

In this tutorial, I will show you the basics of doing a query for multiple documents in a Mongo database instance with Go code. I will be on local and provide you with all steps.

Setup

I'm assuming you have Mongo installed on your local machine. I also recommend installing Mongo Compass to make it easier to see your data.

I am assuming you have Go installed and have successful compiled at least one Go file. This would confirm your environment works. We will be using my Go boierplate code to save time on setting up our code structure. It follows a handlers, models, repositories, services, and utils structure.

In short:

• handlers handles the inbound request.
• models contains all structs that represent data.
• repositories has all methods for interacting with the database.
• services contains the business logic of the app.
• utils has any helper methods.

Download the Go boilerplate. Please give it a star on Github so I know people are still using and enjoying it. Move the files into your new project root. Open the files up in a text editor.

You need to do one thing for this boilerplate to work. Open services/user_service.go and within the IsValidPassword function. You will need to switch the variable to true. Do NOT use this code in production without fixing it. I leave encryption and password requirements up to you. This will store a plain text password till you add code to GetEncryptedPassword and IsValidPassword. For demo and dev purposes, this is fine.

You will need data in your database and checkout the insert tutorial.

Query Code

As you can imagine, we will be working with the repositories layer since it works with the database. If we open the repositories/cars_repository.go file, you can see the List method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
func (c *CarsRepository) List(email string, query models.ListCarQuery) ([]models.Car, error) {
	filters := query.Filter(email)
	var cars []models.Car

	options := options.Find()

	// Add paging
	options.SetLimit(int64(query.Limit))
	options.SetSkip(int64((query.Page * query.Limit) - query.Limit))

	// Add timestamp
	options.SetSort(bson.M{"created": -1})

	cursor, err := c.db.Collection("cars").Find(context.Background(), filters, options)
	if err != nil {
		return []models.Car{}, err
	}

	for cursor.Next(context.Background()) {
		car := models.Car{}
		err := cursor.Decode(&car)
		if err != nil {
			//handle err
		} else {
			cars = append(cars, car)
		}
	}
	return cars, nil
}

In the code above, we build a filter using the query parameters. We set the options for paging. Next, we sort by a column. In this case, we used most recently created. We then query for the cars and build a slice of them to be returned. The only issue with this method is the err within the loop. If a car cannot be decoded, it is simply excluded. You can choose your own way of handling the error. Whether it would be building another list, grabbing an id for an error list, etc.

Let's take a closer look at the query attribute being passed in. In models/cars.go you can find ListCarQuery.

1
2
3
4
5
6
7
type ListCarQuery struct {
	Page  int    `json:"page"`
	Limit int    `json:"limit"`
	Make  string `json:"make"`
	Model string `json:"model"`
	Year  int    `json:"year"`
}

It takes a few search attributes. If you look at the Filter method, it uses Regex. I explain how to use regex and Go in another lesson. You can find out how model, make and year values are used in that lesson. The two main attributes we are focusing on are Page and Limit. Limit is the maximum number of elements someone is allowed to show. This helps limit long living queries, and taking too much of the database resources. Page allow us to iterate through sets of results. This is why we need to sort or results in the List method.

What if page or limit are not provided? How do we set them? You can look in handlers/car_handler.go and at GetAll. Right at the top of the method, we grab the query parameters.

1
2
page := c.DefaultQuery("page", "1")
limit := c.DefaultQuery("limit", "25")

We then convert them from being strings to being integers. This stops the requests that could be ?limit=foobar. We then build our model object with all the set values. This block of code does lack a few extra checks. We should be checking to see if page is greater than 0. We should be verifying that limit is within some bounds (Ex. 1 to 30). Technically, a user can go /cars/?page=1&limit=10000000 and pull the entire database. Right above the code to setup the new ListCarQuery, you can add:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
if (pageInt < 1) {
	c.JSON(400, gin.H{"message": "error: page must be greater than one"})
	return
}

if (limitInt < 1 || limitInt > 30) {
	c.JSON(400, gin.H{"message": "error: limit must be be between 1 and 30"})
	return
}

query := models.ListCarQuery{
	Page:  pageInt,
	Limit: limitInt,
	Make:  make,
	Model: model,
	Year:  yearInt,
}

When you do not provide a model, make or year. The only attribute in the filter is email. This just validates only your "cars" are being returned.

1
2
3
4
5
6
7
8
9
10
11
12
13
func (q *ListCarQuery) Filter(email string) bson.M {
	andFilters := []bson.M{
		bson.M{
			"email": email,
		},
	}
	... Hidden Some Code ...
	if len(andFilters) == 0 {
		// Handle empty and, since there must be one item.
		return bson.M{}
	}
	return bson.M{"$and": andFilters}
}

Demo

In your terminal:

1
go run main.go

The first step is to run sign up or sign in. I execute my sign up request with Postman. Copy down the token in the response.

Or you can run this curl command:

1
2
3
4
5
6
curl --location --request POST 'http://localhost:8080/user/signup/' \
--header 'Content-Type: application/json' \
--data-raw '{
 "email": "me@keithweaver.ca",
 "password": "demodemo1"
}'

Write down the token. Connect to your database and confirm that you have multiple values.

Next, you will submit a GET request.

Or the curl command:

1
2
3
curl --location --request GET 'http://localhost:8080/cars/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <YOUR_TOKEN_FROM_SIGN_UP_RESPONSE>'

And the response returns a list of cars:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
    "cars": [
        {
            "id": "5f8045a605b78ca65b72193b",
            "make": "Jaguar",
            "model": "XK",
            "year": 1957,
            "status": "",
            "email": "me@keithweaver.ca",
            "created": "2020-10-09T11:12:38.491Z"
        },
        {
            "id": "5f80458205b78ca65b72193a",
            "make": "Jaguar",
            "model": "F-Type",
            "year": 2020,
            "status": "",
            "email": "me@keithweaver.ca",
            "created": "2020-10-09T11:12:02.001Z"
        },
        ...
    ],
    "message": "Cars retrieved"
}

Here is a screenshot of my request:

Thanks for reading!