An introduction to choropleth mapping using renovation permits in Toronto

Context

In this lab, we will explore the challenges of mapping messy urban data. We will be sourcing from Toronto’s open data portal with a particular focus on renovation permit data. Renovation permits are an intriguing indicator of the intentions of property owners to modify the city’s built form, so visualizing them spatially could provide an intriguing view of urban change across the city. We will be using QGIS which, like many other open source tools, has entered the mainstream in recent years.

The lab is primarily focused on data engineering and cartographic styling, and provides a tutorial on these which you should work from as you take on your own self-directed exercise in section 8.

Objectives

Guidelines

1. Fetching the data

To do this lab, download the 3 following datasets.

  1. Toronto’s cleared building permits: download since 2017 in CSV format.
  2. Toronto’s georeferenced addresses select the MTM10 projection from the dropdown and download in the GeoPackage format.
  3. Toronto’s neighbourhood geometries select the MTM10 projection from the dropdown and download in the GeoPackage format.

2. Setting up your workspace

  1. It’s important to set a folder where you will be storing your working files. This keeps things neat, avoids misplacing any files you will be creating, and also makes your work portable. For this lab, we will create a folder on the Desktop.
    1. On your Desktop (Win+D), right-click > New > Folder. Name the folder qgis-lab
    2. Move the files you downloaded into this folder.
  2. Open QGIS Desktop 3.40.8.
  3. Once inside, Project > New to create a new project.
  4. Before doing anything else, you should make sure to set the save location for your current work session: Project > Save As…. Navigate to your qgis-lab folder and give it a similar name (such as just qgis-lab). This saves your work session as a .qgz file, which is essentially a compressed folder of all your settings and configurations that can be opened only in QGIS. This file will not contain your data!
  5. The Browser panel on the left is where you can access data on your machine from within QGIS. Here, we will create a shortcut to the folder workspace you created, so you can keep track of what’s in your folder as you progress:
    1. Right click Favorites > Add a Directory…
    2. In the system browser, navigate to and select the folder you created which should be under Desktop. Once that’s done, you should see the folder and its contents listed under Favorites. If you ever add new contents to the folder, simply right-click and hit Refresh.
  6. Finally, we will create a geopackage to store all our data. Right-click on the folder you added to your Favorites list > New > Geopackage… and give it a meaningful name (Right-click it > Manage > Rename), such as data.

A geopackage is an open source standard for storing geographic data. What’s both interesting and confusing about it is that we can store individual datasets as geopackages, but also hundreds of them inside a single geopackage. Think of a geopackage as a specialized geospatial folder that can contain all your data. This makes your workspace much easier to manage and move around…

Put all your datasets into the geopackage you just made. This will greatly facilitate things down the line, notably since QGIS doesn’t work well with raw CSV…

  1. To do this, click the dropdowns for the addresses and neighbourhoods geopackages to reveal their contents and drag their contents into your project geopackage (data.gpkg).
  2. For the CSV, you might need to right-click > Export Layer > To File… and then select your data.gpkg as the input for File name, the click OK.
  3. Right-click your data.gpkg and click Refresh to view your imported data and make sure everything is there.

3. Inspect the data

You have three datasets, two of which are geospatial. Our goal is to create a map of renovation permits by combining these three. Why do we need to combine the three?

  1. Drag your three datasets from your data.gpkg in your browser pane into the map view. The map view should automatically visualize the geospatial datasets for you. You will also see these files listed as layers in lefthand ToC/Layers List.

4. Cleaning up the permits dataset

Some basic tabular data manipulations can be done from within QGIS, which avoids us having to export the data to Excel for doing so.

  1. Open the permits table by right-clicking the file in the ToC > Open Attribute Table.

Each row in the permits table has a permit number as well as a revision number (REVISION_NUM). The data dictionary tells us that revisions are made to a same permit, meaning they might represent corrections after feedback made by urbanists to applicants, for example. We do not want to give more weight to permits that got revised multiple times, so we will need to treat revisions to a same permit as duplicate rows to be removed. Our goal: each row represents one unique renovation permit. To remove the revisions:

  1. First, display the Processing Toolbox by selecting it under View > Panels. It should then appear as a tab on the bottom-right.
  2. Processing Toolbox > search for Delete Duplicates by Attribute
  3. Load the permits table as the Input layer.
  4. Set Field to match duplicates by to PERMIT_NUM.
  5. As the Filtered (no duplicates) output, we will save it as a new file in our geopackage: click the … > Save to GeoPackage, and in the popup, browse to and select your data.gpkg. In the popup that appears, give your output a name (i.e. unique-permits).
  6. Right-click > Refresh the data.gpkg in your browser panel and drag your output data into your workspace if it’s not already there.

Doing this keeps the first feature it the algorithm hits for each permit number (keeping only an arbitrary revision), which is fine for quantifying permits by neighbourhood. The process should have removed many thousands of rows!

We would also like to only keep recent permits. Our data go back to 2017, a long time ago. To focus the dataset temporally, we could cut it off at around the time of the last census, which would give us a snapshot of the last 5 years of urban change in Toronto.

  1. Open the unique-permits attribute table.
  2. Click the Select features using an expression icon and in the popup, paste the following in the expression box: to_date("APPLICATION_DATE", 'yyyy-MM-dd') < to_date('2020-01-01').
  3. Exit the attribute table and right-click on unique-permits > Export > Save selected features as….
  4. Under File name, make sure to browse to and select your project geopackage (data.gpkg), then as a layer name, you can name it something like unique-permits-recent. click OK.

5. Joining tables

In this processing step, we will join the addresses data to the permits data. We will join these datasets thanks the addresses information contained in both datasets.

5.1 Creating a join key

Notice that STREET_NAME in the permits table is all uppercase, and LINEAR_NAME in the addresses file uses title case. In addition, the street names and civic numbers are in separate fields/attributes in both files. We need full addresses (actual locations) to connect the address geometries to our permits. We will do this by standardizing the street names and combining them with the civic numbers in a new field. This join field will presumably allow an address in the addresses file to match the same address in the permits file…

  1. Inside the popup attribute table of your permits data, click Open field calculator (Ctrl+I). The field calculator is very useful for calculating attributes from feature geometries or other attributes.
  2. We will be creating a new field (column) for the percentage of cycle paths: for your Output field name, enter join_key.
  3. Choose Text (string) as the Output field type.
  4. In the Expression box, you will enter your equation: to_string("STREET_NUM") || ' ' || lower("STREET_NAME")
  5. Hit Apply and, if successful (it will take some time), then hit Ok. your attribute table should now have a new field at the end called join_key containing cleaned addresses.
  6. Repeat the same steps for your addresses data, but make sure to replace the appropriate field names in the expression box (i.e. replace with ADDRESS_NUMBER and LINEAR_NAME).

5.2 Generating coordinates for the join

We want to map renovation permits, not addresses. This means we should join the addresses file to our permits table, rather than the other way around. The problem with the traditional join is the following: if we join the addresses to the permits, we lose the address geometries since QGIS only keeps geometries from the primary join dataset (what is being joined to). If we join the permits to the addresses, many permits are discarded since there is often more than one permit per address (condo buildings addresses contain hundreds of units at one street address). So we need a workaround…

The workaround will be to create text columns representing coordinates from the address geometries. We will then join these columns like any other column to the permits, which will then allow us to locate the permits.

  1. Open the attribute table of your addresses layer and then the field calculator.
  2. You will again be creating new fields with Output field names called xcoord and ycoord respectively, and an Output field type of Decimal.
  3. In the expression box, you will enter $x and $y respectively.
  4. Once you’ve applied the changes and clicked OK, you might need to click the Toggle off editing mode icon and save your changes.

You should now have 2 new columns containing the eastings and northings for each address in metres (these are coordinates specific to MTM zone 10, the local projection used for accurately mapping Toronto).

5.3 Doing the join

  1. Search the Processing Toolbox for Join attributes by field value. This is the best way to join in QGIS.
  2. Choose your cleaned up permits layer as the first Input layer, and select join_key as the Table field.
  3. Choose your addresses file as the Input layer 2, and select join_key as the Table field 2.
  4. Select Discard records which could not be joined.
  5. Choose your save method: temporary or to disk in your data.gpkg (the disadvantage of a temporary layer is that it will be lost if QGIS shuts down accidentally).

5.4 Generating points from the coordinates

The final step will be to use the joined coordinates to turn our permits into points.

  1. In the Processing Toolbox, search Create Points Layer from Table.
  2. Select the joined layer as the input layer
  3. For the X and Y fields, select the xcoord and ycoord columns.
  4. For the Target CRS, select the Project CRS (MTM10).
  5. Choose an output strategy (temporary layer or to disk) and hit Run.

You should now have your permits mapped! Make sure they’re saved to your data.gpkg. If not already, right-click your output points layer > Make Permanent to save it to your geopackage.

6. Aggregating points to the neighbourhood level

Since we want to use neighbourhoods as a framework for mapping in this exercise, we need to quantify how many points (permits) there are per neighbourhood. This will require a simple spatial join, unlike the previous tabular join.

  1. In the Processing Toolbox, search Join attributes by location (summary).
  2. Join to features in > choose your neighbourhoods layer.
  3. By comparing to > choose your permits points file.
  4. Under Fields to summarise > select fid is fine.
  5. Under Summaries to calculate, select count (number of permits per neighbourhood is all we need).
  6. Select your output/save strategy and click Run.

You should now have a neighbourhoods file with a column fid_count that tells you how many permits there are in that neighbourhood.

7. Creating a map

Now that some data is prepared, we can move from the GIS component of the lab to the cartography component. Our goal here is to make a simple choropleth map of permits.

  1. QGIS offers many tools for producing beautiful maps. These can be found in the Layer Styling pane, which may not be visible. To open tit: View > Panels > Layer Styling. It should appear on the righthand pane, but you can resize or drag and drop it anywhere. At the top of the pane, under Layer Styling, make sure your final polygon file is selected. This selection ensures you are styling the right layer.
  2. To create a choropleth map, first make sure the Symbology vertical tab is selected:
    1. Select Single Symbol which should reveal a dropdown, allowing you to choose a styling approach for polygons: select Graduated.
    2. For Value, select your fid_count field.
    3. At the bottom, look for the Classify button and click it. This should generate your first choropleth map!
    4. You can select a classification scheme by selecting from the Mode dropdown. You can also modify the number of Classes to the right. You can then explore how your scheme relates to the data’s frequency distribution by clicking on the Histogram tab and then clicking Load Values. Try different schemes. Which one do you prefer? Which tells the most compelling story?
    5. Select a Color ramp from the dropdown to adequately communicate the cycling theme (red may not be ideal).
    6. Finally, you can also change the outlines by clicking the Symbol dropdown > Configure Symbol…. Don’t be confused by the fact that the fill colour may not reflect your graduated colour ramp, which overrides the single-colour fill. To modify the outline, select Simple Fill and modify the Stroke color, width and style.
  3. To add and edit labels, click the Labels vertical tab:
    1. Click the dropdown that appears to select Single Labels
    2. As the Value, select AREA_NAME to render the neighbourhood name as each polygon’s label.
    3. There are many settings you can explore to improve your labels and make them more visible, organized as submenus under the horizontal tabs. To add a halo effect for example, click the Buffer tab and check off Draw text buffer. You can also play with label placement by selecting the Placement tab further to the right.
    4. You can also load in and style other geospatial layers from Toronto’s open data portal, such as streets, railways, waterways, parks, etc. As long as none of this distracts from the main objective of your map!

8. Correcting this very flawed map!

There are major issues with this map, however. Fundamental ones, actually. Can you name why?

First, choropleth maps cannot display raw count data. This is due to the modifiable areal unit problem. Data displayed on a choropleth map need a common denominator, since areal units (neighbourhoods) have varying sizes which can over- or underrepresent magnitude in any given location based simply on the size of the neighbourhood (a larger neighbourhood will often have more permits, but that says nothing about the magnitude or density of permits in that neighbourhood compared to another). To solve this, we can consider doing the following:

There’s a challenge for you! You have the skills to do this. Good luck!

9. Publishing your map

Preparing your map for export is an important final step that involves much cartographic decision-making, since this is where your map layout is designed.

  1. To start creating your map layout, Project > New Print Layout. Name your layout as you choose (often, mapmakers will have many print layouts already prepared for use depending on the kind of map they’re making). Click OK.
  2. This should open the print composer window, where you will add all your map elements:
    1. First, create a ‘mapped area’ by selecting the Add Map icon on the left. Then drag and drop a rectangle to create the mapped area within your canvas. To choose an appropriate extent and zoom scale for your mapped area, press C or the Move item content cursor to pan around. Return to the Select/move item (V) cursor when you’re done. If you ever make changes to your map in the main window that you want to see reflected in the mapped area of your print layout, make sure to click the refresh button under Item properties in the righthand pane with your mapped area selected…
    2. Find the Add Legend icon and drag and drop to create the legend area. This should generate an automatic legend.
      1. You might find that this legend is too cluttered, since it contains all layers loaded into your current session. To clean up the legend, make sure it’s selected and on the right, click the Item properties tab and under Legend Items, uncheck Auto update[^2], then delete all the layers except for your choropleth (you can do this by selecting the layers and hitting the red minus icon).
      2. To remove the layer name itself, right-click it and select Hidden. Now your legend should look slick!
    3. Add a scale bar, as well as a title and other important textual information. Like with the legend, you can make many design choices in the Item properties tab for each selected map element…
  3. Now for some finishing touches: to tighten up your print layout around your map elements, right-click anywhere on the canvas and click Page Properties…, then under the Layout tab in the righthand pane, click Resize Layout under Resize Layout to Content.
  4. To export your map, you can select various formats… Selecting SVG will allow you to make more detailed changes (such as precise label placement) in a vector graphics editor such as Inkscape. If you wish to avoid doing that, Layout > Export as PDF.