Home

Regular tasks

• Tasks: Data maintenance
• Tasks: Represent CSV Schema
• Tasks: Request boundaries
• Tasks: Miscellaneous
• Troubleshooting

Documentation

Read the documentation of:

• Represent API
• represent-canada: the represent.opennorth.ca Django project, which depends on:
  • represent-boundaries: Django app
  • represent-postcodes: Django app
  • represent-reps: Django app
  • represent-canada-data: public boundary data
  • represent-canada-private-data: private boundary data
• scrapers_ca_app: the scrapers.herokuapp.com Django project, which uses:
  • scrapers-ca
• represent-canada-csv-proxy: the represent-csv-proxy.herokuapp.com Sinatra app
  • Issue: Roll into scrapers-ca

Overview

At a high level, the represent-canada Django project and its three Django apps run represent.opennorth.ca and rarely change. Most of the work is feeding them data for boundaries, representatives, and postcodes.

Data sources

The project imports boundary data by cloning two repositories: one public and one private. Read Tasks: Request boundaries for more.

The project imports representative data by requesting a data_url for each representative set. The requests are either to scrapers_ca_app or represent-canada-csv-proxy.

scrapers_ca_app wraps scrapers-ca, which contains HTML and CSV scrapers. HTML scrapers are more expensive to maintain, so we regularly request municipalities with open data catalogs to publish CSV data.

In rare cases, we work with external partners to manually fill a Google Sheet whose headers match our CSV Schema. We do this rarely, because automated systems can't determine when the manual data goes stale. We have done this for provincial elections and for major users who want to run a large campaign against multiple municipalities; in the latter case, we create a representative set with a slug like campaign-set-# to indicate to other users that it is an irregular representative set. We track the contact information of the maintainers of these Google Sheets, and collect the Google Sheets into a folder.

A representative set whose source is a Google Sheet has a data_url containing represent-csv-proxy.herokuapp.com. A representative set whose source is an HTML or CSV scraper has a data_url containing "scrapers.herokuapp.com.

Other pieces

Open Civic Data Division Identifiers (OCD-ID) hold many pieces together, as described here. Without OCD-IDs, it would be difficult to maintain a consistent and coherent database.

We have also created API clients, including a Ruby client, Drupal module and WordPress plugin. Since the API has not changed in years, these have required no maintenance.

Deployment

Read the deployment instructions of:

• represent-canada on dedicated servers
• scrapers_ca_app on Heroku (scrapers)

Other deployments are:

• image-proxy on Heroku (represent-image-proxy)
  • Caches photos, used by the scrapers-ca ca_sk scraper via AWS Cloudfront
• scrapers-ca-ruby on Heroku (scrapers-ruby)
  • Publishes MP's Twitter accounts, used by the scrapers-ca ca scraper
• represent-canada-csv-proxy on Heroku (represent-csv-proxy)
  • Issue: Roll into scrapers-ca

Add a maintainer as a collaborator on Heroku.