Home

Regular tasks

• Tasks: Data maintenance
• Tasks: Represent CSV Schema
• Tasks: Request boundaries
• Tasks: Miscellaneous
• Troubleshooting

Overview

Browse the Represent website and read its API documentation to start.

At a high level, the represent-canada Django project and its three Django apps run represent.opennorth.ca and rarely change. Most of the work is feeding them data for boundaries, representatives, and postcodes.

The service is heavily automated and can be maintained by a single highly trained developer. The service's data should be correct, complete, consistent, timely, and from authoritative sources. Many checks exist to ensure these service standards are maintained.

Data sources

The project imports boundary data by cloning two repositories: one public and one private. Read Tasks: Request boundaries for more.

The project imports representative data by requesting a data_url for each representative set. The data_url points to scrapers_ca_app, which runs on Heroku.

scrapers_ca_app wraps scrapers-ca, which contains HTML and CSV scrapers. HTML scrapers are more expensive to maintain, so we regularly request municipalities with open data catalogs to publish CSV data.

In rare cases, we work with external partners to manually fill a Google Sheet whose headers match our CSV Schema. We do this rarely, because our automated systems can't determine when that manual data goes stale and because the source is non-authoritative. We have done this for provincial elections and for major users who want to run a large campaign against multiple municipalities; in the latter case, we create a representative set with a slug like campaign-set-# to indicate to other users that it is an irregular representative set. The Google Sheets are imported using CSV scrapers, in which we track the contact person and the creation date to measure timeliness. We collect the Google Sheets into a folder.

Other pieces

Open Civic Data Division Identifiers (OCD-ID) hold many pieces together, as described here. Without OCD-IDs, it would be difficult to maintain a consistent and coherent database.

We have also created API clients, including a Ruby client, Drupal module and WordPress plugin. Since the API has not changed in years, these have required no maintenance.

Documentation

Read the documentation of:

• Represent API
• represent-canada: the represent.opennorth.ca Django project, which depends on:
  • represent-boundaries: Django app
  • represent-postcodes: Django app
  • represent-reps: Django app
  • represent-canada-data: public boundary data
  • represent-canada-private-data: private boundary data
• scrapers_ca_app: the scrapers.herokuapp.com Django project, which uses:
  • scrapers-ca

Deployment

In addition to the instructions below, add a new maintainer as a collaborator on Heroku.

Read the deployment instructions of:

• represent-canada on dedicated servers
• scrapers_ca_app on Heroku (scrapers)

Other deployments are:

• image-proxy on Heroku (represent-image-proxy)
  • Caches photos, used by the scrapers-ca ca_sk scraper via AWS Cloudfront
• scrapers-ca-ruby on Heroku (scrapers-ruby)
  • Publishes MP's Twitter accounts, used by the scrapers-ca ca scraper