Disclaimer: This will be a data and programming heavy blog post. For a higher level overview, read this.
The City of Syracuse is one of the snowiest cities on record within the United States. Given this fact, it was deemed important that the residents of Syracuse should be able to see when their streets were plowed. Earlier in 2018 the Office of Accountability, Performance, and Innovation held a Civic Hackathon in partnership with Syracuse University’s iSchool to find insights on plow data. This project was a continuation of the mission of that hackathon to tackle our snow problem. Also, the snow plow mapping tool would decrease the number of calls coming through to Cityline saying a street was not plowed. The City of Syracuse’s snow plow mapping application is an in-house project, created by me (Edward Deaver, IV), a computer science intern with the City.
The creation of the mapping software took the form of three parts: Planning, Development, and Deployment. Planning was a 3.5 month process, while Development took 2.5 months, and deployment is ongoing. The following post details the first iteration of the program.
The proof of concept had 2 key parts:
Operate in near real time: showing streets being updated in 1 to 10 minute intervals. This posed the challenge of requiring specialized software to operate a geofencing server. The server would store geofences and be able to associate actions with them (i.e., if a truck enters the area, send a message with truck number).
The application had to be light on the user’s browser. Initial testing produced 10-20 megabyte web pages that would throw a “Page Unresponsive” error (Chrome, Firefox, Edge) on computers. This problem would be solved through using minified JSON files (as small as a JSON file can get) as data sources for a lightweight client to load and then store into memory, reducing the resources required for initial access.
Planning was a two part process: Assessment and GIS research. Assessment started at understanding the capabilities of the Application Programming Interface (APIs) to which I had access. The first of those was our vehicle tracking API. The vehicle tracking API returns City vehicle locations as GPS coordinates: latitude, longitude. Translating those coordinates into a city street block is a two step problem. Using geofences, a polygon of coordinates that have an action attached to them, I was able to generate a geofence per city block and track when a vehicle encountered that fence.
Above: Planning of Snow Plow Map architecture and processing.
The development process took the form of a modified agile workflow with small proof of concepts being produced and eventually combined for a larger product. The development process was also an experiment for using Amazon’s AWS(EC2) as a development environment alongside using other Amazon services for the stack: DynamoDB(NoSQL) for the database and S3 for static site hosting (Like Google Drive, or Github). The first step using the City Streets 2011 KML (file format used to display geographic data) found at data.syrgov.net was to convert the four corner street addresses of a block for a CITYST_ID block into usable geocoordinates (geocoding). This created 4 unique points that could be combined to form a polygon shape for a specific city block. Then those pairs of points would be stored in arrays per CITYST_ID.
This process allowed the creation of visualizations of the geofences and filtering of them before they were implemented into a larger program. The program was designed so that the geofences were fed into a Geofencing server, Tile38, then sent to a message broker, RabbitMQ. From there, a python program would listen to the broker to make updates to a DynamoDB database. This design for the geofencing was too complicated to bring to fruition.
In order for the system to map the street data coming from the server-side messages to the front end, the system must establish a universal primary key: CITYST_ID from the initial KML file. Within the DynamoDB database, the live street data had its own table with the primary key of CITYST_ID. The idea was Tile38 would have webhooks named after the CITYST_ID of the geofence they represented. So when the webhook was triggered the system would know a vehicle was in that geofence.
Another part was getting the data from the City’s Snow & Ice vehicles (this includes snow plows). This was fulfilled by a python function that called all of the snow plow trucks, stored their data into an array, then created a minified JSON file of that data, and transferred it to the S3 bucket (similar to a folder in Google Drive). The client browser would load that and then store it into memory, reducing the resources required for initial access.
Above: Example of a vehicle location JSON message.
* The time the vehicle tracking API received the message from the vehicle. **Unit way of keeping time within a system. It uses the seconds that have elapsed since 00:00:00 Thursday, 1 January 1970. It provides an easy way to sort dates without having to use other date formats(Ex. Jan. 1, 1970 00:00:00).
Once the backend architecture was taking shape, I moved onto the client side part of the application. The client side was designed to be light in terms of initial download. This created a few different consequences. While yes, the initial download would be an order of 75% lighter than a fully constructed foilium (leaflet map creation library) html file map, it generates all of its objects on the users device except for some assets stored on Content Delivery Networks (servers around the globe providing high speed access to media or other assets). In all tests I conducted this method was faster. Loading a completed map (10mb-20mb file) would crash the chrome instance I was using, and freeze the machine, because browsers are not designed to download excessively large html files, but they do well with gradual streams of data such as media files.
The application was developed using Python which does not natively compile into a distributable binary (Example: Window’s EXE), unlike languages like Java(jar) or C. To combat this I created an image of the machine I was working on and used that for deployment. The application’s multiple external dependencies made it hard to distribute as well, because they all needed extensive setup processes to enter in all relevant API keys and passwords.
The Current Solution
After presenting my work to Sam Edelstein, the City’s Chief Data Officer, he was able to build a solution in R (statistical programming language). His solution utilized a 60 minute update cycle, and historical data. Each update pulls a 60 minute chunk of data from our vehicle tracking API. That point data is then fed into an API that can turn the points into direction data, similar to Google Maps. From there, that direction data is cross referenced with the City Streets 2011 shapefile to create a historical record for that 60 minute time frame of when streets were last plowed.
The snow plow map can be found at: https://cityofsyracuse.github.io/snowmap/snowmap/snowplow_map.html