Introduction: Flooding & the Urban Microbiome

Hello and welcome to my first instructable! This guide serves as a (mostly) comprehensive documentation of the work I completed for my senior project in Integrated Digital Media at the NYU Tandon School of Engineering. While this is a step-by-step tutorial process behind what I made, some parts are noticeably missing some detail. I still go through what I did, but in those parts I don't explain exactly how I did it, simply because at the time of writing this, I'm still working on this project. However, parts of this will be directing you to other online tutorials that explain things much better than I can and have already written everything out so I don't need to!

First link: https://engineering.nyu.edu/news/changing-microbi...

This is the student research project I was a part of at NYU. Essentially, the Gowanus neighborhood in Brooklyn is prone to frequent floods, which can leave harmful pathogens on surfaces even after flood waters recede. I was tasked with creating a "flood monitor" that would alert us when flooding occurs, so we could rush down to the site and collect samples of the water and surfaces. Before we go any further, let's take a closer look at some of the most important components of this project and understand why they're necessary.

Raspberry Pi

The Raspberry Pi is basically a mini computer capable of a lot of things. This project could also probably be done with Arduino, but the pi has a lot of functionality and ease of use that Arduino doesn't. In this project, the pi acts as the "brain" to which all other components/sensors are connected, and handles everything that needs to happen in terms of data collection/transfer/alerting/etc. All components listed below are connected to the pi.

Read this for more info:

https://www.raspberrypi.org/help/what-%20is-a-rasp...

eTape Liquid Level Sensor

The main component in this project is called the eTape Liquid Level Sensor, which has a range from 0-12 in. It's commonly used for measuring the depth of liquid substances - in this case, it will be used to measure flood depth. Since this is an analog resistive sensor, the value of resistance that it outputs will change based on the water depth. This is how we'll know whether it's flooding. For example, with no water at 0 inches, my sensor outputs 2248 Ω. At 12 in, it outputs 490 Ω. Of course, these values will differ slightly for each sensor, but in general it is an inverse relationship: as flood depth increases, resistance decreases, and vice versa. So, if we know the resistance is 1450 Ω, we can say the water depth is around 6 in. The only way to find your sensor's depth vs. resistance relationship is through testing.

MCP3008 ADC

One thing to note about Raspberry Pis (at least current models) is that they don't accept analog inputs, only digital. Digital inputs are like buttons, where the input can only be on or off, 1 or 0. Analog inputs include things like a temperature sensor, potentiometer, or this eTape sensor, where a range of values could be inputted. The MCP is an analog-to-digital converter, which will allow the pi to interface smoothly with the eTape sensor.

Real-Time Clock (RTC) Module

The RTC module is for time-keeping and will continue to keep the correct time even if the pi loses power, since it has its own button-cell battery. Raspberry Pis automatically get the time and date when connected to the internet through a protocol called NTP. However, if the WiFi is down or the Pi loses power, we still want the correct time/date to make sure our data is accurate; thus, the reason for the RTC.

So what does this device actually do? Glad you asked! It's 3 things, mainly:

  1. Data Collection
  2. Data Transfer
  3. Notification/Community Interaction


Data Collection

When this device is deployed in the Gowanus community, it will constantly be collecting flood data, and flood data only. The 4 things that are currently being recorded are time, date, voltage, and resistance. The time and date are provided by the RTC, and the voltage/resistance by the eTape. This data is logged to a csv file and stored on the pi's sd card.

Data Transfer

Every 15 seconds, pi will upload the resistance value to an IoT platform called Thingspeak. Thingspeak is an amazing service that can connect different apps and other things (more on this soon). It's also great for visualization, which is what this project does. On Thingspeak, I created a channel and set up a resistance vs. time graph as well as a resistance counter, so any time I access the site, I can see the values that are being outputted.

Notification/Community Interaction

Another powerful feature that Thingspeak has is to connect other apps and services to your channel. I created a Twitter account dedicated to this project and set certain thresholds for resistance, then linked Twitter to Thingspeak to create a twitter bot. So, for instance, is the resistance is less then 2000 Ω, the bot will send out a tweet with something like "it's flooding! depth is around 2 inches."

In addition, a QR code on the device itself links to the twitter account, so residents of the Gowanus community can follow the account and get updates on whether their neighborhood is flooding or any other news.

Supplies

Here is everything (more or less) that you'll need to complete this project:

Components

  • Raspberry Pi 3 (or any other WiFi-enabled model)
  • 8 GB (at least) Micro SD Card
  • Cobbler Breakout Board (to connect Pi 3 to breadboard)
  • Raspberry Pi W 0
  • eTape Liquid Level Sensor
  • MCP3008
  • RTC Module

Tools

  • Soldering iron & solder
  • Multimeter
  • Pliers
  • Wire cutters/strippers
  • External Monitor/Keyboard (for initial Pi setup_

Materials

  • Wire (lots of it)
  • Breadboard

Software

  • Mac OS system
  • Terminal
  • Python
  • Thingspeak
  • Twitter

With the intro out of the way, let's get into the actual process of building this project! Note: I created and coded this project using a Mac; I'm sure it's possible using Windows/Linux but this tutorial will assume you are using a Mac computer.

Step 1: Setup

Step 1: Get your Raspberry Pi!

The first thing we'll do is go through setting up the Raspberry Pi, and for this, I'll direct you to this amazing tutorial from the raspberry pi website: https://projects.raspberrypi.org/en/projects/raspb...

This tutorial assumes you have things like a micro sd card, monitor, keyboard, power supply, among a few other things. It also assumes your sd card is formatted, which you can do through disk utility on a mac. However, if that doesn't work, check out this forum which explains troubleshooting and formatting through the terminal: https://apple.stackexchange.com/questions/220744/c...

You might not always have access to a monitor and external keyboard, so it would be good to set up SSH as well, which is secure shell from your computer (the client). With SSH, you can control your pi from your own computer through the terminal. But before disconnecting your monitor/keyboard, open up the terminal from the top left hand corner of the interface and type in:

hostname -I

This will give you the IP address of your pi, which we'll need later on.

For SSH, again the raspberry pi site has great documentation on how to set this up: https://www.raspberrypi.org/documentation/remote-a...

After you've connected your pi to WiFi, enabled SSH in the settings, and have it up and running, you can go on your client machine (your laptop) and open up the terminal. In the terminal, type

ssh pi@<IP>

Substitute with the ip address of your pi, which you found before.

Below is another useful tip on adding WiFi network credentials. If you're on SSH or for some reason don't have access to the GUI (the sd card may become corrupt, for instance), adding WiFi network creds through the terminal isn't so hard.

Navigate to the top level directory of the pi. So, if you're in /home/pi, type:

cd ../../

Now write "ls" to see a list of everything that's there. You should see a directory called "etc." type:

cd etc

to navigate inside the etc directory. "ls" again and you should see a "wpa_supplicant," which you'll also want to "cd" into. "ls" once more and you'll see a .conf file called "wpa_supplicant.conf" which holds all the network credentials. Do:

sudo nano wpa_supplicant.conf

to open the file for editing. You may already see your WiFi network there if you've connected your Pi to the Internet. In the file, you can add or remove creds by simply listing them one below the other or deleting them, respectively. Use this format:

network={
        ssid="SSID"
        psk="PSK"
}

and replace SSID with the name of your WiFi network and PSK with its password.

With all this set up, let's move on to testing!

Step 2: Testing: LEDs and Potentiometers

The reason for testing is to make sure each component works individually before connecting them all because when the circuit stops working and breaks (and it will) then you'll have no idea which component is causing it. So, let's make sure the Cobbler board is functioning first.

The Cobbler breakout board essentially mirrors the pins on the raspberry pi and "breaks" them out on a breadboard so we can access them easier. It has two parts: one is a board and the other a ribbon. When you buy it, make sure that your cobbler works with your pi - some boards may not work with the Rpi 3, for instance. Your board might be T-shaped or it might be straight, which doesn't matter as long as you confirm it works with your pi; both have the same functionality in the end. The T-shape allows you to see and connect pins more easily, but takes up more room on the breadboard.

Before doing anything else, it's good practice to unplug your pi and make sure it's off before connecting any components. If you have a monitor/keyboard you can shutdown the pi from the menu. If you're on SSH or want to use the terminal, type

sudo shutdown -h now

and wait for the pi's blinking green light to stop. Then, unplug the pi and you're good to go!

Now, insert the cobbler board into your breadboard (pictured above). Insert one end of the ribbon cable into the cobbler board and the other end into your pi. I don't think the orientation of the cable matters (but please correct me if I'm wrong!). Refer to picture above for what it should look like.

LED Blink Test

Let's do a simple led blink test to make sure the cobbler and pi are functioning correctly. Pi My Life Up is an amazing, amazing resource for anything raspberry pi-related and has a great tutorial on youtube that goes through what GPIO pins are, how to use them, the process of hooking up an led to the pi, and writing code to make it blink.

Here's the link:

A couple things to note: first, he's using the T-shaped cobbler mentioned before, which makes accessing cobbler pins easier. He also goes through enabling I2C and SPI protocols in the raspi-config menu, which are methods of communication between components and the pi. Again, after doing this make sure to safely shut down your pi and remove from power before wiring the led circuit. When running your led_blink.py script, if

sudo python led_blink.py

doesn't work, check which version of python you have. Check the minor version of python with

python --version

and check the major version of python with

python3 --version

In this guide, I used Python 3 for pretty much everything - there are times when I tried the above command to run code, which didn't work. Then when I used python3 in place of python, it worked! So if you run into problems, try this:

sudo python3 led_blink.py

And your led should be blinking! In general, you should default to using python3 over python.

Potentiometer Test

Next, we'll hook up the MCP3008. Again, this IC (integrated circuit) is a chip that will allow your pi to read in analog inputs, which it cannot natively do. And... here's another outside tutorial to help you with this!

https://learn.adafruit.com/reading-a-analog-in-and...

This awesome tutorial from Adafruit goes through the process of hooking up your MCP chip to the cobbler and connecting it to a potentiometer (POT). The reason we're using a POT is because it pretty closely simulates the eTape sensor - as both sensors are analog inputs, they'll provide a range of values. Sensors connected to the MCP use SPI to communicate with the pi. Refer to above pictures for how my circuit turned out.

Also, here's the datasheet for the MCP3008, which has a pinout schematic: https://cdn-shop.adafruit.com/datasheets/MCP3008.p...

Run the code, and see if you're getting readings from the potentiometer! The Adafruit tutorial is intended for controlling volume, I think, but for our purposes we won't need speakers to see if it works. Turn the POT knob left and right and hopefully, the values outputted in the terminal should change accordingly. If so, let's move on!

Step 3: Attaching the ETape Sensor

With all this done, we can now add the actual sensors! At this point, you should have tested your pi, cobbler, and MCP to ensure all are fully functioning. Let's remove the potentiometer from the circuit and the accompanying wires and hook up the eTape sensor. I would also recommend picking up a graduated cylinder or some other form of enclosure so you can test the tape by pouring and removing water.

Now, there are two different 12" eTapes you can buy, both of which I found on Adafruit. One of them is this 4-pin one: https://www.adafruit.com/product/464

And the other this 3-pin one: https://www.adafruit.com/product/2656

Pros and cons of both: The 4-pin one seems to be the one that more people are using, so there is more online documentation and code for how to use it. You may have to use a voltage divider (explained below) in your circuit, but the wiring is fairly straightforward. It's also much cheaper.

Another great tutorial I found by Daniel Porrey on hackster.io for 4-pin which goes through everything in a very detailed way: https://www.hackster.io/porrey/etape-f4b457#overvi...

That tutorial provides a lot of insight into connecting the tape sensor, but won't work with the 3-pin which is the one I got, since it has protective casing that will be necessary in real-world deployment. So for this guide, I'll be going through hooking up the 3-pin sensor in the circuit. That being said, the wiring (which took me a lot longer to figure out then I care to admit) is a bit unintuitive.

Here's why. If you're reading just the raw value from the sensor - a value from 0 to 1023 - that's easy. If you want the analog voltage from the sensor, that's easy too. The Adafruit site above for 3-pin has a short description on how to connect it.

Red - power

Black - ground

White - analog out

But we want the resistance from the sensor, because ultimately, it will give us a clue as to the surrounding water level. The key is, you have to treat the eTape like a photocell (also a resistive sensor), using only 2 out of the 3 leads provided. We just need the "power" and "analog out" leads of the sensor, connected to the MCP and with a pull-up resistor.

This also uses a voltage divider, which is usually used for lowering the input voltage to an appropriate amount. Say the microcontroller or pi uses 5V and the sensor you hooked up only needs 3.3V - you would use a voltage divider, which usually includes two resistors, to lower that voltage to a safe level for the sensor/circuit.

All About Circuits has a great tutorial explaining these and also giving different applications for how these dividers can be used, so check out their website for more info: https://www.allaboutcircuits.com/tools/voltage-div...

You should also test the eTape with a multimeter: plug in the red and black meter probes and turn the dial to resistance. Touch one probe (doesn't matter which one) to the red wire of the tape, and the other probe to the white wire of the tape. You should get a value around 2200 Ω or so (assuming your sensor isn't submerged in water).

Pictured above is a diagram of how the circuit is laid out. If you understand how a voltage divider works, then here's how it's being used: our eTape resistive sensor will be taking the place of one of the resistors in the circuit. So, the power (red wire) from the eTape is connected to ground, and the analog out (white wire) connects to CH0 of the MCP. Add a ~600 Ω resistor between the eTape's analog out and CH0 of the MCP and hook that up to power. Picture of the circuit on my breadboard is also above!

Here's a great tutorial on Analog inputs for pis by Ralph S Bacon, in which he goes through standard python code for printing out values with the MCP chip.

Watch this for reference, but I will walk through my version of the eTape-specific code in the next step!

Step 4: ETape: Printing Out Values & Data Logging

Printing Out Values

Here's the python code in all its glory!

from gpiozero import MCP3008
import time
resistor_value = 670
# I used a 670 Ω resistor, which is the reason for this
# Anything around 600 Ω (but above 590 Ω) should be ok
while True:
    adc = MCP3008(channel=0, device=0)
    #print('{:0f}'.format(adc.value * 3.3))
    reading = float(adc.value * 1023)
    voltage = reading / 1023 * 3.3
    print("Raw Value: ", '{:0f}'.format(reading))
    print("Voltage: ", voltage)
    resistance = (voltage * resistor_value) / (3.3 - voltage)
    print("Resistance: ", '{:0f}'.format(resistance))
    print("\n")
    time.sleep(2)

The code (along with some extraneous comments) is also attached below labeled "eTape_read.py".

After attaching your tape sensor to the circuit and running this code, hopefully you should see the raw output, voltage, and resistance values on the screen of the terminal!

Data Logging

Another important aspect (possibly the most important) of this project is to be able to log these values that we've now outputted into a file that we can later view. In this case, I'll be logging the date, time, voltage, and resistance into a csv file around every 3 seconds when the code is running.

The code for this is attached below as well, labeled "eTape_monitor_2.py". Run the updated code (which is printing values into a csv file called "monitor_log.csv"), being sure to use python3 rather than python (again, this goes for all scripts). Let it run for a minute, and stop the script with ctrl-C. Then open your data log file (depends on what you call it, but let's just say it's called "data_log") with:

sudo nano data_log.csv

You should see a list of values for the date, time, voltage, and resistance! The data logs I've attached below include values from when I did various tests with the eTape submerging it in water.

Now, does all this require a WiFi connection to work? Yes and No. To actually run these scripts, you don't. All it's doing is reading the value from the eTape sensor and recording those values into a file, which all happens within the circuit and pi. However, to make sure you get the correct time and date, you would indeed have to be connected to WiFi. As explained in the intro, the Raspberry Pi automatically collects time and date information when it's connected to the internet through a protocol called NTP. Without internet/WiFi, your eTape readings will be correct, but the time and date may be inaccurate.

Let's fix that!

Attachments

Step 5: Attaching the Real-Time Clock (RTC) Module

The RTC will allow us to get the correct time for our system! I've already explained this in the "Intro" how it works, so look up there for more info.

We only need to connect 4 pins from the RTC to the Cobbler: Vin, GND, SCL, and SDA.

Vin is power, GND is ground, and SCL and SDA are the clock and data pins, respectively, that help with information transfer between the RTC and Raspberry Pi.

There are a lot of RTC tutorials out there online, but I found Adafruit's to be the most comprehensive: https://learn.adafruit.com/adding-a-real-time-clock-to-raspberry-pi/overview

In general, the wiring for this isn't too difficult:

Vin -> 3.3 V

GND -> GND

SCL -> SCL (GPIO 3)

SDA -> SDA (GPIO 2)

SCL on the RTC goes to the clock pin on the pi, which should be GPIO pin 3. SDA on the RTC goes to the data pin on the pi, which is GPIO pin 2.

Take a look through Adafruit's tutorial (props to them), which explains how to set this up much better than I can. If done right, ID #68 should pop up in the table on your terminal screen, which is the address of the RTC. There's some talk of a kernel or something like that and a "UU" replacing the 68 - I never got the UU (it's always been 68) and it seems to be working fine.

After disabling the fake hwclock, make sure to sync the Pi time with your RTC. This is also in Adafruit's tutorial, but I'm repeating it here because it's so important. The purpose of this is to ensure that the RTC has the correct time, which will always be correct even if the pi loses power since the clock itself has its own battery. First, use

sudo hwclock -D -r

to read the time from the RTC. It may be correct or incorrect - either way, let's sync it to the pi's time, which you can check by running

date

Make sure this is the right date and time! At this point, your pi should be connected to WiFi so it can get the correct time. Type

sudo hwclock -w

to write the pi's date/time to the RTC and then 

sudo hwclock -r

to read that time and make sure it's right. Insert the button cell battery (it's ok if it was already in there the whole time) to save the time.

The first time I did this, it didn't work. The pi time was already incorrect, so I'm pretty sure I was just writing the incorrect time to the RTC. This might be because the Pi isn't getting the date/time through NTP, even if connected to WiFi/Ethernet. Run

sudo apt-get install ntp

to make sure it does indeed have the ability to do this. After doing this, I redid the steps above and was able to get the correct time!

Step 6: Thingspeak and Twitter Bot

Thingspeak

All the hardware is done, yay! Now onto something just as fun, the actual IoT part.

IoT stands for the Internet-of-Things, and basically refers to "smart", interconnected devices. What makes IoT so interesting is its ability to seamlessly link separate entities together into a system that can be used to affect change that ultimately can help the world in some way.

When I say we're getting to the "actual IoT part," that's not entirely true. What we're really doing is looking at the software side of it, since IoT really does focus on both hardware and software.

In this case, we'll be using a third-party service called Thingspeak. Time for another tutorial, this time from the amazing Matt over at the raspberrypi-spy website!

https://www.raspberrypi-spy.co.uk/2015/06/basic-te...

You can think of Thingspeak as an online "data logger." Basically, it's doing the same thing that we did in previous steps, where we recorded eTape values into a csv file, only now we're doing it over the Internet. There are a couple benefits to this:

  • Data visualization! Hooking up your system to a service like Thingspeak allows you to visualize the data you get. In this case, the service will automatically plot (provided you've included the python code in the pi, which we'll get to soon) the resistance vs. the time. This way, you can see how the resistance is changing over time in a clear, visual way.
  • Testing! If the project you're working on has any scientific/engineering basis, you'll undoubtedly be testing your sensors to see how they work. In this case, testing the eTape included measuring the resistance at each level (0 in, 1 in, 2 in... 12 in) for 5 minutes, for a total of 60 minutes. With live data graphing, this is super easy. Just hook up your sensor to Thingspeak, and all you have to do is pour water while it handles all the heavy lifting
  • Interconnection! Thingspeak, specifically (not sure about other services) allows us to link the channel through which we're collecting this data to other services. This could include Twitter, HTTP functions, and more. Having this feature really expands the IoT capabilities of this project, allowing multiple devices/services to "talk" to one another.

I've included a few pictures of my channel on this platform, which includes a data visualization and counter that I had open during testing.

I've attached the code below that will allow you to interface your system with Thingspeak, called "eTape_thingspeak.py."

Twitter Bot

Over the course of this project, I had spent a lot of time in my room just fiddling around with circuits or writing code on my laptop. Well, one day over the weekend, I went down to the Gowanus and spoke to some of the community members to get their insight on the flooding problem. I won't mention too much here, as some wanted their information to be anonymous, but I was able to gain a lot of insight from what they told me. I settled on creating a Twitter Bot with a dual purpose:

  1. To notify us in the event of a flood. After we've deployed the device on the street, if it starts to flood, the system will send out a tweet with a message that includes the estimated flood depth.
  2. As a means of interaction for community residents. The device itself will have QR codes, one linking to this instructable, and one to the twitter account. Residents can scan the code, follow this account, and get live updates on the situation in the Gowanus.

There are many, many ways to implement a Twitter Bot. Here's a tutorial from pimylifeup that you might follow, in which you create an app on Twitter and use Twython, a python package, to communicate with Twitter's API.

https://pimylifeup.com/raspberry-pi-twitter-bot/

This is great, but may not be accessible to everyone. Recently, Twitter has strengthened its privacy and other policies, which makes it harder for the average creator to make an app. While previously, anyone was able to create an app on Twitter, you now have to register for a developer account to be able to do this. This is a bit of a lengthy process (mentally, at least), and you need to have a good reason for why you want to register as a developer. You need to tell Twitter what kind of data you might want to collect and how you'll be using it. After submitted your dev application, they'll email you saying if you were accepted.

Long story short, I did not follow the above tutorial simply because I found an easier solution in my current situation. Thingspeak actually has a built-in feature that allows your devices to talk to Twitter! I can't find the exact tutorial I followed for this - though I'm sure there are many you can find online - but it should be fairly intuitive. It mainly involves creating a channel and then using the ThingTweet App to tweet out values based on your channel output. No extra python code is necessary for this step; everything happens within Thingspeak and Twitter.

However, I did attach a file called "flood_monitor_2.py" below that puts together all the code in a neat, organized way that includes printing out values, data logging, and data transfer to Thingspeak.

Here's the official twitter account of the Gowanus Flood Monitor, which you can check out: https://twitter.com/FloodGowanus

I've included a picture of the account tweeting out info based on data from Thingspeak.

Attachments

Step 7: Preparing for Deployment

Since my project actually involves deploying this system in the real world, my project advisors and I thought that it would be a good idea to minimize everything, to an extent. We picked up a Raspberry Pi Zero W (much smaller and still has WiFi) and basically transferred the circuit onto there. I soldered everything onto a "hat" for the pi and put it into a small, protective casing. I then moved the sd card from my Pi 3 to the Pi Zero W and tested it to make sure it was working.

I then created an enclosure for the sensor out of PVC, which required going to Home Depot, buying the materials, and getting them cut and drilled to size. Assembly pictures above, as well as a picture of the final product in a mock deployment in the Gowanus!

Step 8: Moving Forward

Lastly, I'd like to end with thinking about future plans for this project.

  • As of right now, while the device is working, we have not quite figured out the battery issue. Any 5V battery pack should be capable of powering the Pi, but not for very long. It would be difficult to find a power outlet out on the street and even if we did, there's the whole problem of this being a flood depth sensor.
  • Currently, the system is still spitting out resistance values, and Thingspeak is doing the "calculation" as to what the flood depth is. What would be more efficient is to write that calculation in the python code itself and have the sensor give depth rather than resistance values. This would likely require more testing with the eTape, then using Excel to figure out the average/standard deviation and create a linear regression to find the relationship between depth and resistance. Then plug that in in the python code and use a dictionary to get the correct values.
  • It would be great to tack on a GPS of some sort, especially since ideally we'd want to have more than one of these sensors deployed, and be able to get and record the location of each sensor.
  • Experiment with different enclosures. We could also try a version where we put the circuit + weatherproof casing within the PVC pipe itself, which would offer more protection, but make it less feasible to take out the circuit if we needed to.

As of December 2019 while I'm writing this, all the above tutorials listed should work, but please comment below if there are any broken links/out of date tutorials and with any questions you have!