Friday, February 10, 2017

Install Rails 5 on Elastic Beanstalk

My posts are mainly notes-to-self.  I try to add in supporting links to help anyone who can benefit from them.

Heroku has been my go to method for deploying Ruby on Rails apps.  However, I find that I want to create inexpensive web apps that do not go to sleep as the Heroku free apps do.  I also do not want to rack up multiple apps with a minimum $7 per month hosting fee for each idea I want to explore.

I have decided to try out Rails 5 on Amazon Web Services using Elastic Beanstalk for my next project to keep costs down and have apps that load immediately.

The project:
I have a Raspberry Pi earthworm bin temperature monitor served on my local network.  The Pi grabs temperatures from soil and air sensors, saves them to a text file, then the web site displays them on a graph with the help of Node.js.  The Node based site is slow and error prone.  I did not build the web interface, but rather grabbed it from an old tutorial on Raspberry Pi.  Since I am more familiar with Rails than Node and I want my site to be public, the idea I have is to write the temperature values to a cloud Postgres database, then graph them with Rails 5, and a graphing gem like Chartkick.

Install Elastic Beanstalk CLI 

I started by following these steps:

My specific steps:

Since I am on Ubuntu v 16.04, I cd to my projects folder to get started.
$ cd ~/projects

Check my python version.
$ python --version => Python 2.7.12

Install EB CLI
$ pip install --upgrade --user awsebcli

Figure out your shell then edit your PATH. (see the aws link above for detailed explanation) Since I use Bash I edited like:
$ nano ~/.profile

but I found that 
export PATH=~/.local/bin:$PATH
was already there so I exited out without changing.

Check my EB version
$ eb --version
=> EB CLI 3.9.0 (Python 2.7.1)

Create Rails 5 Instance

I am following along with this Rails and Beanstalk how-to

I already have ruby and rails installed so I go ahead and create a new Rails project.  If you want to install ruby and rails check this out: https://gorailrails new

I check my Rails version:
$ rails -v
=> Rails 5.0.1

Create Rails project
$ rails new wormbin
$ cd wormbin

Initialize git:
$ git init && git add -A && git commit -m "new rails 5 setup for elastic beanstalk"

First Model

My basic first need is to log the time and temperature.  I already have many months of data written to csv file on my pi.  I will need to begin writing the new values to the database we are creating now with our Elastic Beanstalk implementation.

$ rails g scaffold outside_temp date:datetime temperature:decimal$ rails db:migrate
$ git add .
$ git commit -am "Add outside temp"
$ git push

Note: to use git push you need someplace to push to.  I use github.   Check this out  

I create a remote repository on github.  Then with the url I create remote git:

$ git remote add origin $ push -u origin master

You can check your remotes with 
$ git remote

Initialize ElasticBeanstalk

$ eb init 

follow the choices shown on above tutorial or choose your own.  I went with ruby 2.3 and puma along with region = US-EAST-1

$ git commit -am "Post EB init"
$ eb create production

this part looks like it will take a while... about 5 minutes.  once done I can

$ eb status

If you head to url provided by status you will get error because of missing rails secret, so add it

$ eb setenv SECRET_KEY_BASE=$(rails secret)

to update 
$ eb deploy

I headed over to the public url and it gave me page not found. I did public-url/outside_temps and everything worked fine.  (I'll need to review muli word model name convention to avoid having underscore in url in the future)

Ok- I need to fix the above mistake with underscore before I go any further.  To do that
$ rake db:rollback
$ rails d scaffold outside_temp date:datetime temperature:decimal

$ rails g scaffold OutsideTemperature date:datetime temperature:decimal
$ rake db:migrate
$ git add .
$ git commit -m "change outside temperatur model name"
$ git push
$ eb deploy

hmm- that worked and it didn't.  I still have underscores.  I will come back to that later.

Setup Postgres Database

gem 'pg' #add to Gemfile
$ bundle install

edit config/database.yaml 
###noticealbe gap in info here###
I originally added a Beanstalk instance with automatic RDS created in it.  If you follow along with the above tutorial to setup Beanstalk it has you add the RDS database in Beanstalk instance.  You should not add rds instance within Beanstalk, but rather add it separately.  The security setup when doing it manually is the biggest hassle.  This explains how to setup:

I eventually worked out how to link my external RDS instance with Beanstalk but my appoach was not documented.  Sorry.  It has to do with security groups and making sure the RDS instance and the Beanstalk instance are in the same group.  I also added my local machine's ip to the firewall on AWS to allow me to manually connect to SQL with a Postgres a client to look at data independent of the web site.

In a future post I will demonstrate the python involved in writing to the database.

Friday, December 2, 2016

Facebook Authentication in Ruby on Rails

Goal:  Make my site easier for vistors by allowing them to register or login (existing Devise account) with Facebook.

I have Rails 4.2, Devise 4.2

I mostly follow the this Devise how-to for setting up OmniAuth:

Exceptions to above how-to:

in devise.rb initializer I added

 # added this for facebook authentication
  if Rails.env == "production"
    fb_callback = "" 
    fb_callback = "http://localhost:3000/users/auth/facebook/callback" 
  config.omniauth :facebook, ENV['AD_FB_APP_ID'], ENV['AD_FB_APP_SECRET'],
                callback_url: fb_callback ,
                info_fields: 'name,email'
The environment variables are
AD_FB_APP_ID and AD_FB_APP_SECRET I set these in my local machine's bashrc file (ubuntu linux installed) to match the values in my newly created facebook app.

my projects are located under /home/sean/projects, so when I am in a specific project directory I edit bashrc like:

nano ../../.bashrc

re-read bashrc after saving:  source ../../.bashrc

Facebook App Dashboard

Facebook settings are tricky. Within

Add Facebook Login product

Then Set 'Valid OAuth redirect URIs' to:

http://localhost:3000 and

Under Settings: Add 'website' platform, then set app domains to

Under App Review: make live

User Methods

After following along with the rest of the devise tutorial above, I came up with the following two methods for my user model:
  def self.from_omniauth(auth)
    user = User.find_by(email:
    if user and user.confirmed? 
      user.provider = auth.provider
      user.uid = auth.uid
      return user
    #where(auth.slice(:provider, :uid)).first_or_create do |user|
    where(provider: auth.provider, uid: auth.uid).first_or_create do |user|
      user.provider = auth.provider
      user.uid = auth.uid =
      user.password = Devise.friendly_token[0,20]
      user.first_name =" ")[0..-2].join(" ")
      user.last_name =" ").last
      #user.first_name = first_name
      #user.last_name = last_name

  def self.new_with_session(params, session)
    super.tap do |user|
      if data = session["devise.facebook_data"] && session["devise.facebook_data"]["extra"]["raw_info"] = data["email"] if
        user.first_name = data["name"].split(" ")[0..-2].join(" ") if user.first_name.blank?
        user.last_name = data["name"].split(" ").last if user.last_name.blank?


I am using Heroku so I need to remember to set the Facebook app key and secret variables in my Heroku settings as well


I realized I need users to finish filling out their registration form to get info that facebook does not provide such as phone.  However, since the password was auto generated for them on registration with facebook, I need a way to get the additional info and not bother them for a password in the process (which they do not know).

First I redirect the new signup to the registration page:
I edit my omniauths_callbacks_controller.rb

class Users::OmniauthCallbacksController < Devise::OmniauthCallbacksController
  def facebook
    # You need to implement the method below in your model (e.g. app/models/user.rb)
    @user = User.from_omniauth(request.env["omniauth.auth"])
    if @user.persisted?
      sign_in @user

      # there is probably a better way to do this but I need to get new accounts to finish registration
      if @user.created_at > - 1.minute 
        redirect_to edit_user_registration_url 
        flash[:alert] = "Please set phone.  Choose texting preference if desired."
        redirect_to root_url


      #sign_in_and_redirect @user, :event => :authentication #this will throw if @user is not activated
      set_flash_message(:notice, :success, :kind => "Facebook") if is_navigational_format?
       flash[:alert] = "Please set phone.  Choose texting preference if desired."
      session["devise.facebook_data"] = request.env["omniauth.auth"]
      redirect_to new_user_registration_url

  def failure
    redirect_to root_path

Then, I add a new controller under controllers/users/registrations_controller.rb:

class Users::RegistrationsController < Devise::RegistrationsController
  # allow facebook registrations to edit account without password

  def update_resource(resource, params)

  # Overwrite update_resource to let users to update their user without giving their password
  def update_resource(resource, params)
    if resource.provider == "facebook"

I add a route for it in routes.rb like so:

devise_for :users, :controllers => { :omniauth_callbacks => "users/omniauth_callbacks", :registrations => 'users/registrations' }

Lastly, I hide the password fields on my views/devise/registrations/edit.html.rb form by wrapping all three of the password fields with

<% if (!current_user.provider=='facebook') %>

<% end %>

Thursday, August 4, 2016

Live Streaming to Youtube with Raspberry Pi 3

After a bit of research on the best way to stream from the Raspberry Pi 3 with camera sensor version 2, I came across this tutorial:

So I followed this until I hit issues.

When compiling ffmpeg for Raspberry Pi 3 I want to use all four cores of processor.

Where these instructions ( ) say:

sudo make
sudo make install

I replaced it with:
sudo make -j4 install
Testing the install of ffmpeg:

gives this error:
/usr/local/bin/ffmpeg: cannot execute binary file: Exec format error

I noticed that my full version of jessie must have had ffmpeg in it already so I removed it but it did not help:

I did not note the exact way I removed it but it was something like:
sudo apt-get remove ffmpeg

I removed the ffmpeg directory I had previously cloned ( /usr/src/ffmpeg) and cloned from another source:

Then rebuilt with no options to see what happened:
sudo ./configure
sudo make -j4 install

then ran stream on command line and it worked!
sudo raspivid -o - -t 0 -w 1280 -h 720 -fps 25 -b 4000000 -g 50 | ffmpeg -re -ar 44100 -ac 2 -acodec pcm_s16le -f s16le -ac 2 -i /dev/zero -f h264 -i - -vcodec copy -acodec aac -ab 128k -g 50 -strict experimental -f flv rtmp://

But while steaming I did get some warnings and errors to investigate further:

Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.

[flv @ 0x16dba70] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[h264 @ 0x168e1f0] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
^Cmmal: Aborting program.0 size= 56962kB time=00:04:02.11 bitrate=1927.3kbits/s speed= 1x

[flv @ 0x16dba70] Failed to update header with correct duration.
[flv @ 0x16dba70] Failed to update header with correct filesize.

Youtube showed my live stream with a starting delay and a status of 'healthy' in green.
Raspberry Pi v2  with alternative pi-compatible camera module .  Camera sensor is v1 based 5 megapixel with adjustable focus and near fish eye - around 160 degrees angle of view

Next I can now investigate the errors and consider recompiling my ffmpeg with some of the options if needed.

After some time playing with video, I resolved jerky capturing and added text display with this new run string

 sudo raspivid -o - -t 0 -w 1280 -h 720 -fps 30 -ex sports -b 4000000 -g 50 -a 12 -ae 10,0x00,0x8080ff -a "My Camera %m-%d-%y @ %I:%M %P" | /usr/src/ffmpeg/ffmpeg -re -ar 44100 -ac 2 -acodec pcm_s16le -f s16le -ac 2 -i /dev/zero -framerate 30 -f h264 -i - -vcodec copy -acodec aac -ab 128k -g 60 -strict experimental -f flv rtmp://

Friday, February 26, 2016

Worm Bin Moisture Sensor with Arduino

Arduino connected to Rapsberry Pi with USB
Arduino Uno with analog moisture sensor and usb connected to Pi

I have my worm bin temperature sensors working very well with my Raspberry Pi.  I have now turned my attention to measuring soil moisture.

Moisture Sensing Choices:

  • Which soil moisture sensor should I use?
  • Which micro controller do I employ?
  • How many sensors are needed for practical use?

After a lot of searching for sensors, it seems that moisture sensing is not nearly as straightforward  as reading air or soil temperatures.  There are different ways of reading moisture.

I honestly do not know the ideal way to read moisture. It seems that measuring the electrical conductivity of the soil to look for moisture is a popular method so I will go with that.

Sensor Corrosion Problem

One significant problem with sensors using electrical resistance is that they do not last very long because they corrode due to the electricity and metal contacting wet soil.  To help minimize this issue, you can get them gold plated and only power the sensor before you take your reading and then turn it off to maximize sensor contact life.

Note: I am a noob regarding electrical things but it looks like I might be able to control power to the sensor with a transistor (It acts as an electronic switch).

I opted to go with a quick-and-dirty approach first.  The Raspberry Pi has no analog pins to read and write to.  Moisture sensors require analog connections to the microcontroller to work.  After all, 0 and 1 tell you wet or dry but no values in between.

The sensor I bought first is

I happen to have an Arduino Uno that I played with a few years back, so its time to dust it off  and put to work.  Actually, my board is revision 3 which looks current.  

I proceed to install the Arduino ide and make sure it works with a few simple LED light blinking sketches.  All good so far.

My moisture sensor arrives from Amazon and I realize it has absolutely no documentation so I start googling what is printed on it "ywRobot mositure sensor" and come across some schematics and this helpful video... (it's actually wRobot I think)

I follow along with the video and within a couple of minutes I have it reading values 0.  I drop it into a shot glass of water and the reading changes to around 600 give or take.

Setup Raspberry Pi to Log Serial Data 

(That is send moisture readings via USB from your Arduino sketch to Pi)

Make sure Python is installed on Pi.  

You can attempt to install or check version.
sudo apt-get install python 
or check python version:
python -V

Install moreutils... 

to get TS (time stamp) package.  moreutils is a basic linux grouping of hand features not included in standing linux distribution.

sudo apt-get install moreutils

Install grabserial

grabserial is a utility to read serial data from serial port of remote linux microcontroller and write to standard out (like a text file for later use).


Test serial (USB) connection


I see all the usb connected devices including my arduino:
Bus 001 Device 004: ID 2341:0043 Arduino SA Uno R3 (CDC ACM)

Get device id
This is a little tricky.  I used grep to get keyword related results since my results spanned multiple pages:

dmesg | grep tty
I looked through results and found my id of TTYACM0
You can just page through results with:
dmesg | more

Incidentally, the /dev directory is a listing of hardware devices and those prepended with 'tty' are serial devices.

NOTE: the symbol | is typed by using shift : on your keyboard.  It's called a pipe command.

Configure grabserial to use the correct serial port

use the text editor nano (You can us vi and others but vi has a steeper learning curve.):  
nano grabserial

within the file use the ctrl-w command to jump right to the lines to configure.  Search for 'sd.port' (which takes your down about 7 pages).

Change your setting to match your serial port id found in dmesg:


You are setting the baud rate to match what your arduino serial port was set to in the sketch you installed.

Within grabserial file save with CTRL-x to exit and Y to save changes.

Check that grabserial is working:

python grabserial

I get output:
  , 0
because my sensor is not in water so all works and I CTRL-c to stop.

to get the timestamp for each reading

python grabserial | ts
Feb 14 16:53:26  , 0

Permanently Log Moisture Readings

There are a lot of ways to process the data now.  You can save to database or save to text (csv) file.  You can then use a web page to display, chart, or export to your computer to use in excel or your favorite charting program.

To write the data to a csv file for use later with use:
python grabserial | ts  >  moisture.csv 

To add a new data file in our sensor directory (to chart with the other sensors):
python grabserial | ts  >  /home/pi/projects/temp-and-humidity/sensor-values/moisture.csv

or even better yet, run the command above with a cron job then edit the worm bin web page to read the data (/home/pi/projects/temp-and-humidity/public/index.html)   You can check this crontab tutorial for more info on doing this.

Using Raspberry Pi Only

In this post I am using the Arduino because the Raspberry Pi does not come with analog pins and I did not have a spare Raspberry Pi lying around.  However, there is a way to use analog devices with Raspberry Pi using an analog to digital controller (ADC).  I will explore that in another post.

Number of Sensors

I have heard suggestiongs to go with 1 sensor every foot.  In this case, my 3 x 4 ft bin could use 3 sensors.  Once I settle on a platform, I will pickup 2 more sensors, figure out the switchin them on and off method, solder up some long wires to a board, connect them to one of the controllers, and coat the electronics with heat shrink tubing and silicone conformal coating to prevent moisture damage.


You can use any micro controller to read analog inputs.  It may just come down to which platform you are confortable with.  There are major differences in Raspberry Pi (a full computer operating system)  and Arduino (real time microcontroller), but simple analog readings can be done with both platforms.

Saturday, February 20, 2016

Keeping Your Raspberry Pi Connected Via WiFi

Is Your Raspberry Pi Loosing Wifi Connection? 

When my local network's access point or router restarts my Raspberry Pi does not always reconnect to the network.   This can also happen if the Pi's wifi is flaky due to distance or other issues.

To fix this I created a shell script that runs at start up based on this post.

in /home/pi add file:

#!/bin/sh -e
while true ; do
   if ifconfig wlan0 | grep -q "inet addr:" ; then
      sleep 60
      echo "Network connection down! Attempting reconnection."
      ifup --force wlan0
      sleep 10
exit 0

I run this script in /etc/rc.local
#!/bin/sh -e
/home/pi/ &
exit 0

sudo nano /etc/rc.local

Then made sure rc.local was executable:
sudo chown root /etc/rc.local
sudo chmod 755 /etc/rc.local

Make sure rc.local is executable:
sudo chmod +x /home/pi/
Check that setting change was made (look for x in permissions of file)
 ls -al /home/pi

Test if it will work in startup:
sudo /etc/init.d/rc.local start

It it works, restart your Pi:
CTRL-C to stop script if needed
sudo reboot

You can further test by restarting your access point or router.

Thursday, February 4, 2016

Worm Bin Monitor Live: Part 3

I am continuing to refine my flow-through worm bin temperature monitoring system.  In part 1 and part 2 I started prototyping what the system would do based on a couple of third party tutorials that I combined.

Worm Bin Up & Running

I am now have everything working in the real world.  To do that I needed

  • a decent wireless network signal outdoors to talk to the system from inside my home
  • more permanent electronic connections so things don't fall apart if moved
  • a bin full of worms
  • mild protection for electronic components

dust-resistant enclosure

Outdoor Wi-fi

I have an indoor wireless access point but it does not get me very far outdoors.  I needed to get good network access at least 250 feet though many large pine trees to monitor anything.  To do this I purchased and installed a wireless access point designed for outdoor use.

I mounted this AP on the outside wall of my second story.  I had my doubts it would work very well, but I was really surprised that it does.

I also boosted the reception of the Raspberry Pi at the worm bin with a USB wireless adapter with an external antenna.  The downside of this particular antenna is that it requires a powered USB hub to stay well connected.

I purchased the EnGenius ens202-ext Outdoor Wireless Access Point and found a powered usb hub on Amazon.  The connection is working great.

Stabilizing The Electronics

In the working photo above, the red circled area is the air temperature and humidity sensor.  The green area is the solder less breadboard I used in previous steps to setup the sensors temporarily.

I had to learn to solder to make it all stable.  I have never soldered so I checked out some how-tos on the subject.  This soldering Instructable got me going.  I was in a hurry so I went down to the local Radio Shack (mine is still in business but has limited supply).

I picked up some 0.35 leaded rosin core solder, a pack of cheap circuit boards, and some 22 gauge solid core wire.  I already had a 25 watt soldering iron I had used years ago for something else.

Soldering takes practice but I was able to recreate my two sensor circuits on a small board.   I removed all the wires and resistors off of the original white breadboard .  I left the breadboard in place so I can add new sensors in the future (soil moisture sensor?).  I soldered wires onto the air sensor (red circle) to extend it outside of the foam box.

My worm bin is outdoors but under cover.  I am not too worried about moisture and dirt but do want some modest protection from dust.  I opted to reuse a foam food container to house the electronics.

Incidentally, these foam boxes are a major sore spot for me because locally all the restaurants hand them out for take out and leftovers.  In California, where I am from, Styrofoam has been illegal for many years.  The southern US is way behind on environmental awareness, so it's nice to re purpose things when I can.

Adding The Worms 

Finally...the worms are added.  I ordered 10K red wigglers online.  They came plump and lively.  I added them to the bin 24 hours ago.  

10 lbs of Eisenia Foetida worms

Final Thoughts on the Benefits of Environmental Monitoring

Our recent North Carolina temperatures have been unseasonable.  In February, we are seeing highs in 70s F and lows in 20s F.  My initial monitoring is demonstrating that the insulation and warming cable is keeping the bin temperature stable.  It's currently in the 50s.  By tomorrow evening we will see temperatures in the lower 20s.  

steady soil temp so far

Since I am a relatively new worm owner with a lot invested in worms I want to react quickly to problems.  The worm bin monitoring system will tell me how far temperatures rise or fall and for how long.  I can see measurable results for my efforts to control temperature with more heat in the winter, ventilation changes, recent feedings, and the addition of ice bottles in the summer months.

Raspberry Pi Model 2

I decided to replace the Raspberry Pi B+ with a Model 2.  It's much faster loading the page now.  However, there was a problem with the DHT (soil) sensor not working with the Adafruit library as I had installed it.

I made sure this was installed:
sudo apt-get install build-essential python-dev

Then reinstalled the Adafruit library to make sure.
cd /home/pi/sources/Adafruit_Python_DHT
sudo python install 

Now all is working correctly.

Sunday, January 24, 2016

Automated Worm Bin Progress: Part 2

See Part 1 of my notes for the first steps.

I have nearly completed building my worm bin.  The temperatures have been too cold to get glue to dry to finish some details. Meanwhile, I will work on the technical portion of the worm bin's temperature monitoring.

nearly complete worm bin with heater rack in foreground

I have the soil temperature part working well.  The soil sensor is collecting the temperature and displaying it on the graph on the bin's home page.

The soil temp line is brown on the graph but shows as grey with the other color settings being a little transparent.  You can see on the far left, I had the probe on an ice pack to help me see that it was there (otherwise it would be almost the same temp at the air).  Near the very end (right), there is a blip up where I held it in my hand for a couple of seconds.

I had to edit a few of the step 1 and step 2 files to get the second sensor working with the others.

I modified the step 1 python script ( to include some of python code from step 2:

You will need to edit the line:
soil_temp_sensor = '/sys/bus/w1/devices/28-000006c3a429/w1_slave'
To match your sensor's id see the step 2 tutorial link.

I also edited the public/index.html file to handle the new sensor info:

Be sure to restart everything (the web server or Raspberry Pi) to make sure it all works.

Now would be a good time to make a backup disk image if you are following along.

Measuring Soil Moisture

I now would like to measure the soil moisture.  This seems to be a bit trickier to find a sensor that will work with Raspberry Pi and Python.  There are some tutorials on Arduino moisture sensor ideas.  This project uses both Aruino and Raspberry Pi for monitoring crops.

So far, this combined temp and soil moisture sensor might work (a bit pricey):
Seems to work with Arduino but not sure about a Python.  I did find python library for it which might work :

There are also some projects that show other options that look interesting:

This Instructable looks really interesting because it gets into many more elements beyond just monitoring and looks at reacting to conditions.

I have a few other related projects to work on.  I'll take a break on the technical part of my worm project for now and think about the moisture sensing a little more.   I need to finish my worm bin construction, put some worms in it, and install on outdoor wireless access  point to connect to the the Raspberry Pi from the comfort of my house (200 feet away).

I hope what I have done so far gets you started.  If you have any improvements or other ideas please let me know.