Post on 13-Jan-2017
@girlie_mac
KittyCam.jsSmile, you’re on camera!Smile, you’re on camera!
Tomomi Imura
@girlie_mac
@girlie_mac
Tomomi (@girlie_mac) is a(n):
● San Francisco Dweller● Front-End Engineer● Open Web + Tech Advocate● N00b Hardware Hacker● Sr. Developer Advocate at Nexmo● Cat lady of InterWeb
@girlie_mac
HTTP Status Cats at https://http.cat
Thanks Rogério Vicente for the .cat domain & the API!
@girlie_mac
@girlie_mac CC-BY-SA 3.0
https://commons.wikimedia.org/wiki/File:Basic_robot.jpg
@girlie_mac
Arduino● MCU-based kit● Open-Source hardware &
software (IDE)● The first developer-friendly
boards
@girlie_mac
Sketch● Language for Arduino● Loosely based on C● Wut, C?● HALP!!1!!!
#include <Servo.h>
#include <Wire.h>
#include <Firmata.h>
#define I2C_WRITE B00000000
#define I2C_READ B00001000
#define I2C_READ_CONTINUOUSLY B00010000
#define I2C_STOP_READING B00011000
#define I2C_READ_WRITE_MODE_MASK B00011000
#define I2C_10BIT_ADDRESS_MODE_MASK B00100000
#define I2C_MAX_QUERIES 8
#define I2C_REGISTER_NOT_SPECIFIED -1
#define MINIMUM_SAMPLING_INTERVAL 1
int analogInputsToReport = 0;
byte previousPINs[TOTAL_PORTS];
byte pinConfig[TOTAL_PINS];
byte portConfigInputs[TOTAL_PORTS];
int pinState[TOTAL_PINS];
unsigned long currentMillis;
unsigned long previousMillis;
unsigned int samplingInterval = 19;
@girlie_mac
Raspberry Pi● $35 single-board computer ● Broadcom chip with
ARM-compatible CPU & GPU● Runs Linux● More language choices: C,
C++, Python...
@girlie_mac
But… I want to code in
@girlie_mac
Johnny-Five● JavaScript robotics framework● Works with
Arduino-compatible Boards● IO plugins for more platform
supports● http://johnny-five.io/
@girlie_mac
Awww, JavaScript all the way!
@girlie_mac
Hello world!
@girlie_mac
KittyCamRaspberry Pi camera with cat facial detection!
● Hardware: Raspberry Pi, camera, and PIR sensor
● Software: Node.js + J5 + More open-source goodies
@girlie_mac
Motion detected!
snap!
Mmm… donut
@girlie_mac
@girlie_mac
@girlie_mac
1. Raspberry Pi 3
2. 5MP Camera Board Module
3. Pyroelectric Infrared (PIR) motion sensor
4. Female/Female wires
@girlie_mac
Programming Raspberry PiPre-installed on Raspbian OS:
C / C++
@girlie_mac
Programming RPi with Node.js
@girlie_mac
@girlie_mac
Installing Node ARM$ wget
https://nodejs.org/dist/v4.4.5/node-v4.4.5-l
inux-armv7l.tar.xz
$ tar -xvf node-v4.4.5-linux-armv7l.tar.xz
$ cd node-v4.4.5-linux-armv7l
$ sudo cp -R * /usr/local/
@girlie_mac
kittyCam.js1. Detect motion 2. Take a photo3. Cat facial detection 4. Store the photo in cloud5. Real-time view on web6. SMS the photo link
HELL,YEAH!
@girlie_mac
kittyCam.js1. Detect motion w/ Johnny-Five IR.Motion obj2. Take a photo w/ Raspistill, command line tool3. Cat facial detection w/ KittyDar4. Store the photo in Cloudinary5. Publish & subscribe the url to display on web
via PubNub 6. Send a text message via Nexmo
@girlie_mac
$ Raspistill
canvas
catDetect.js
app.js
kittyDar
Motion detected
take a photo
Store the photo if
cats are detected
display photos on web browsers real-time anywhere
Johnny-Five w/ raspi-io
child process
Returns url
Notify with SMS via
@girlie_mac
Johnny-Five w/ Raspi-ioconst five = require('johnny-five');
const raspi = require('raspi-io');
let board = new five.Board({io: new raspi()});
board.on('ready', () => {
console.log('board is ready');
...
});
@girlie_mac
Motionconst five = require('johnny-five');
const raspi = require('raspi-io');
const board = new five.Board({io: new raspi()});
board.on('ready', function() {
// Create a new `motion` hardware instance
const motion = new five.Motion('P1-7');
...
});
a PIR is wired on pin 7 (GPIO 4)
VCC
Ground
Data
@girlie_mac
Raspistill Command Line Tool
$ raspistill -o myPhoto.jpg
@girlie_mac
PIR Sensor > Run Camera const child_process = require('child_process');
board.on('ready', () => {
const motion = new five.Motion('P1-7');
motion.on('motionstart', () => { // Motion detected
let filename = 'photo/image_'+i+'.jpg';
let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1'];
let spawn = child_process.spawn('raspistill', args);
spawn.on('exit', function() {
console.log('A photo is saved as '+filename);
...
motion detected!Take a photo!
Spawns a new process w/ a given shell command
@girlie_mac
Processing Photospawn.on('exit', () => {
let imgPath = __dirname + '/' + filename;
// Child process: read the file and detect cats with KittyDar
let args = [imgPath];
let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js');
fork.send(args);
// the child process is completed
fork.on('message', (base64) => {
if(base64) {
uploadToCloud(base64); // Send to cloud storage
}
}); ...
Create another worker by running a new instance of V8 engine.
@girlie_mac
detectCatsFromPhoto.jsconst kittydar = require('kittydar');
const Canvas = require('canvas');
process.on('message', (m) => {
fs.readFile(m[0], (err, data) => {
...
let canvas = new Canvas(w, h);
let ctx = canvas.getContext('2d');
ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h);
let cats = kittydar.detectCats(canvas);
console.log('There are', cats.length, 'cats in this photo');
if(cats.length > 0) {base64Img = canvas.toDataURL();}
process.send(base64Img);
process.exit(0);
});
Running in an another process
a cat detected!
KittyDar
@girlie_mac
KittyDar● Open-source JavaScript cat facial
detection written by Heather Arthur● Takes a canvas obj & calculate the
locations of cats in the image
let cats = kittydar.detectCats(canvas);
@girlie_mac
KittyDar: Behind the Scene1. Chops up the image up into many “windows”2. Extracts data by measuring a set of gradients,
from light & dark in order to find edges3. Compares the direction of these edges to the
edges found in known cat images
Neural network (JSON w/ vector data) is pre-trained w/ thousands pics of cats & non-cats
@girlie_mac
Cat Facial Detection
http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf
@girlie_mac
Cat Facial Detection
http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf
@girlie_mac
Cat Facial DetectionThe annotation data sequence:(Number of points, default is 9), (Left Eye), (Right Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3), (Right Ear-1), (Right Ear-2), and (Right Ear-3)
@girlie_mac
Cat Facial Data
9 247 94 294 92 273 127 213 81 207 29 247 53 286 50 320 20 322 74
00000561_012.jpg.cat00000561_012.jpg
@girlie_mac
Cat Facial Data Collection
@girlie_mac
Neural Network
Positive
Negative
JSON data
hog-descriptor
kittydar.jsCompare data
@girlie_mac
Send the Pic w/ Kitty to Cloudconst cloudinary = require('cloudinary');
// the child process is completed
fork.on('message', (base64) => {
if(base64) {
cloudinary.uploader.upload(base64, (result) => {
// Done! - Get the URL and do more stuff
});
} else deletePhoto(imgPath);
});
@girlie_mac
View Photos Real-time via Socket
const pubnub = require(pubnub);
Publish Subscribe
@girlie_mac
http://www.girliemac.com/RPi-KittyCam/
@girlie_mac
Send SMS via Nexmoconst Nexmo = require(nexmo);
let nexmo = new Nexmo({//config with API keys});
nexmo.message.sendSms(
FROM_NUMBER, TO_NUMBER, ' '+ url, options,
(err, responseData) => {
if (err) console.log(err);
else console.dir(responseData);
});
@girlie_mac
@girlie_mac
QA TeamLead QA:Jamie
Ginger Basil Alice YugiVenom@kittenVenom
@girlie_mac
github.com/girliemac/RPi-KittyCam
@girlie_mac
Next Steps● Upgrade Hardware
○ Raspberry Pi 3○ NoIR Night Vision Camera
● Upgrade Node (was 0.12) & all dependencies● More features (Maybe)
○ Cat Identification w/ RFID○ Photo Booth w/ Filter effects & props
@girlie_mac
Next Project?● Selfie bot
(à la Mannie the Selfie Cat)
Mannie the Selfie Cat by @yoremahm on Instagram
https://www.instagram.com/yoremahm/
@girlie_mac
Tack så mycket!@girlie_macgithub.com/girliemac