Autonomous/Unmanned Vehicles and Robots in NH: Dual picamera setup for driveless car data collection

Autonomous vehicle development of all sorts, from driverless cars to unmanned vehicles (UAVs, UGVs, UUVs, etc) requires a robust set of data and algorithms to truly operate autonomously. The purpose of this blog is to address some of those needs. I will be writing about some of my work in AV development.

First up is a discussion about a camera system I've built for data collection within my vehicle. It consists of a pair of Raspberry Pi PiCameras along with a PC with ROS/OpenCV installed (all connected through ethernet), a serially connected GPS receiver (LS20031) and networking/power devices as needed. These PiCameras are low cost HD cameras with an evolving Python interface and an active worldwide development group. I have not yet added a Triple-Axis Magnetometer (HMC5843) I have sitting on my bench to the I2C bus, that is something I wish to do in the very near future. That would allow the capture of an additional heading reading.

The code base I've created to date for communicating between the PiCameras and the PC is located at:
Server (PC with storage): https://github.com/mdsousa/video_read
Client (PiCameras): https://github.com/mdsousa/pi_video
As the project progresses, the plan is to improve the code base (remove globals, improve error handling, clean up the threading, etc) while it expands into new areas.

On my test vehicle (Toyota RAV4), the cameras are setup at the front interior top, inside the windshield, about 22” apart, looking out the windshield. At the moment, they collect images at five frames-per-second, although it's quite clear that there is not enough bandwidth to broadcast them at a 'real-time' rate. This will be one of the issues to be worked out over time.

The reason for the data collection is to develop algorithms to handle driving conditions that are less than ideal such as snow/ice, poorly marked roads, unmapped roads, road work, etc. Here is NH, there exists a bountiful supply of such conditions. But first after looking at my first data collect (sample images from it are posted below) I will need to work on a few things; removing camera motion blur, develop better techniques to align the two cameras, come up with ways to improve through-put of the image streams and make the software easier to use while on the road. Besides the algorithms, I am now going to follow up this work to determine if it's possible that a low cost smart camera could be developed from the PiCameras. Other items to work on will be to add vehicle CAN data such as steering wheel motion and gas/break pedal force which will be used to train/test the algorithms while operating in the stated degraded scenarios.

I plan to post updates regularly, which will also include discussions of other UV projects I have going. I can be reached through this blog, and I'm also on the OpenCV, ROS and Raspberry Pi forums.

Samples from couple different data collections I've done so far are below.