LED controller

Virtual Reality Goggles for Laboratroy Mice

This project is a part of Master thesis and builds on a device from Cornell University - Eye-Tracking Goggles. This readme is also a guide for anyone at the 4th floor who would try to make some changes or redesign the whole thing.

This code is an improvement on top of their existing build. We still work with two Raspberry Pis; one for running the game engine and one for eye tracking.

However, this project largely improves the device with the following points:

• Addition of standard neuroscience stimuli
• Web-based GUI
• Synchronization via pins with other devices (like microscope)
• Automatic file handling (conversion of binary videos to .avi format and transfering all files to flash drive.)
• PCB for LEDs control
• 3D priinted case and screens mounting

Structure of the code

Folder Camera Pi has a simple camera controller that runs on boot and waits for the signal to start recording. That goes into Raspberry Pi 5 (It is actually not important what Raspberry Pi model is used).

Godot Pi contains all the code that goes into the other Raspberry Pi. This one needs to be a Pi that is able to run Bullseye OS. The setup created by Cornell University relies on a framebuffer that can only be only compiled with this legacy operating system. There are definitely other solutions how to stream the screen headlessly, however that is beyond the scope of what I was able to do during my limited time (maybe it is really easy solution, I didn't have time to try to solve it).

Godot Pi code

In our setup, the framebuffer runs on boot and the wallpaper is set to black color. The framebuffer takes a while to initalize, thus has to be prepared before the experiment. Since the screens are OLED screens, there is no stimulus with black color.

On boot operations here are done through /etc/xdg/lxsession/LXDE-pi/autostart because running the controller as a service resulted in several problems.

Controller

Python script controller.py is ran on boot. It has to be ran with superuser permissions in order to do everything that is needed. It runs several threads to take care of the following things:

• Starts the server and listens for signals from it
• Starts the game engines
• Runs a udp server to communicate with the game engines and Video recording Raspberry Pi
• Listens for TTL signals
• Listens to neurotar (Mouse movement tracker), processes the signal and passes it to game engines
• Handles all the files after the experiment is done

Server

Very simple http server that serves one web page with all the settings. You can connect to it from a different computer and thus there is no need to connect a screen to Raspberry Pi. Web page communicates via API calls and Server can send events via channel that the client had subscribed to when requesting the web page. All the code is html and vanilla javascript to make it as lightweight as possible. Copilot was used when coding the javascript.

Graphical User Interface (GUI)

This device wasnt the only part of my thesis, but rather one third. Thus I needed to iterate as fast as possible to prototype asn used a copilot. Thus there is big room for improvement in this code, but it works and if it works it's not stupid.

• index.html is the main served file. It has the main layout. If you want to add new tab its id must match the content div(e.g. id="tab-gratings" for the button and id="gratings" class="tab-content" for the div with the settings). All the javscript is added at the bottom of the code.
• buttons.js takes care of the buttons (obviously) including the prepare button with its play functions. Then it sources a lot of functions from play.js
• events.js handles events coming from the raspberry pi. Mainly toggle the possibility to use inputs and buttons and shows or hides the loader
• play.js takes care of extracting the data from the settings and putting them into a json structure that is then sent to raspberry pi

Godot

All the scenes were made by scientists at Cornell University and are available in their repository linked above. However, the files here have significant changes, neccessary for correct functioning. In the original design, Godot was the master controller. That is no longer true here and the engine is controlled by the python controller. It listens through a UDP connection for signals and movement data. The scene is no longer selected in the godot interface, but rather in the web GUI.

Python games

Since most of the standard stimuli are very simple, there is no need to use a strong game engine like godot and everything can be created by a simple python scripts running Pygame. This allows us to quickly add new stimuli if requested.

Connecting Raspberry Pi to laboratory computer for neurotar

After considering several solutions, given our laboratory setup at Copenhagen University, I decided to connect Raspberry Pi to computer via bluetooth. However, raspberry Pi doesn't support serial communication by default. You will need to follow this tutorial and restart the windows computer at the end to connect to connect via bluetooth and use serial port. You can then check the bluetooth devices and see what port it uses. You should see outgoing and incoming ports. You want to stream neurotar to the outgoing one.

Connecting raspberry pis to each other

I managed to connect both raspberry pis to IoT wifi (NC_IoT), however they always get a different IP address. You can ping both of them using godot.local and CameraUnit.local from a personal computer, however this for some reason doesn't work when they want to ping each other. Thus I connected them with an ethernet cable as suggested in the orignal design. If you make it work via wifi, it is definitely more elegant solution. However even for ethernet connection is necessary that you go to this file /etc/dhcpcd.conf and add configuration. Follow this tutorial

Trasnfering files to USB

This code expects USB flash drive with the name VR_USB. If you want to change that, go to utils/consts.py/ and change USB_NAME.

Utils

Utils have some handy code that is used in controller.py.

• binary_to_avi.py - videos are recorded in binary format to enhance performance. After experiment is done and we have time, the binary video is converted to .avi file for easier access.
• gpio_setup.py takes care of setting up and cleaning of GPIOs
• start_games.py formats the arguments from GUI into proper bash commands that are run in a separate thread to startup the games.
• thread_safe_storage.py creates a new class that is basically a dictionary but is threadsafe as we operate with several threads in the controller.py. This way you get only the nice python UI. It also implements callback that is called everytime a some value changes or new value is added. I used it to automatically send updates to GUI every time some status variable changes.

Zerotier (Important)

Now comes the part how the GUI is served to the lab computer. The lab computers are connected to internet via ethernet cables and they can not be connected to the IoT wifi. Thus they are not on the same network and the computer can not reach the raspberry pi via godot.local.8080 like a personal computer can. I created new account at ZeroTier.com and added both devices there. It is a free solution that gives both devices a permanent ip address they can use to communicate to each other. Virutally they are on the sam network. If you ever reinstall the raspberry pi with new OS, either contact me so we can solve it together or do the same thing yourself and just contact me to tell me you dont need it anymore.