Skip to content
nephridium labs

AI powered Furby

Last updated:

Play

Introduction

The 2023 Furby is an adorable toy to keep kids engaged. It comes with a set of modes that can be triggered by saying certain phrases, e.g. it will start singing a song when it hears “Dance party”. Although Furby offers dozens of context-specific responses it is not capable of understanding phrases outside of its programming.

This is where LLM based AI comes in - while models don’t really “understand” arbitrary phrases either they do a good job simulating it (good enough to pass the Turing test). What if we could make Furby “intelligent” by connecting it with an AI service? For this project will leverage Home Assistant and ChatGPT to make that happen.

The Furby hardware is great for this use case as it provides:

  • a microphone for listening
  • a speaker for answering
  • LEDs that can be used as a status indicators
  • motors stearing eyes, ears and legs may also be useful for additional status indicators

Home Assistant implements some key ingredients for use as the “brains” of a smart companion:

  • Speech-to-text for receiving questions/commands
  • Text-to-speech for responding
  • AI integrations for creating interesting responses, e.g. via ChatGPT, Gemini or local LLMs
  • Sensor integrations for sampling the environment, e.g. for proximity, temperature, imaging (via cameras)

Home Assistant runs on the affordable Raspberry Pi 4 (and above). Though they have a fairly small footprint they are unfortunately still a bit too big to integrate into the Furby’s body. While a Raspberry Pi Zero would fit the current version is not performant enough to run Home Assistant well.

So instead we will use a “Home Assistant Voice” device to interface with our Home Assistant server. It is uniquely suitable as it handles the voice input task well using performant microphones and a dedicated sound chip. It will fit in the Furby once we make a few minor modifications. Additionally, it features an indicator LED ring that shines through the Furby’s tummy letting us know if Furby is listening or thinking.

(Alternatively we could consider using a Voice Assistant satellite running on a Raspberry Pi Zero with a microphone HAT, although it will not be as performant.)

Software Setup

Home Assistant OS

We will run the standard Home Assistant installation on a Raspberry Pi 5 and set it up with the OpenAI ChatGPT integration, then connect the Home Assistant Voice device.

Prerequisites

In order to use the cloud based TTS/STT services which are faster and offer more voices compared to local solutions you will need a Nabu Casa account.

(It is also possible to set up a fully local smart Furby that does not require an internet connection by running TTS/STT on the Raspberry Pi and by connecting Home Assistant to a local Ollama server we have set up for the AI agent. We will do that in a follow-up post.)

Network connectivity

For internet connectivity we could simply plug a LAN cable into the Raspberry Pi. I would like to be able to easily move the Furby between different locations so I’ll set up WIFI instead:

  • Create a separate subnet/WLAN SSID so the new Home Assistant instance won’t interfere with the existing one.
  • Create a Home Assistant SD Card using Raspberry Pi Imager.
  • Boot it up the Raspberry Pi with a monitor and keyboard connected.
  • Execute login to get to shell of the Home Assistant docker container:
    Terminal window
    login
  • From here you can connect to your dedicated SSID, to make sure the WIFI password is not saved in the shell history run
    Terminal window
    HISTFILE=/dev/null
    then connect using
    Terminal window
    nmcli device wifi connect [AP name] password [password]
  • A few commands to analyse your WLAN connectivity are:
    Terminal window
    mncli radio  # shows WIFI adapters and their state
    nmcli device wifi  # displays a list of reacheable access points
    ip addr show  # display IP address
  • Home Assistant will now attemp to download the latest version of Home Assistant Core. This may take several minutes.
  • Once the update is done you can connect to the Home Assistant instance on port 8123 using a web browser to go through the initial set up process.

Raspberry Pi 5 cooling

Depending on where you are going to store the Raspberry Pi you might want to make sure the fan keeps running. To achieve this we need to add fan control params to the boot config (taken from here):

From the Home Assistant shell log into the Home Assistant container

Terminal window
login

Edit the config

Terminal window
vi mnt/boot/config.txt

Type i for insert/edit mode. At the bottom in the [all] section add:

[all]
dtparam=fan_temp0=35000
dtparam=fan_temp0_hyst=3000
dtparam=fan_temp0_speed=75


dtparam=fan_temp1=48000
dtparam=fan_temp1_hyst=3000
dtparam=fan_temp1_speed=100


dtparam=fan_temp2=55000
dtparam=fan_temp2_hyst=5000
dtparam=fan_temp2_speed=200
  • dtparam=fan_temp0=35000 will turn the fan on once CPU temperature exceeds 35°C (35000 millidegrees)
  • dtparam=fan_temp0_hyst=3000 set the hystersis value to 3°C (3000 millidegrees), which means the fan will keep running after hitting 35°C all the way to 32°C (35°C - 3°C)
  • dtparam=fan_temp0_speed=75 sets the fan speed to “75”, which is practically inaudible, with values above 100 the fan starts to become noticeable.

Save/quit vi by typing <ESC> : wq <Enter>.

If in need for more aggressive cooling (with constantly audible fan noise) we could use something like this:

dtparam=fan_temp0=35000
dtparam=fan_temp0_hyst=15000
dtparam=fan_temp0_speed=175

Home Assistant Voice

Setup of the device is pretty straight forward, Home Assistant will guide you through the process. Once you connect the device to a power source an option to add the device should show up under “Integrations” within seconds. During the process you will be prompted to enter the WIFI credentials and a few other options such as wakeword and area.

OpenAI Integration

To make Furby respond as a furry little cognizant pet we can leverage ChatGPT:

  • Under “Integrations” add “OpenAI Conversation” and add you Open AI API key.
  • Then for the personality you can add “Instructions” such as this one: 
    You are a cute cuddly Furby. Your answers should be accurate, funny and comprehendible by a 10 year old. Assume your answer is spoken, so do not use emojis or other visual cues.
  • Make sure to name it appropriately (e.g. “FurbyGPT”) if you plan on adding different Chatbot configs.
  • If you want to add another service, e.g. with different personalities or languages (in which case the instructions should be written in the target language), go to the “Open AI” integration -> “Add service”, and then simply enter the API key again and go through the other steps as above.

AI Assistants

Now it’s time to connect the AI Conversation endpoints with a Voice Assistant.

  • Go to “Settings” -> “Voice assistants” -> “Add Assistant”
  • Pick a name and language (for multiple languages it pays off to have consistent naming with the Conversation integration, languages could be represented by a suffix based on ISO 639 language codes or IETF language tags).
  • As “Conversation agent” select your previously added AI powered assistant (e.g. “FurbyGPT”).
  • For “Speech-to-text” and “Text-to-speech” select “Home Assistant Cloud”.
  • Finally select the desired voice. For Furby’s voice I found “SaraNeural” quite adequate (American English).

Hardware Setup

The Home Assistant Voice’s mainboard has just the dimensions to fit into the Furby. To implant it we need to open it up and remove some plastic casing. We will use the Furby’s speaker and Home Assistant Voice’s excellent microphones which work well through the fur.

Uncovering the old brain

  • The patient can be sedated by taking out the batteries at the bottom, then remove all screws from the battery compartment.
  • The fur is kept in place by latches that can be opened inserting a SIM card tray eject tool into the small holes while pulling.
  • Detach the module containing the speaker and tummy sensor. We will remove the module’s plastic casing but we can keep the capacitive cage for protecting the speaker. This will also allow us later on to place it on top the Home Assistant Voice so it can be used to push its button.

Implanting the new brain

  • The space between Furby’s beak and its feet is just missing about 3 mm. We can cut that amount from the beak so the devices fits snuggly in there.
  • If you want to keep the original mainboard inside cover it wil electrical tape to be sure there can be no shorts.
  • The two pinouts prevent us from placing the board in snuggly so unfortunately we will have to clip off the pins.
  • To fix the Home Assistant Voice to the case we drill through the board’s two holes at the bottom so we can fix it with two screws (at the top it is fixed by the beak cutout).
  • Connect Furby’s speaker with the Home Assistant Voice’s speaker cable at the back of the board.
  • Place the speaker in the capacitive cage and stick it to the Home Assistant Voice’s button with double sided tape.
  • Orient the board so the USB-C and audio plug are easily accessible at the bottom.
  • Pull the fur back over the Furby, there should be enough tension to keep it in place without re-engaging the latches. Make sure there is no pressure on the button so it is no permanently pressed.

Hello Furby!

Brain transplant complete! We can now connect the USB-C to a power source (preferrably with an angle connector), once Furby has woken up it will enthusiastically put its new AI capabilities to use ;)