Build Your Own AI Voice Assistant: ESP32-S3, Home Assistant & Local Audio (No cloud needed)

In our previous guides, we loved the ESP32-C3 for temperature sensors and WLED. It is cheap and efficient. But today, we are building ears and a mouth for your home.

For audio processing, the C3 is too weak. To detect a wake word like "Hey Jarvis" locally—without sending audio to the cloud—we need heavy processing power. We need the ESP32-S3.

Step 1: Preparing Home Assistant (The Brains)

Before we wire up the hardware, we need to ensure Home Assistant has the "brains" to understand English and talk back. We need to install three add-ons.

1. Go to Settings -> Add-ons -> Add-on Store.
2. Install Whisper (Speech-to-Text). This converts your voice recording into text.
3. Install Piper (Text-to-Speech). This creates the computer voice that talks back.
4. Install openWakeWord. Even though the S3 chip handles detection, this add-on manages the models.

Once installed, go to Settings -> Voice Assistants and make sure you have a pipeline active that uses these three services. This is the "server" your ESP32 will talk to.

Step 2: The Shopping List

Unlike a smart plug, we are building this from components. You will need:

Step 3: The Wiring (I2S Protocol)

We are using I2S (Inter-IC Sound), which is a standard for connecting digital audio devices. It requires 3 wires: Clock (BCLK), Word Select (LRC), and Data (DIN/DOUT).

Note: You can change these GPIO pins in the software, but these are standard defaults for many S3 boards.

Step 4: The Software (ESPHome)

We will use ESPHome to program the chip. You will need a specific configuration that includes the "Micro Wake Word" component.

Create a new device in ESPHome and use this configuration block for the I2S setup:

i2s_audio:
  - id: i2s_bus
    i2s_lrclk_pin: GPIO42
    i2s_bclk_pin: GPIO41

microphone:
  - platform: i2s_audio
    id: board_microphone
    i2s_din_pin: GPIO40
    adc_type: external
    pdm: false

speaker:
  - platform: i2s_audio
    id: board_speaker
    i2s_dout_pin: GPIO39
    dac_type: external
    mode: mono

voice_assistant:
  microphone: board_microphone
  speaker: board_speaker
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0

Step 5: The Test

Once you flash the ESP32-S3, Home Assistant will auto-discover it. Add it, and then go to Settings -> Voice Assistants.

1. Select your device.
2. Choose your Wake Word (e.g., "Okay Nabu" or "Hey Jarvis").
3. Speak!

When you speak the wake word, the ESP32-S3 detects it locally. It then streams the audio to the Whisper add-on (Step 1) to convert it to text. Home Assistant processes the command, and sends the response back to Piper, which plays the audio out of your DIY speaker.

Conclusion

You have just built a device that rivals Amazon Echo in functionality but respects your privacy completely. It costs about $15 in parts and gives you total control over the hardware.