Building an Interactive AI-Powered Robotic Assistant with BillAI Bass

Practical Summary and Prerequisites

The BillAI Bass project provides a framework to convert a standard Big Mouth Billy Bass animatronic into an interactive, AI-powered voice assistant. The system utilizes a Raspberry Pi 5 running a Strands Agents BidiAgent to facilitate bidirectional streaming with Amazon Nova 2 Sonic via Amazon Bedrock. This setup enables the fish to listen, process speech, and respond with synchronized mouth and head movements. No prior robotics or soldering experience is required, though the project involves basic hardware assembly.

Before beginning, ensure you have an AWS account with access to the amazon.nova-2-sonic-v1:0 model in the us-east-1 region. You must create a dedicated IAM user with an inline policy that restricts permissions exclusively to bedrock:InvokeModelWithBidirectionalStream for the specified model. This security practice ensures that the credentials stored on the device have the least privilege necessary for operation. Additionally, set up a billing alarm in the AWS console to monitor usage costs, as the system is designed for continuous operation. If you are new to this, the project documentation suggests using an AI assistant to pace the build and debug specific error messages.

System Setup and Audio Configuration

The software environment requires Raspberry Pi OS Lite (64-bit) to avoid conflicts with desktop-specific audio management services like Wayfire, which can interfere with audio configurations. After flashing the SD card and enabling SSH, install the necessary system dependencies: python3-dev, portaudio19-dev, swig, and git. Proper audio configuration is critical; you must identify your USB microphone and speaker using arecord -l and aplay -l, then define them in a ~/.asoundrc file to ensure consistent device mapping across reboots.

Avoid using numeric device IDs, as these can change between sessions. Instead, reference devices by their specific names. Once configured, use wpctl to manage volume levels, as WirePlumber may override standard ALSA settings during the boot sequence. Verify your audio path by recording and playing back a test file before proceeding to the agent implementation. If you encounter issues, ensure the sample rate is set to 16000 Hz, which is the requirement for Nova Sonic. Always verify your configuration by checking that the file exists after saving, as silent failures can occur.

Implementing the BidiAgent

The core functionality is driven by the BidiAgent, which manages the streaming connection to the AI model. You must create a Python virtual environment and install the required packages, including strands-agents, gpiozero, and lgpio. The following code snippet demonstrates the initialization of the Nova 2 Sonic model and the agent's basic execution loop:

import asyncio
from strands.experimental.bidi import BidiAgent, BidiAudioIO
from strands.experimental.bidi.models import BidiNovaSonicModel

model = BidiNovaSonicModel(
    model_id="amazon.nova-2-sonic-v1:0",
    provider_config={
        "audio": {
            "input_rate": 16000,
            "output_rate": 16000,
            "voice": "matthew",
            "channels": 1,
            "format": "pcm",
        }
    },
)

agent = BidiAgent(
    model=model,
    system_prompt=(
        "You are Billy, a wisecracking animatronic singing bass mounted on a "
        "wooden plaque. Keep responses short, punchy, and conversational."
    ),
)

audio_io = BidiAudioIO()

asyncio.run(agent.run(inputs=[audio_io.input()], outputs=[audio_io.output()]))

Always verify your code with python -m py_compile before execution to catch syntax errors introduced during copy-paste operations. If the agent fails to start, ensure your virtual environment is active and that all dependencies are correctly installed. The BidiAgent is designed to handle the bidirectional stream, allowing for real-time interaction between the user and the AI model.

Hardware Integration and Operational Limitations

The physical assembly involves connecting the fish's internal motors to the Raspberry Pi using an MX1508 dual H-bridge motor driver. The modern Gemmy version of the fish typically contains two motors: one for the mouth and one for the head and tail. It is essential to test these motors using the provided motors.py script before integrating them with the main billy.py logic to ensure correct wiring and polarity. Use DuPont jumper wires to connect the driver board to the Pi's GPIO pins.

Be aware of potential limitations, such as audio feedback where the microphone captures the speaker output, causing the model to interpret the fish's own voice as user input. Physical placement and volume adjustment are the primary methods for mitigating this. Additionally, ensure that your AWS credentials are stored securely in ~/.aws/credentials rather than using environment variables, which may not persist across all session types. If the system fails to respond, check the logs for AccessDeniedException or OSError, which often indicate configuration or permission issues. The project is intended for personal use and should be managed with appropriate security practices, such as deactivating keys if the device is lost or compromised.