Build a Smart AI Voice Assistant with Alexa, Claude & Home Assistant

Skill level: Intermediate → Advanced | Time to complete: 3–4 hours

What you'll build: A fully self-hosted "Jarvis" — Alexa captures your voice, Claude thinks, your home responds. Ask questions, control devices, have natural conversations. No subscriptions, no cloud dependency beyond the Claude API.

What This Actually Is

Most Alexa + Home Assistant setups work like a lookup table: you say a fixed phrase, a fixed action happens. This is different.

This integration makes Alexa the microphone and speaker for Claude, with Home Assistant as the brain connecting everything. You say something natural — "is the battery charged?" or "it's too warm in here" — and Claude figures out what to do, checks your actual sensors, and responds in plain speech.

Architecture

You speak
  → Alexa (speech-to-text, free, instant)
  → AWS Lambda / Jarvis Skill (free tier)
  → Home Assistant /api/conversation/process
  → Anthropic Claude (your API key)
  → Response text back to Lambda
  → Alexa speaks the reply

Note on Piper TTS: You do NOT need the Piper add-on for this setup. Alexa handles text-to-speech natively and does it extremely well. Piper is only needed for setups where HA plays audio through a media player.

What You'll Need

• Home Assistant — HA Green or any recent HA OS installation

• Anthropic API key — from console.anthropic.com, pay-as-you-go, ~cents/day for home use

• Amazon Echo — any Echo device

• AWS account — free, Lambda free tier covers this usage completely (1 million requests/month)

• Amazon Developer account — free, must be the same email as your Echo's Amazon account

  This last point is critical — if your developer account and Echo are registered to different Amazon accounts, the skill will never appear on your device.

Part 1: Set Up Claude in Home Assistant

1.1 Install the native Anthropic integration

Skip any HACS integrations. HA has a native Anthropic integration built right in.

Settings → Devices & Services → Add Integration → search "Anthropic"

Enter your API key from console.anthropic.com. Done. Why not "Extended OpenAI Conversation"? Anthropic's API is not fully OpenAI-compatible, particularly around tools and function calling. The native integration handles this correctly.

1.2 Configure Claude's personality

Settings → Devices & Services → Anthropic Conversation → Configure. Set the model to claude-sonnet-4-6. Paste a system prompt like this, customised for your home:

You are Jarvis, the AI home assistant for [your name]'s house in [your city], Netherlands.
Your responses will be read aloud by Alexa, so keep them concise — maximum 2-3 sentences unless asked for more detail.
Never use markdown, bullet points, emoji, or formatting. Speak in plain natural language only.
Use Celsius for temperature. Use 24-hour time.
The house is located in [your city]. When asked about local weather or anything location-dependent, assume [your city] unless told otherwise.
You can control smart home devices and answer general questions.
If you successfully control a device, confirm it in one short sentence.
If you cannot do something, say so briefly.

Important: Never use emoji in the system prompt or messages. Alexa's TTS will say "emoji" or skip characters when it encounters them. Set Max tokens to 300.

1.3 Expose your devices to Claude

Settings → Voice Assistants → Expose tab. Toggle on lights, switches, blinds, climate, and key sensors you want Claude to see and control. The more you expose, the more Claude can do — but keep it to what's genuinely useful.

1.4 Create your Jarvis Assist pipeline

Settings → Voice Assistants → Add Assistant. Name it (e.g. "All House Jarvis"), set Conversation agent to Anthropic Conversation. STT and TTS are not needed — Alexa handles both. Save.

1.5 Find your agent ID

Go to Developer Tools → Template and paste:

{{ integration_entities('anthropic') }}

Note the entity starting with conversation. — something like conversation.all_house_jarvis. You'll need this in Part 2.

1.6 Enable the Alexa Smart Home endpoint

Add this to configuration.yaml, then do a full HA restart:

alexa:
  smart_home:

1.7 Test Claude directly first

Before touching Alexa, verify Claude can control your home. Developer Tools → Services:

service: conversation.process
data:
  text: "turn on the living room lights"
  agent_id: conversation.all_house_jarvis

If the lights turn on, the HA + Claude side is working.

Part 2: AWS Lambda — The Jarvis Skill Backend

The Lambda function is the bridge between Alexa and HA. When you speak to Alexa, it calls this function, which forwards your words to Claude and returns the spoken response.

2.1 AWS account and region

Create a free account at aws.amazon.com. Set your region to eu-west-1 (Ireland) — top right of the AWS console. Alexa skills must use us-east-1 or eu-west-1.

2.2 Create a Long-Lived Access Token in HA

HA → your profile (bottom left) → Long-Lived Access Tokens → Create token. Name it jarvis-lambda. Copy it immediately — you only see it once.

2.3 Create the Lambda function

AWS Console → Lambda → Create function. Name: jarvis-skill, Runtime: Python 3.12, Architecture: x86_64. Create.

2.4 Paste the Lambda code

Replace all existing code with the following. Replace conversation.all_house_jarvis with your actual agent ID from Step 1.5.

import json
import urllib.request
import os
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

HA_URL = os.environ.get('HA_URL')
HA_TOKEN = os.environ.get('HA_TOKEN')

def call_ha_conversation(text):
    url = f"{HA_URL}/api/conversation/process"
    payload = json.dumps({
        "text": text,
        "language": "en",
        "agent_id": "conversation.all_house_jarvis"  # <- replace with your agent ID
    }).encode('utf-8')
    
    req = urllib.request.Request(
        url,
        data=payload,
        headers={
            'Authorization': f'Bearer {HA_TOKEN}',
            'Content-Type': 'application/json'
        }
    )
    
    with urllib.request.urlopen(req, timeout=25) as response:
        result = json.loads(response.read())
        return result['response']['speech']['plain']['speech']

def build_response(text, end_session=False):
    response = {
        "version": "1.0",
        "response": {
            "outputSpeech": {
                "type": "PlainText",
                "text": text
            },
            "shouldEndSession": end_session
        }
    }
    if not end_session:
        response["response"]["reprompt"] = {
            "outputSpeech": {
                "type": "PlainText",
                "text": "Is there anything else?"
            }
        }
    return response

def lambda_handler(event, context):
    logger.info(f"Event: {json.dumps(event)}")
    request_type = event['request']['type']
    
    if request_type == 'LaunchRequest':
        return build_response("Jarvis online. What can I do for you?")
    
    elif request_type == 'IntentRequest':
        intent_name = event['request']['intent']['name']
        
        if intent_name in ('AMAZON.StopIntent', 'AMAZON.CancelIntent'):
            return build_response("Goodbye.", end_session=True)
        
        query = None
        try:
            query = event['request']['intent']['slots']['query']['value']
            logger.info(f"Got query from slot: {query}")
        except (KeyError, TypeError):
            pass
        
        if not query:
            return build_response("I didn't catch that. What would you like to know?")
        
        try:
            reply = call_ha_conversation(query)
            logger.info(f"Got reply: {reply}")
            return build_response(reply)
        except Exception as e:
            logger.error(f"Exception: {e}")
            return build_response("I had trouble reaching home assistant. Please try again.")
    
    elif request_type == 'SessionEndedRequest':
        return {"version": "1.0", "response": {}}
    
    return build_response("I'm not sure how to handle that.")

Click Deploy.

2.5 Environment variables and timeout

Configuration → Environment variables → Add: HA_URL (your HA external URL including port if needed, e.g. https://yourhouse.duckdns.org:8123) and HA_TOKEN (your long-lived access token).

Configuration → General configuration → set timeout to 30 seconds. The default 3 seconds is too short for a Claude API round-trip.

Note on port: If your ISP blocks inbound port 443 (common with Dutch residential fiber including Glasoperator), keep the :8123 port in the URL. Lambda calls HA directly, so port 443 is not required.

Copy your Lambda ARN from the top of the function page — you need it for Part 3.

Part 3: Create the Alexa Custom Skill

3.1 Create the skill

developer.amazon.com → Create Skill. Skill name: My Jarvis (must be at least two words). Primary locale: English (GB). Experience: Other → Custom. Hosting: Provision your own. Template: Start from scratch.

Why Custom and not Smart Home? Smart Home skills handle device control commands but cannot do open-ended conversation. Custom skills can receive any spoken text and return any spoken response.

3.2 Set invocation name

Build → Invocations → Skill Invocation Name → set to: my jarvis. Save.

3.3 Set up the intent model

Build → Interaction Model → JSON Editor. Replace everything with the following, then click Save Model and Build Model. Wait for the green Build Successful checkmark — Save alone does not activate changes.

{
  "interactionModel": {
    "languageModel": {
      "invocationName": "my jarvis",
      "intents": [
        {
          "name": "JarvisIntent",
          "slots": [{"name": "query", "type": "AMAZON.SearchQuery"}],
          "samples": [
            "tell me {query}", "please {query}", "what is {query}",
            "what's {query}", "can you {query}", "how {query}",
            "how's {query}", "when {query}", "when's {query}",
            "where {query}", "who {query}", "why {query}",
            "which {query}", "turn {query}", "set {query}",
            "show {query}", "check {query}", "is {query}",
            "are {query}", "do {query}", "does {query}",
            "will {query}", "would {query}", "could {query}",
            "should {query}", "I want {query}", "I need {query}",
            "give me {query}", "find {query}", "make {query}",
            "run {query}", "start {query}", "stop {query}",
            "open {query}", "close {query}", "increase {query}",
            "decrease {query}", "raise {query}", "lower {query}",
            "switch {query}", "put {query}", "get {query}",
            "jarvis {query}", "ok {query}", "okay {query}",
            "yes {query}", "no {query}", "and {query}",
            "also {query}", "actually {query}", "well {query}",
            "now {query}", "{query} please"
          ]
        },
        {"name": "AMAZON.FallbackIntent"},
        {"name": "AMAZON.CancelIntent"},
        {"name": "AMAZON.StopIntent"},
        {"name": "AMAZON.HelpIntent"}
      ]
    }
  }
}

3.4 Set the endpoint and Lambda trigger

Build → Endpoint → AWS Lambda ARN. Paste your Lambda ARN into the Default Region field. Save. Copy the Skill ID shown on this page.

Back in AWS Lambda → Configuration → Triggers → Add trigger. Source: Alexa Skills Kit. Paste the Skill ID. Save.

Part 4: Account Linking with Login With Amazon

The standard approach of using your HA URL for OAuth fails if your ISP blocks port 443 (very common with Dutch residential fiber). The reliable solution is Login With Amazon (LWA) — Amazon authenticates on their own servers, completely bypassing your HA URL during the login flow.

4.1 Create a Login With Amazon security profile

developer.amazon.com → Apps & Services → Login With Amazon → Create a New Security Profile. Name: HA, Description: Home Assistant, Privacy URL: your HA external URL.

Click the gear icon on the new profile → Web Settings → Edit. In Allowed Return URLs, add all three (replace YOUR_SKILL_ID with your actual skill ID, e.g. M2DOHXCC9DYMG9):

https://pitangui.amazon.com/api/skill/link/YOUR_SKILL_ID
https://alexa.amazon.co.jp/api/skill/link/YOUR_SKILL_ID
https://layla.amazon.com/api/skill/link/YOUR_SKILL_ID

In Allowed Origins, add your HA external URL (including port). Note the Client ID and Client Secret from this page.

4.2 Configure Account Linking in the Alexa Developer Console

Your skill → Account Linking:

Authorization URI:               https://www.amazon.com/ap/oa
Access Token URI:                https://api.amazon.com/auth/o2/token
Client ID:                       (from your LWA security profile)
Client Secret:                   (from your LWA security profile)
Client Authentication Scheme:    Credentials in request body
Scope:                           profile

Make sure "Allow users to enable the skill without account linking" is off. Save.

4.3 Enable the skill on your Echo

Alexa app → More → Skills & Games → Your Skills → Dev tab → your Jarvis skill → Enable to use. Log in with your Amazon account when prompted.

Using Jarvis Day-to-Day

The usage pattern is: say "Alexa, open my Jarvis" to start a session, speak naturally, say "Alexa, stop" to end.

"Alexa, open my Jarvis"
→ "Jarvis online. What can I do for you?"

"what is the temperature outside"
→ "It's currently 24 degrees outside."

"and the battery level of the Marstek"
→ "The Marstek battery is at 87 percent."

"turn off the garden lights"
→ "Done, garden lights are off."

"Alexa, stop"

The session stays open after each response. Alexa listens for your next sentence immediately. On Alexa intercepting some queries: Alexa has built-in handlers for weather, music, and timers. If she answers instead of Jarvis, rephrase to something more specific ("what does the Buienradar sensor say" instead of "what's the weather"). Within an open Jarvis session this is rarely a problem.

Troubleshooting

CloudWatch logs are your best debugging tool. Every Lambda invocation is logged in full: AWS Console → Lambda → jarvis-skill → Monitor → View CloudWatch logs.

• "Sorry, something went wrong" — Lambda timeout or exception. Check CloudWatch logs immediately.

• "I didn't catch that" — Query didn't match any sample utterance. Start with a carrier phrase like "tell me" or "please".

• Skill not in Dev tab — Developer account and Echo must be the same Amazon account.

• "Invalid redirect URI" — Follow Part 4 (LWA approach) exactly. Do not use your HA URL as Authorization URI.

• Lambda not triggered at all — Check Lambda → Configuration → Triggers. The Alexa Skills Kit trigger must be there with the correct Skill ID.

• Wrong assistant responding or no location context — Confirm agent_id in the Lambda code matches your exact entity ID from Step 1.5.

• Runtime.ImportModuleError in CloudWatch — The Lambda code provided uses only Python standard library (urllib, json, os, logging) — no external dependencies needed. If you see this error, you may have old code that imported the requests library.

What Jarvis Can and Cannot Do

• Can control: lights, blinds, switches, climate, and any script you expose

• Can answer: general questions, sensor readings, entity states — with your location context baked in

• Cannot control number entities directly (e.g. fridge temperature slider) — workaround: create an HA script that sets the value and expose the script to Claude

• Does not retain memory between sessions — each "Alexa, open my Jarvis" starts fresh. Within a session, follow-up questions work naturally.

Guide written by a Home Assistant enthusiast in Utrecht. Built with Home Assistant, a lot of CloudWatch logs, and one very patient AI.