Voice notes¶
Talk to Humagen from wherever you are. A voice note is transcribed before the run it starts, so an agent reads what you said, and the recording stays with the message. It is off until a workspace admin sets up a voice provider, or the System spec names a speech kind.
A workspace voice provider¶
An admin (anyone who may manage the workspace's clients, clients:write)
sets one up in Workspace settings › Voice:
- Choose Groq or OpenAI.
- Paste the provider's API key, or give an
env:HUMAGEN_*reference this server grants the workspace (-secret-refs). The key is never shown again. - Optionally pick the voice, the models and the prices.
- Press Try Voice: the provider speaks a sample and hears it back.
With a provider set up, the console's microphone and voice notes from every channel are heard by it, and voice mode is on. Turn Off forgets the provider and its key.
What each call costs is counted in the workspace's usage, toward its spend limits, from the provider's published price unless the admin set another:
| Kind | Hears | Speaks | Defaults |
|---|---|---|---|
groq |
$0.04 an hour of audio, at least 10 seconds a request | $22 a million characters | whisper-large-v3-turbo, canopylabs/orpheus-v1-english, voice autumn |
openai |
$0.006 a minute of audio | $15 a million characters | whisper-1, tts-1, voice alloy |
A transcription whose length the provider did not say is counted as an
unpriced call, never as free. The API: GET, PUT and DELETE
/api/voice/provider, and POST /api/voice/provider/check.
Voice mode in a session¶
With a workspace voice provider set up, a session's composer has a voice mode button beside the microphone. In voice mode you talk with the session's agent: what you say is heard by the provider and sent as an ordinary message, marked spoken, so the agent (and the entry agent's routing) answers it as it would a typed one, with its own model, tools and approvals. The answer is written in the session as usual and spoken a few sentences at a time.
| Call | What it does |
|---|---|
POST /api/sessions/{id}/voice/transcriptions |
Hears a recording (its own bytes, at most 10 MiB) and answers its words. Nothing is kept |
POST /api/messages/{id} with "spoken": true on the message (or POST /api/sessions/{id}/messages with "spoken": true) |
Sends the words as any message is sent, so it gets the same one answer; spoken only marks the message |
POST /api/sessions/{id}/voice/speech with {"run", "from"} |
Speaks the next few sentences of that run's answer, from offset from (0, then each answer's next) |
PATCH /api/sessions/{id}/messages/{mid} with {"text"} |
Puts right a spoken message's words, for its author only, so a misheard word is right in every later turn; the runs it already started are not changed |
What is spoken is read on the server from the answer of a run of that
session, never sent by the client, so voice mode cannot speak anything else
on the workspace's account. It is plain words: no Markdown, a link by its
text, an address as "a link", a table left out, a code block as "The code is
in the session." While the answer is still being written and has no whole
sentence after from, the answer is pending and nothing is spent.
Each call is counted in the usage ledger, toward the session's project, its
agent and the workspace, and on the session's own voice usage (seconds
heard, characters spoken, cost). When a spend limit over any of them is
reached, the calls answer 402 and nothing is sent to the provider. Both take
speech:write and a session the caller can see.
Spoken answers in Slack¶
A voice clip sent to the app (a direct message, or a thread the app takes
part in) arrives on its run as an audio attachment with no words. When the
run's turn starts, the clip is heard with the workspace's voice provider and
its words go into what the agent reads, marked [Voice note, english], as a
voice note posted through the API is. The words are kept with the run, so a
retried or resumed turn neither hears nor pays for the clip again, and the run
is remembered as answering a spoken message.
When that run completes, the answer is written in the thread as usual, and a
completion step then speaks it and uploads it as an audio clip in the same
thread ("Spoken answer"). It runs once a run, after the run has ended; when
it cannot speak (no voice provider, a spend limit, no words, Slack refusing
the upload) it logs why and leaves the written answer. The Slack app needs the
files:write scope (in humagen slack manifest and the Slack connection's
OAuth scopes); an installation from before it was added reinstalls to grant it.
POST /api/runs/{id}/speech makes the same clip for any caller: the answer
of a run that has ended, as plain words, at most 1,200 characters, a longer
answer cut at a sentence and ending "The rest is in the thread." It speaks
only the run's own words, takes speech:write on a run the caller can see,
and is counted like voice mode's calls. It answers 409 while the run works.
Slack huddles are out of reach: Slack's public API has no supported way for an app to join a huddle or to hear or speak its audio (it offers huddle status events, and the Calls API for linking a call hosted elsewhere).
A deployment speech kind¶
Groq transcribes with Whisper. Set GROQ_API_KEY in the server's
environment and add a speech slot to the System document:
That is all: the key reference defaults to env:GROQ_API_KEY, the model to
whisper-large-v3-turbo, the size cap to 25 MiB and one request to 60
seconds. Every field is in
the spec reference. The same key can run
agents on Groq's models: add a model of kind: groq to a Platform document
and name it as an agent's provider.
api_version: humagen/v1
kind: Platform
metadata:
name: default
spec:
models:
- name: groq
kind: groq
Where you can send one¶
| From | How | What the agent reads |
|---|---|---|
| Slack | Record an audio clip in a thread the app answers, or attach an audio file | The message's text, then [Voice note, english] what you said. With a workspace voice provider the agent reads the clip's words, and the answer comes back as text and as an audio clip in the thread |
| The console | Press the microphone in the message box, speak, press it again | The words come back into the box, marked the same way, for you to read, change and send |
| WhatsApp, iMessage, SMS | Not yet: the messaging connectors in urmzd/saige will carry a voice note as an audio attachment and call the same hook |
The same |
A message has at most four voice notes transcribed. A note that is too long,
has no words in it, or reaches a provider that fails or does not answer in
time is said to be so in the message ([Voice note: clip.m4a could not be
transcribed]), and the message still posts. Without a voice provider or a
speech kind a voice note is still attached, and the console's microphone
falls back to the browser's own dictation where the browser has it.
Who can use it¶
Recording a voice note in the console takes speech:write, which operators,
managers and admins hold; GET /api/voice says whether it is on. A voice
note posted from Slack is transcribed under the poster's own right to post.
The console's recording is sent to the workspace's voice provider (or the
deployment's speech kind) and not kept by Humagen; a voice note from a
channel is kept as the attachment it arrived as, like any other file.
For a connector¶
The hook is (*platform.Platform).TranscribeVoiceNotes(ctx, tenant, text,
attachments). A connector stores the audio as an attachment, as it does any
file, and passes the message's text and attachments; it gets back the text
with each transcript after it. It knows nothing of the channel, so WhatsApp,
iMessage and SMS use it as Slack does. A new speech provider is a kind in the
speech slot: humagen.WithKind(humagen.Speech, ...) returning a
speech.Transcriber.