Membrane Framework

Membrane is a versatile multimedia streaming & processing framework. You can use it to build a media server of your need, that can:

• stream via WebRTC, RTSP, RTMP, HLS, HTTP and other protocols,
• transcode, mix and apply custom processing of video & audio,
• accept and generate / record to MP4, MKV, FLV and other containers,
• handle dynamically connecting and disconnecting streams,
• seamlessly scale and recover from errors,
• do whatever you imagine if you implement it yourself :D Membrane makes it easy to plug in your code at almost any point of processing.

The abbreviations above don't ring any bells? Visit membrane.stream/learn and let Membrane introduce you to the multimedia world!

If you have questions or need consulting, we're for you at our Discord, forum, GitHub discussions, X (Twitter) and via e-mail.

You can also follow Membrane on X (Twitter) or join our Discord to be up to date and get involved in the community.

If you already had a chance to use Membrane, we will be grateful if you could fill out a quick survey to help us improve framework and decide on what to do next.

For people who write Membrane code with AI assistants, we ship a dedicated skill for them.

Membrane is maintained by Software Mansion.

Quick start

Mix.install([
  :membrane_hackney_plugin,
  :membrane_mp3_mad_plugin,
  :membrane_portaudio_plugin,
])

defmodule MyPipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, mp3_url) do
    spec =
      child(%Membrane.Hackney.Source{
        location: mp3_url, hackney_opts: [follow_redirect: true]
      })
      |> child(Membrane.MP3.MAD.Decoder)
      |> child(Membrane.PortAudio.Sink)

    {[spec: spec], %{}}
  end
end

mp3_url = "https://raw.githubusercontent.com/membraneframework/membrane_demo/master/simple_pipeline/sample.mp3"

Membrane.Pipeline.start_link(MyPipeline, mp3_url)

This is an Elixir snippet, that streams an mp3 via HTTP and plays it on your speaker. Here's how to run it:

• Option 1: Click the button below:

  It will install Livebook, an interactive notebook similar to Jupyter, and it'll open the snippet in there for you. Then just click the 'run' button in there.

• Option 2: If you don't want to use Livebook, you can install Elixir, type iex to run the interactive shell and paste the snippet there.

After that, you should hear music playing on your speaker :tada:

To learn step-by-step what exactly happens here, follow this tutorial.

Learning

The best place to learn Membrane is the membrane.stream/learn website and the membrane_demo repository. Try them out, then hack something exciting!

Membrane skill for AI assistants

This repo includes skills/membrane-framework/SKILL.md, a structured guide that helps AI coding assistants write Membrane code.

If you use Claude Code, install it via the plugin system:

/plugin marketplace add membraneframework/membrane_core
/plugin install membrane-framework@membrane

After install, Claude automatically pulls in the skill whenever you work on Membrane code (mentioning Membrane.Pipeline, Bin, Filter, Pad, etc).

If you use Cursor, drop the skill into your project's rules directory:

mkdir -p .cursor/rules && curl -fL https://raw.githubusercontent.com/membraneframework/membrane_core/master/skills/membrane-framework/SKILL.md -o .cursor/rules/membrane-framework.mdc

For a cross-agent install, use skills.sh's CLI via npx:

npx skills add membraneframework/membrane_core -g

For agents skills.sh doesn't support, include the SKILL.md file directly in the agent's context, or copy its contents into whatever instruction file the tool reads.

Structure of the framework

The most basic media processing entities of Membrane are Elements. An element might be able, for example, to mux incoming audio and video streams into MP4, or play raw audio using your sound card. You can create elements yourself, or choose from the ones provided by the framework.

Elements can be organized into a pipeline - a sequence of linked elements that perform a specific task. For example, a pipeline might receive an incoming RTSP stream from a webcam and convert it to an HLS stream, or act as a selective forwarding unit (SFU) to implement your own videoconferencing room. The Quick start section above shows how to create a simple pipeline.

Membrane packages

To embrace modularity, Membrane is delivered to you in multiple packages, including plugins, formats, core and standalone libraries. The complete list of all the Membrane packages maintained by the Membrane team is available here.

Plugins

Plugins provide elements that you can use in your pipeline. Each plugin lives in a membrane_X_plugin repository, where X can be a protocol, codec, container or functionality, for example membrane_opus_plugin. Plugins wrapping a tool or library are named membrane_X_LIBRARYNAME_plugin or just membrane_LIBRARYNAME_plugin, like membrane_mp3_mad_plugin. Plugins are published on hex.pm, for example hex.pm/packages/membrane_opus_plugin and docs are at hexdocs, like hexdocs.pm/membrane_opus_plugin. Some plugins require native libraries installed in your OS. Those requirements, along with usage examples are outlined in each plugin's readme.

Formats

Apart from plugins, Membrane has stream formats, which live in membrane_X_format repositories, where X is usually a codec or container, for example membrane_opus_format. Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container.

Core

The API for creating pipelines (and custom elements too) is provided by membrane_core. To install it, add the following line to your deps in mix.exs and run mix deps.get

{:membrane_core, "~> 1.3"}

Standalone libraries

Last but not least, Membrane provides tools and libraries that can be used standalone and don't depend on the membrane_core, for example video_compositor, ex_sdp or unifex.

Goals

The main goals of Membrane are:

• To make work with multimedia a more pleasant experience than it is now.
• To provide a welcoming ecosystem for learning multimedia development.
• To power resilient, maintainable and scalable systems.

Elixir language

We chose Elixir for Membrane because it's a modern, high-level, easy-to-learn language, that lets us rapidly develop media solutions. Elixir's main selling points are built-in parallelism and fault-tolerance features, so we can build scalable systems that are self-healing. It also runs on the battle-tested BEAM VM, that's been widely used and actively developed since the '80s. When we need the performance of a low-level language, we delegate to Rust or C.

If you don't know Elixir, try this tutorial - it shouldn't take long and you'll know more than enough to get started with Membrane.

Supported platforms

Membrane was created and is being worked on mainly with UNIX-based operating systems in mind - in most cases that means MacOS and Linux. When it comes to Windows, if a project uses any native code, not just pure Elixir, we can't guarantee it works correctly. To use Membrane on Windows we strongly recommend using WSL (Windows Subsystem for Linux), which allows for running a Linux environment on a Windows machine. If you wish to get more insight into direct support for Windows, see this discussion.

Contributing

We welcome everyone to contribute to Membrane. Here are some ways to contribute:

• Spread the word about Membrane! Even though multimedia are present everywhere today, media dev is still quite niche. Let it be no longer!
• Create learning materials. We try our best but can cover only a limited number of Membrane use cases.
• Improve docs. We know it's not the most exciting part, but if you had a hard time understanding the docs, you're the best person to fix them ;)
• Contribute code - plugins, features and bug fixes. It's best to contact us before, so we can provide our help & assistance, and agree on important matters. For details see the contribution guide.

Support and questions

If you have any questions regarding Membrane Framework or need consulting, feel free to contact us via Discord, forum, GitHub discussions, X (Twitter) or e-mail.

All packages

General

Package	Description	Links
membrane_sdk	Full power of Membrane in a single package
membrane_core	The core of Membrane Framework, multimedia processing framework written in Elixir
kino_membrane	Utilities for introspecting Membrane pipelines in Livebook
docker_membrane	[Labs] A docker image based on Ubuntu, with Erlang, Elixir and libraries necessary to test and run the Membrane Framework.
membrane_demo	Examples of using the Membrane Framework
membrane_tutorials	Repository which contains text and assets used in Membrane Framework tutorials.
boombox	Boombox is a simple streaming tool built on top of Membrane

Plugins

General purpose

Package	Description	Links
membrane_file_plugin	Membrane plugin for reading and writing to files
membrane_hackney_plugin	HTTP sink and source based on Hackney
membrane_scissors_plugin	Element for cutting off parts of the stream
membrane_tee_plugin	[Labs] Membrane plugin for splitting data from a single input to multiple outputs
membrane_funnel_plugin	[Labs] Membrane plugin for merging multiple input streams into a single output

AI

Package	Description	Links
membrane_whisper_plugin	Membrane plugin for integrating OpenAI's Whisper in audio processing pipelines
membrane_yolo_plugin	Membrane Plugin for applying YOLO object detection on raw video frames
membrane_gemini_plugin	A Membrane plugin for easy integration with the Gemini Live API, establishing a WebSocket connection for low-latency audio streaming.

Streaming protocols

Package	Description	Links
membrane_webrtc_plugin	Plugin for streaming via WebRTC
membrane_rtmp_plugin	RTMP server & client
membrane_ertmp_plugin	Membrane plugin for streaming as RTMP/ERTMP client
membrane_http_adaptive_stream_plugin	Plugin generating manifests for HLS
membrane_srt_plugin

Containers

Package	Description	Links
membrane_mp4_plugin	Utilities for MP4 container parsing and serialization and elements for muxing the stream to CMAF
membrane_mpegts_plugin	MPEG-TS muxer
membrane_matroska_plugin	Matroska muxer and demuxer
membrane_flv_plugin	Muxer and demuxer elements for FLV format
membrane_ivf_plugin	Plugin for converting video stream into IVF format

Audio codecs

Package	Description	Links
membrane_aac_plugin	AAC parser and complementary elements for AAC codec
membrane_aac_fdk_plugin	Membrane AAC decoder and encoder based on FDK library
membrane_flac_plugin	Parser for files in FLAC bitstream format
membrane_mp3_lame_plugin	Membrane MP3 encoder based on Lame
membrane_mp3_mad_plugin	Membrane MP3 decoder based on MAD.

Video codecs

Package	Description	Links
membrane_h26x_plugin	Membrane H.264 and H.265 parsers
membrane_h264_ffmpeg_plugin	Membrane H264 decoder and encoder based on FFmpeg and x264
membrane_vpx_plugin	Membrane plugin for decoding and encoding VP8 and VP9 streams
membrane_av1_plugin	Membrane plugin for decoding and encoding AV1 streams
membrane_abr_transcoder_plugin	ABR (adaptive bitrate) transcoder, that accepts an h.264 video and outputs multiple variants of it with different qualities.

Raw audio

Package	Description	Links
membrane_raw_audio_parser_plugin	Membrane element for parsing raw audio
membrane_portaudio_plugin	Raw audio retriever and player based on PortAudio
membrane_audio_mix_plugin	Plugin providing an element mixing raw audio frames.
membrane_audio_filler_plugin	Element for filling missing buffers in audio stream
membrane_ffmpeg_swresample_plugin	Plugin performing audio conversion, resampling and channel mixing, using SWResample module of FFmpeg library

Raw video

Package	Description	Links
membrane_raw_video_parser_plugin	Membrane plugin for parsing raw video streams
membrane_video_merger_plugin	Membrane raw video cutter, merger and cut & merge bin
membrane_smelter_plugin	Membrane plugin for video and audio mixing/compositing
membrane_camera_capture_plugin	A set of elements allowing for capturing local media such as camera or microphone
membrane_rpicam_plugin	Membrane rpicam plugin

External APIs

Package	Description	Links
membrane_aws_plugin	[Labs]
membrane_agora_plugin	Membrane Sink for Agora Server Gateway
membrane_element_gcloud_speech_to_text	[Labs] Membrane plugin providing speech recognition via Google Cloud Speech-to-Text API
membrane_element_ibm_speech_to_text	[Labs] Membrane plugin providing speech recognition via IBM Cloud Speech-to-Text service
membrane_s3_plugin	[Maintainer: YuzuTen] Membrane framework plugin to support S3 sources/destinations

Formats

Package	Description	Links
membrane_rtp_format	Real-time Transport Protocol format for Membrane Framework
membrane_cmaf_format	Membrane description for Common Media Application Format
membrane_matroska_format	Matroska Membrane format
membrane_mp4_format	MPEG-4 container Membrane format
membrane_raw_audio_format	Raw audio format definition for the Membrane Multimedia Framework

Standalone media libs

Package	Description	Links
ex_webrtc	[Maintainer: elixir-webrtc] An Elixir implementation of the W3C WebRTC API
ex_sdp	Parser and serializer for Session Description Protocol
ex_libnice	[Labs] Libnice-based Interactive Connectivity Establishment (ICE) protocol support for Elixir
ex_libsrtp	Elixir bindings for libsrtp
ex_m3u8	Elixir package for serializing and deserializing M3U8 manifests.

Utils

Package	Description	Links
unifex	Tool for generating interfaces between native C code and Elixir
bundlex	Multiplatform app bundler tool for Elixir
beamchmark	Elixir tool for benchmarking EVM performance
bunch	A bunch of helper functions, intended to make life easier
bunch_native	Native part of the Bunch package
shmex	Elixir bindings for shared memory

Authors

Membrane Framework is created by Software Mansion.

Since 2012 Software Mansion is a software agency with experience in building web and mobile apps as well as complex multimedia solutions. We are Core React Native Contributors and experts in live streaming and broadcasting technologies. We can help you build your next dream product – Hire us.

Copyright 2018, Software Mansion

Licensed under the Apache License, Version 2.0