Creating Calculators in Mediapipe: Beyond the Documentation
Prerequisite
https://google.github.io/mediapipe/framework_concepts/calculators.html
Before reading this article, it’s necessary to read MediaPipe’s official documentation on how calculators, graphs, and packets work. This article will build upon the basic knowledge from the documentation. Unfortunately, the documentation is sparse and the topics discussed here today can only be learned by rummaging through the already created examples. Since we’re working with MediaPipe, an intermediate understanding of C++ is also required.
Before you create your MediaPipe calculator
Before you begin, you must chart out the purpose of your calculator. A calculator that works well with MediaPipe’s system is a calculator that either can be used many times or is a block of non-parallelizable code.
Parallelizable code is important because MediaPipe is multi-threaded. Calculators are run every time all inputs are fulfilled (with an exception we will get into later). Calculators come in many different complexities, but there should be an obvious floor to your complexity. For example, a calculator with the sole purpose of addition will cost more CPU cycles through overhead than save by being a parallelizable calculator.
Bazel BUILD Files
It is heavily suggested that you modify existing projects before creating your own. Firstly, you should create your own folder in the /mediapipe/calculators/ directory. Add a file called “BUILD” and your C++ file here (referred to as “example.cc”). In your BUILD file you must add a reference to your C++ file as follows:
cc_library(
name = “example”,
srcs = [“example.cc”],
visibility = [“//visibility:public”],
deps = [
“//mediapipe/framework:calculator_framework”,
],
alwayslink = 1,
)
This is a basic Bazel definition. Notice that we have to add a dependency of the calculator_framework, which will be a required dependency for every calculator you create. We are done with the BUILD file for now. Next, we should reference our finished BUILD file from another build file further up the build tree. If you are creating your own MediaPipe project, go to /mediapipe/examples/desktop and create your folder there, adding a BUILD file inside of it. If you are modifying an existing project, go to the BUILD file in that project’s folder. You will need to find what code is used to compile your project. Generally, it will be:
bazel build -c opt — define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/FOLDER_NAME:CC_LIBRARY_NAME
Where FOLDER_NAME should be the folder in your /mediapipe/examples/desktop that you are modifying or creating. Now inside of that folder, go to your BUILD file. If you are modifying an already existing project, add the build file you made earlier as a dependency in the deps list as such:
deps = [
“//mediapipe/calculators/CALCULATOR_FOLDER_NAME:example”,
],
If you are creating a project, you will have to create a cc_library as follows:
cc_library(
name = “CC_LIBRARY_NAME”,
deps = [
“//mediapipe/calculators/CALCULATOR_FOLDER_NAME:example”,
],
)
Now put some nonsense code into your .cc file that you know cannot compile. If you see a compile error caused by that code, you did it right! Remove the nonsense and get ready for the next stage. If you manage to compile with nonsense in your .cc file, you made a mistake somewhere and should backtrack to find it.
Basic Defines in the Calculator
Now to comply with MediaPipe’s calculator framework. Firstly we want to go into the .cc file and add it as a header:
#include “mediapipe/framework/calculator_framework.h”
To make life easy, work in the namespace of MediaPipe by enclosing everything in this file within the MediaPipe namespace:
namespace mediapipe{
all_your_code
}
Let’s begin with a skeleton and go from there. We’ll call our calculator ExampleCalculator. We want to inherit from CalculatorBase and define the four functions mentioned in the MediaPipe documentation: GetContract, Open, Process, and Close. All four are expected to return a ::mediapipe:Status. After declaring the class, make sure to run the REGISTER_CALCULATOR function on the newly created calculator. In code, the skeleton calculator will look like this:
class ExampleCalculator: public CalculatorBase {
public:
ExampleCalculator(){};
~ExampleCalculator(){};
static ::mediapipe::Status GetContract(CalculatorContract* cc){
return ::mediapipe::OkStatus();
}
::mediapipe::Status Open(CalculatorContext* cc){
return ::mediapipe::OkStatus();
}
::mediapipe::Status Process(CalculatorContext* cc){
return ::mediapipe::OkStatus();
}
::mediapipe::Status Close(CalculatorContext* cc){
return ::mediapipe::OkStatus();
}
};
REGISTER_CALCULATOR(ExampleCalculator);
Don’t forget your header. Also note that if ::mediapipe::OkStatus() is not returned, execution will stop as MediaPipe will think it has encountered an error.
Declaring Inputs and Outputs (GetContract())
Here we have to define what inputs and outputs we are expecting. We have to keep in mind C++ types and .pbtxt types (more on that later). There are two types of input/outputs: explicitly tagged and indexed. We will begin with an explicitly tagged.
Explicitly Tagged Input/Outputs
When you know how many inputs your calculator needs per frame, explicitly tagged is the way to go. We simply need to declare our expected C++ type and a corresponding tag. The code is as follows:
cc->Inputs().Tag(“TAG_NAME”).Set<CLASS_TYPE>();
cc->Outputs().Tag(“OUTPUT_TAG_NAME”).Set<OUTPUT_CLASS_TYPE>();
The CLASS_TYPE is limited only by your included headers. Feel free to include vectors, custom classes, or just a standard Boolean.
Indexed Input/Outputs
When there could be anywhere from one to a dozen inputs to a calculator, then it’s time to use indexes. Now, don’t forget that MediaPipe requires all inputs to be fulfilled before running a calculator (with an exception that will be mentioned later in this article). In the .pbtxt file, you will have to explicitly declare your inputs/outputs.
Index Input/Outputs are useful when a calculator is used in multiple places with a different amount of inputs. For the most basic example, think of a multi-argument adder. In one part of your graph structure, it could have two inputs declared. Then the calculator can be reused later on in the graph structure with three, five, ten, or more inputs.
To make use of an indexed input/output, each index must be explicitly given a type as shown in the following code:
cc->Inputs().Index(0).Set<CLASS_TYPE>();
To work with an unspecified number of indexes, make use of the following for loop:
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
cc->Inputs().Index(i).Set<CLASS_TYPE>();
}
for (int i = 0; i < cc->Outputs().NumEntries(); ++i) {
cc->Outputs().Index(i).Set<CLASS_TYPE>();
}
Adding to your graph structure (.pbtxt)
No matter if the inputs/outputs are declared by index or explicitly tagged, they have to be explicitly mentioned in your .pbtxt file called a graph. Graphs are found in /mediapipe/graphs/FOLDER_NAME. If you are making your own project, make your own folder, and add an empty .pbtxt file. If you are working off an already made project, find the folder and the corresponding .pbtxt file, normally mentioned in the command to run the MediaPipe project as an argument.
If you are creating your own project, I have to direct you to the MediaPipe “Build your own Calculator” tutorial for this step. The gist of this is that you need to declare input_stream and output_stream at the top of your .pbtxt. The difficult part is in delivering those streams of data, which is beyond the scope of this tutorial.
Otherwise, if you are working on an already existing project, you already have the input_stream and output_stream taken care of. Instead, your job will be properly integrating your calculator into the already made system. The streams have two parts, the tag and the subtag with the given format: “TAG:subtag”. The Tag should match the expected tag in the .cc calculator. If you opted for an index system instead, then do not use a tag and only have the “subtag” as the stream.
First is an example of an explicitly tagged calculator:
node {
calculator: “ExampleCalculator”
input_stream: “TAG:subtag”
output_stream: “OUTPUT_TAG:subtag_two”
}
Or for a calculator using indexes:
node {
calculator: “ExampleCalculator”
input_stream: “subtag_one”
input_stream: “subtag_two”
output_stream: “OUTPUT_TAG:output”
}
Note that a calculator may have an index input with a tag output. The reverse is also possible, as is index to index.
One thing to make sure of is that your C++ types line up. If you declared “TAG:subtag” to be an int in your C++ file but “TAG:subtag” is declared as a double in a previous calculator, then you will run into a runtime error which will tell you that such an issue has happened.
Now compile your code. If there is an issue, backtrack. If it compiled, you are not out of the woods yet as .pbtxt issues are found at runtime. Run your code. The following is an example run command:
sudo bazel-bin/mediapipe/examples/desktop/FOLDER_NAME/CC_LIBRARY_NAME — calculator_graph_config_file=mediapipe/graphs/FOLDER_NAME/GRAPH_NAME.pbtxt
Do not get too worried if you get a runtime error. You most likely either made a typo or have a TAG:subtag with mismatched C++ types. Either way, I’ve found Mediapipe to be pretty good at describing graph caused errors.
Before moving on, make sure that if a calculator is expecting an output that it receives it. If a calculator is missing even one input, it will indefinitely wait for it to arrive. For your first calculators, I suggest not declaring an output_stream and just logging it with the built-in logging tool (included with the calculator framework header) as with the following example:
LOG(INFO) << data_stream;
Beyond Basics — Input Stream Handlers
As mentioned before, MediaPipe executes calculators when all inputs are fulfilled. However, to make more complex graphs, it may be necessary to run a calculator the second an input arrives. A use case could be to make a sort of “if command” within the graph. How this could be done is left as an exercise to the reader.
To make calculators run after receiving any input, add the input stream handler called ‘ImmediateInputStreamHandler’ to the .pbtxt file. The following is an example:
node {
calculator: “ExampleCalculator”
input_stream: “subtag_one”
input_stream: “subtag_two”
output_stream: “OUTPUT_TAG:output”
input_stream_handler {
input_stream_handler: ‘ImmediateInputStreamHandler’
}
}
One thing to look out for is that the calculator needs input timestamps to be monotonically increasing. Make sure that if an immediately running calculator is receiving input streams, that there is no chance it could receive a stream from an earlier time (even if it’s a millisecond).
Make sure to use the following code to prevent accessing null pointers by implementing them in if commands:
cc->Inputs().Index(i).IsEmpty()
Or
cc->Tag(“TAG_NAME”).IsEmpty()
Beyond Basics — Side Packets
Side packets are input/outputs that deliver constants. They are added to .pbtxt files as
input_side_packet: “test_packet”
Or
output_side_packet: “test_packet”
They are implemented in GetContract as follows:
cc->OutputSidePackets()
.Index(0)
.Set<CLASS_TYPE>();
They can either be indexed or explicitly tagged. The above example is indexed.
Beyond Basics — Options
Options allow for even further calculator reusability. They allow for constants to be declared in the .pbtxt file. As a basic example, think of a pow() calculator. Rather than making a pow_2() pow_3(), … pow_n() calculator we can have a single power calculator.
To add options to your calculator, you will have to start in your .cc directory. Create a file with a .proto extension, referred to as “example.proto”. The file must have the following code:
syntax = “proto2”;
package mediapipe;
import “mediapipe/framework/calculator.proto”;
message ExampleCalculatorOptions {
extend mediapipe.CalculatorOptions {
optional ExampleCalculatorOptions ext = 123456789;
}
optional bool option_parameter_1 = 1 [default = false];
optional float option_parameter_2 = 2;
}
The optional ExampleCalculatorOptions ext. value must be a unique value not shared by any other CalculatorOptions. For each option, “optional” can be swapped for “required” and the number of supported types is limited, as far as I know, they are [“string”, “float”, and “bool”]. When we say option_parameter_1 = 1 we are not setting the default value — that is done by [default=value]. It is just verbosity needed to make the CalculatorOptions system work. Make sure to increment the value of each parameter by one starting at one as seen in the above example.
Now that the .proto file is finished, we must add it to the BUILD file in the same directory.
Firstly do:
proto_library(
name = “example_calculator_proto”,
srcs = [“example.proto”],
visibility = [“//visibility:public”],
deps = [
“//mediapipe/framework:calculator_proto”,
],
)
Then try to compile. If successful then add:
mediapipe_cc_proto_library(
name = “example_calculator_cc_proto”,
srcs = [“example.proto”],
cc_deps = [“//mediapipe/framework:calculator_cc_proto”],
visibility = [“//visibility:public”],
deps = [“:example_calculator_proto”],
)
And try to compile again. Finally, go to where you declare your cc_library for the calculator you are trying to add options to and add your mediapipe_cc_proto_library as a dependency as shown:
cc_library(
name = “example”,
srcs = [“example.cc”],
visibility = [“//visibility:public”],
deps = [
“//mediapipe/framework:calculator_framework”,
“:example_calculator_cc_proto”,
],
alwayslink = 1,
)
Then go into your .cc file and include the following header:
#include “mediapipe/calculators/FOLDER_NAME/options_name_without_proto.pb.h”
Please, for the first few calculators, stick to the naming conventions used by this guide because I’ve had issues with veering off and trying to create my own.
To access your options inside the calculator, use the following:
const auto& options = cc->Options<::mediapipe::ExampleCalculatorOptions>();
auto var = options.option_parameter_1();
It’s recommended to access options in the open() script and save them as private variables in the calculator class.
Open and Close Functions
As mentioned in the Mediapipe documentation, these functions run when the calculator is opened and closed, which is generally done when the program is open and closed. Practical examples could include opening a file on the open function and closing it on the close function. There is not much much to say about these functions.
Process Function
When all the inputs are fulfilled (or just one if using ‘ImmediateInputStreamHandler’) the process function of the calculator is run. The input is enclosed within the cc pointer.
To access your input data, use the following code given index:
CLASS_TYPE last_data = cc->Inputs().Index(0).Get<CLASS_TYPE>();
Or given Tag:
CLASS_TYPE last_data = cc->Inputs().Tag(“TAG”).Get<CLASS_TYPE>();
Then do whatever you need to do with your given inputs.
To give an output, the data must first be a unique_pointer. If this is beyond the scope of your c++ experience, simply work with regular data types. When it’s time to create the default_pointer use the following code:
std::unique_ptr<CLASS_TYPE> output_stream = std::make_unique<CLASS_TYPE>(regular_data_of_class_type);T
To output the unique pointer, use the following code:
cc -> Outputs().Tag(“OUTPUT_TAG”).Add(output_stream.release(), cc->InputTimestamp());
By this point, the index alternative should be obvious. Simply replace .Tag() with .Index().
You should now have all the knowledge needed to create your own MediaPipe calculators.
Who am I and how did I learn Mediapipe?
I am Arian Alavi, an applied math student at the University of California, Santa Barbara. I’ve been working with mediapipe on and off for the past year to create a live ASL alphabet interpreter. Most of what I’ve learned about Mediapipe was learned through scouring their hand tracking example and modifying the code.
With the help of many of my friends, we’ve managed to deploy the ASL interpreter to the Android app store. To learn more about our project you may visit our GitHub page: www.github.com/AriAlavi/SigNN
If you have an Android phone you can download our app directly from the Play Store: https://play.google.com/store/apps/details?id=com.signn.mediapipe.apps.handtrackinggpu
