mediastreamer2. Applying the Lua Machine to Filters
Chapter 8 Applying the Lua Machine to Filters
Earlier we considered filters, the behavior of which, after the start, can be controlled only partially – by calling the methods provided in them. In this article, we will create a programmable filter, the behavior of which will be completely determined by the Lua machine built into it, or rather the script loaded into it. This will allow you to change the filter algorithm without recompiling the executable code.
The code of the programs of this chapter can be downloaded from Github at the link:
https://github.com/chetverovod/Mediastreamer2_LuaFilter
Let’s get down to practical implementation. To do this, you can recall how a new filter is created, see the chapter 4. In this scheme, the source of the audio signal can be either a signal from the line input of the sound card (sound_card_read) or DTMF signal generator (dtmf_generator). Further, the data enters the input of the developed Luafilter (lua_filter), which converts them in accordance with the script loaded into it. The data is then sent to the splitter (Tee) which forms two copies of the input stream, which produces two outputs. One of these streams goes to the registrar (recorder) and to the sound card for playback (sound_card_write). The registrar saves them to disk in the format raw (wav-header file). This way we can listen to and record the result of the Lua filter.
8.1 Filter implementation
Lua-filter will be arranged as follows. In the filter, when it is initialized, an instance of the Lua machine will be created. Moving data through the filter is shown in the figure 8.1.
The input data block will be passed to the Lua machine through its stack using two global variables:
lf_data – a long string containing the bytes of the data block;
lf_data_len – integer, string length lf_data in bytes.
After these two variables are pushed onto the stack, execution is transferred to the Lua machine. It will activate the loaded script and convert the data string received from the stack. The resulting block of data is again placed in a long string and placed in a global variable and then on the stack. For this, another pair of global variables is used:
lf_data_out – the string containing the bytes of the output data block;
lf_data_out_len – integer, string length lf_data_out in bytes.
The filter then pops the data from the stack and forms an output data block that is exposed to the filter output.
Thus, the use of a Lua filter makes it possible to change the processing algorithm and the behavior of the media stream transmission scheme without recompiling and stopping the main application, i.e. on the fly.
In fact, a text string of the appropriate size is used to transfer the block to the Lua machine and back. More sophisticated options for data exchange can be implemented using types[User‑Defined Types]( https://www.lua.org/pil/28.html)their implementation is beyond the scope of this book.
Since it is known that the block of input data consists of 16-bit signed samples, then in the string received by the script, each sample will be encoded by two bytes in two’s complement code.
8.2 Organization of the script
Initially, the filter does not contain an executable Lua script, it must be loaded there. In this case, it is advisable to divide the script into two parts. The first part does the initialization. This part of the script is executed once. The second part is intended for multiple, cyclic execution, for each ticker tick. Let’s call these parts of the script as:
script preamble – executed once upon receipt of the first tick from the ticker.
script body – this part is launched for execution on each ticker tick (10 ms).
To load the preamble and the body, the filter provides the LUA_FILTER_SET_PREAMBLE and LUA_FILTER_RUN methods, respectively.
8.2.1 Script Preamble
This is a one-time executable Lua code that includes libraries, declarations of functions (used by the script body) and the necessary initial initialization of variables. Because the preamble starts on the first tick of the ticker, we need to define its code before the ticker is fired. To do this, the LUA_FILTER_SET_PREAMBLE method will be used, to which the already familiar pointer to the filter structure is passed as the first argument, and the preamble text as the second argument. The preamble can be written/rewritten to the filter repeatedly and at any time. Each rewriting of the preamble is followed by its one-time execution on the next cycle.
8.2.2 Script body
Unlike the preamble, this part of the script is executed on every ticker tick (every 10ms by default). This part of the code can be written/rewritten to the filter at any time. To do this, the LUA_FILTER_RUN method will be defined, the arguments are similar to the preamble.
8.2.3 Stopping a script
At any time, the script can be stopped (paused) using the LUA_FILTER_STOP method. After calling this method, incoming data blocks are transferred immediately to the filter output, bypassing script processing. You can resume processing by calling the LUA_FILTER_RUN method. Substituting a null pointer or a pointer to a new text instead of a pointer to the script body text.
8.2.4 Additional functions
In order for the script to extract data from the string and put the result of its work back into the string, we need two data access functions get_sample() And append_sample(). The first one extracts the signal sample from the string. With the help of the second, you can add a countdown to a long line. Their text is given in the listing. 8.1.
Listing 8.1: Data Access Functions
-- funcs.lua
-- Функция извлекает из строки один 16-битный отсчет.
function get_sample(s, sample_index)
local byte_index = 2*sample_index - 1
local L = string.byte(s, byte_index)
local H = string.byte(s, byte_index + 1)
local v = 256 * H + L
if (H >= 128) then
v = - ((~(v - 1)) & 65535)
end
return v
end
-- Функция добавляет в строку один 16-битный отсчет.
function append_sample(s, sample_value)
local v = math.floor(sample_value + 0.5)
if v < 0 then
v = - v
v = ((~v) + 1) & 65535
end
local H = v // 256
local L = v - H * 256
return s .. string.char(L, H)
end
Function get_sample() is used to access signal samples stored in a row. It returns one 16-bit sample extracted from the string s. According to the given reference index sample_index() the indexes of the 2 bytes in which it is stored are determined. Further, a 16-bit number is assembled from these two bytes. Since the count is a signed number, these bytes store the number in two’s complement, we need to convert the number to its usual form. First, we determine the state of the 15th bit of the number. If it is equal to 0, then the number is positive and no additional transformations are needed. If the bit is set, then the number is negative. Then we subtract one from the number and do the inversion of each bit and multiply by -1.
Function append_sample() used to assemble a string with output data. It adds two bytes to its first argument (string), representing the second argument (signal count) in two’s complement.
The function file will be called funcs.lua it must be placed in the directory where the filter code will run.
8.2.5 Sample script
Let’s create a script that will simply forward the data through itself without changing them.
The preamble is shown in the listing 8.2:
Listing 8.2: Script Preamble
-- preambula2.lua
-- Этот скрипт выполняется в Lua-фильтре как преамбула.
-- Подключаем файл с функциями доступа к данным.
require "funcs"
preambula_status = 0
body_status = 0 -- Эта переменная будет инкрементироваться в теле скрипта.
local greetings="Hello world from preambula!\n" -- Приветствие.
print(greetings)
return preambula_status In the preamble, we include a file with signal data access functions (funcs.lua).
The body of the script is shown in the listing 8.3:
Listing 8.3: Script body
-- body2.lua
-- Этот скрипт выполняемый в Lua-фильтре как тело скрипта.
-- Перекладываем результат работы в выходные переменные.
lf_data_out =""
if lf_data_len == nil then
print("Bad lf_data_len.\n")
end
for i = 1, lf_data_len/2 do
s = get_sample(lf_data, i)
lf_data_out = append_sample(lf_data_out, s)
end
lf_data_out_len = string.len(lf_data_out)
return body_status
Nothing special is done in the body of the script, just the samples from the input string are transferred one by one to the output string. It is obvious that here between the functions get_sample() And append_sample() any transformation of samples can be arranged. Another option is also possible, when the filter does not process samples, but can control other filters in accordance with the input data.
It should be noted that when writing scripts, it is convenient to put a comment containing the file name in the first line of the file, as is done in the examples: then if an error occurs in the diagnostic message, the first place will be the name of the file in which the error was detected: and it will become clear to you which part of the script is meant.
-- preambula2.lua Filter <LUA_FILTER> Lua error. Lua error description:<[string "-- preambula2.lua ..."]:12: attempt to perform arithmetic on a nil value>.8.3 Filter source code
The filter header file will look like this:
Listing 8.4: Lua filter header file
#ifndef lua_filter_h
#define lua_filter_h
/* Подключаем заголовочный файл с перечислением фильтров медиастримера. */
#include <mediastreamer2/msfilter.h>
/* Подключаем интерпретатор Lua. */
#include <lua5.3/lua.h>
#include <lua5.3/lauxlib.h>
#include <lua5.3/lualib.h>
/*
Задаем числовой идентификатор нового типа фильтра. Это число не должно
совпадать ни с одним из других типов. В медиастримере в файле allfilters.h
есть соответствующее перечисление enum MSFilterId. К сожалению, непонятно
как определить максимальное занятое значение, кроме как заглянуть в этот
файл. Но мы возьмем в качестве id для нашего фильтра заведомо большее
значение: 4001. Будем полагать, что разработчики добавляя новые фильтры, не
скоро доберутся до этого номера.
*/
#define LUA_FILTER_ID 4001
/* Имя глобальной переменной, в которую функция фильтра помещает блок входных
данных. */
#define LF_DATA_CONST "lf_data"
/* Имя глобальной переменной, в которую функция фильтра помещает размер блока входных
данных.*/
#define LF_DATA_LEN_CONST "lf_data_len"
/* Имя глобальной переменной, в которую функция фильтра помещает блок выходных
данных.*/
#define LF_DATA_OUT_CONST "lf_data_out"
/* Имя глобальной переменной, в которую функция фильтра помещает размер блока выходных
данных.*/
#define LF_DATA_OUT_LEN_CONST "lf_data_out_len"
/* Флаг того, что входная очередь фильтра пуста. */
#define LF_INPUT_EMPTY_CONST "input_empty"
/* Определяем константы фильтра. */
#define LF_DATA LF_DATA_CONST
#define LF_DATA_LEN LF_DATA_LEN_CONST
#define LF_DATA_OUT LF_DATA_OUT_CONST
#define LF_DATA_OUT_LEN LF_DATA_OUT_LEN_CONST
#define LF_INPUT_EMPTY LF_INPUT_EMPTY_CONST
/*
Определяем методы нашего фильтра. Вторым параметром макроса должен
порядковый номер метода, число от 0. Третий параметр это тип аргумента
метода, указатель на который будет передаваться методу при вызове.
*/
#define LUA_FILTER_RUN MS_FILTER_METHOD(LUA_FILTER_ID,0,char)
#define LUA_FILTER_STOP MS_FILTER_METHOD(LUA_FILTER_ID,1,int)
#define LUA_FILTER_SET_PREAMBLE MS_FILTER_METHOD(LUA_FILTER_ID,2,char)
/* Определяем экспортируемую переменную, которая будет
хранить характеристики для данного типа фильтров. */
extern MSFilterDesc lua_filter_desc;
#endif /* lua_filter_h */Here macros are created with the names of global variables in the context of the Lua machine and the three filter methods mentioned above are declared:
LUA_FILTER_RUN;
LUA_FILTER_STOP;
LUA_FILTER_SET_PREAMBLE.
We will consider the source code of the filter only in its important part, i.e. method work control_process() (the source code is given in full in appendix A). This method runs the Lua machine on each ticker tick. Its text is shown in the listing. 8.5.
Listing 8.5: The control_process() method
static void
control_process(MSFilter *f)
{
ControlData *d = (ControlData *)f->data;
mblk_t *im;
mblk_t *out_im = NULL;
int err = 0;
int i;
if ((!d->stopped) && (!d->preabmle_was_run))
{
run_preambula(f);
}
while ((im = ms_queue_get(f->inputs[0])) != NULL)
{
unsigned int disabled_out = 0;
if ((!d->stopped) && (d->script_code) && (d->preabmle_was_run))
{
bool_t input_empty = ms_queue_empty(f->inputs[0]);
lua_pushinteger(d->L, (lua_Integer)input_empty);
lua_setglobal(d->L, LF_INPUT_EMPTY);
/* Кладем блок данных со входа фильтра на стек Lua-машины. */
size_t sz = 2 * (size_t)msgdsize(im); /* Размер блока в байтах.*/
lua_pushinteger(d->L, (lua_Integer)sz);
lua_setglobal(d->L, LF_DATA_LEN);
lua_pushlstring(d->L, (const char *)im->b_rptr, sz);
lua_setglobal(d->L, LF_DATA);
/* Удаляем со стека все, что там, возможно, осталось. */
int values_on_stack;
values_on_stack = lua_gettop(d->L);
lua_pop(d->L, values_on_stack);
/* Выполняем тело скрипта. */
err = luaL_dostring(d->L, d->script_code);
/* Обрабатываем результат выполнения. */
if (!err)
{
int script_body_status = lua_tointeger(d->L, lua_gettop(d->L));
if (script_body_status < 0)
{
printf("\nFilter <%s> bad script_body_status: %i.\n", f->desc->name,
script_body_status);
}
/* Извлекаем размер выходного блока данных, возможно он изменился. */
lua_getglobal(d->L, LF_DATA_OUT_LEN);
size_t real_size = 0;
char type_on_top = lua_type(d->L, lua_gettop(d->L));
// printf("Type on top: %i\n", type_on_top);
if (type_on_top == LUA_TNUMBER)
{
real_size =
(size_t)lua_tointeger(d->L, lua_gettop(d->L));
// printf("------- size from lua %lu\n", real_size);
}
lua_pop(d->L, 1);
/* Извлекаем длинную строку с преобразованными данными входного блока
данных. И пробрасываем его далее. */
lua_getglobal(d->L, LF_DATA_OUT);
size_t str_len = 0;
if (lua_type(d->L, lua_gettop(d->L)) == LUA_TSTRING)
{
const char *msg_body = lua_tolstring(d->L, -1, &str_len);
if (msg_body && str_len)
{
size_t msg_len = real_size / 2;
out_im = allocb((int)msg_len, 0);
memcpy(out_im->b_wptr, msg_body, msg_len);
out_im->b_wptr = out_im->b_wptr + msg_len;
}
}
lua_pop(d->L, 1);
/* Вычитываем и отбрасываем все, что возможно осталось на стеке. */
values_on_stack = lua_gettop(d->L);
lua_pop(d->L, values_on_stack);
}
else
{
printf("\nFilter <%s> Lua error.\n", f->desc->name);
const char *answer = lua_tostring(d->L, lua_gettop(d->L));
if (answer)
{
printf("Lua error description:<%s>.\n", answer);
}
}
}
mblk_t *p = im;
if (out_im)
p = out_im;
for (i = 0; i < f->desc->noutputs; i++)
{
if ((!disabled_out) && (f->outputs[i] != NULL))
if (p)
ms_queue_put(f->outputs[i], dupmsg(p));
}
freemsg(out_im);
freemsg(im);
}
}When the method control_process() receives control, it checks if there is data at the filter input and sets the global variable LF_INPUT_EMPTY so that, if necessary, the script can handle the situation when there is no input data. Then, if there is data at the input, as in any other filter, it starts to subtract them. Each block, by default, has a size of 160 samples or 320 bytes, however the block size is determined. The result is pushed onto the stack of the Lua machine, and from it into a global Lua variable lf_data_len (ceoe). After that, the method puts the data block itself on the stack, and from it to the global variable lf_data (long line). Next, control is transferred to the Lua machine, this is done by calling the function:
luaL_dostring(d->L, d->script_code)the machine starts to execute the script loaded earlier. After the script is executed, the process of transferring the results of the script from the context of the Lua machine to the method will take place.
8.4 Test application
Once the Lua filter is implemented, it’s time to create a test application for it. To test the developed filter, we apply the scheme shown in the figure 8.2.
Figure 8.2: Schematic for testing a Lua filter
The algorithm of the application will be as follows. After starting, when the RAM already contains a circuit of filters, but the clock sources are not activated, the filter initialization procedure will begin. It consists in the fact that each filter runs its method init(). The created filter will not be an exception, but in addition to the mandatory actions, in init() it will execute the Lua preamble code that performs the initial setup of the Lua machine. When starting the program, we will have to pass it the path to the preamble file using the command line switch “scp“The other part, the body of the script, is passed with the key”scb“. A complete list of program keys is given in the listing 8.6.
Code Listing 8.6: Test Application Command Line Options
--help List of options.
--version Version of application.
--scp Full name of containing preambula of Lua-script file.
--scb Full name of containing body of Lua-script file.
--gen Set generator's frequency, Hz.
--rec Make recording to a file 'record.wav'.
The source code of the test application is given in Appendix B. Earlier, at the beginning of the chapter, a link was given where you can download this source code and according to the instructions in the file
README.mdperform assembly.
An example of running a test application:
$ ./lua_filter_demo --scb ../scripts/body2.lua --scp ../scripts/preambula2.lua --gen 600 After starting, a 600 Hz tone will be heard in the headphones for 10 seconds. This means that the signal has passed through the filter.
8.5 Filter example
As an example, let’s write a script that, starting from the 5000th sample (ie 5/8 of a second), will multiply the input signal by a low frequency sinusoidal signal (ie modulate in amplitude) for 2 seconds. The signal will then become unmodulated again.
Listing 8.7: Modulator script preamble
-- preambula3.lua
-- Этот скрипт выполняется в Lua-фильтре как преамбула.
-- Подключаем файл с функциями доступа к данным.
require "funcs"
preambula_status = 0
body_status = 0 -- Эта переменная будет инкрементироваться в теле скрипта.
-- Переменные для расчетов.
samples_count = 0
sampling_rate = 8000
low_frequency = 2 -- Модулирующая частота.
phase_step = 2 * math.pi / sampling_rate * low_frequency
return preambula_status Modulation will be implemented as follows:
Listing 8.8: Modulator script body
-- body3.lua
-- Это скрипт выполняемый в Lua-фильтре как тело скрипта.
-- Скрипт выполняет модуляцию входного сигнала.
lf_data_out =""
if lf_data_len == nil then
print("Bad lf_data_len.\n")
end
for i = 1, lf_data_len/2 do
s = get_sample(lf_data, i)
if (samples_count > 5000) and (samples_count < 21000) then
output_s = s * math.sin( phase_step * samples_count )
else
output_s = s
end
samples_count = samples_count + 1
lf_data_out = append_sample(lf_data_out, output_s)
end
lf_data_out_len = string.len(lf_data_out)
return body_status
We start the application with a new script:
$ ./lua_filter_demo --scb ../scripts/body3.lua --scp ../scripts/preambula3.lua --gen 1200 --rec after 5 seconds, to stop the program, press the “Enter“. You can play the file, just as we did earlier in 3.8:
$ aplay -t raw --format s16_be --channels 1 ./record.rawLet’s convert the output file to wav-format:
$ sox -t raw -r 8000 -b 16 -c 1 -L -e signed-integer ./record.raw ./recording.wav To draw a signal in gnuplot, we need to convert it to a file with two columns of data. The same tool will do it sox paired with grep:
$ sox ./recording.wav -r 8000 recording.dat && grep -v «^;» recording.dat > clean_recording.datNext, we transfer the resulting file recording.wav gnuplot utility:
$ gnuplot -e "set terminal png; set output 'recording.png'; plot 'clean_recording.dat' using 1:2 with lines"On the image 8.3 shows the result of the script.
In the figure, we see the envelope of a sinusoidal signal after it has passed through the filter. For about 5/8 of a second, the signal amplitude remained unchanged, then the script branch with the modulation algorithm was put into operation:
output_s = s * math.sin( phase_step * samples_count)and for a second there was a modulated signal at the output. After that, the script turned off the modulation and the signal began to be transmitted to the output without modulation.
