Fault-tolerant VoIP server in the cloud

We at Axmor have implemented more than a dozen VoIP projects, for example, the following:

Unified communications system.
Routing telephone calls.

We would like to summarize our experience and share it with you. This article is intended for specialists who want to gain knowledge about the architecture of fault-tolerant VoIP systems. Basic knowledge of FreeSWITCH, Asterisk or analogues is desirable, but not required. In this article we will give examples of real configurations for organizing a cluster.

Introduction to VoIP

If the words FreeSWITCH, Asterisk, SIP, RTP, WebRTC are not empty words for you, you can safely skip this part.

The VoIP world stands on two big elephants: SIP and RTP. These 2 protocols were developed at the end of the last century and came to us from the world of telephony. SIP - command protocol. Responsible for selecting codecs, starting/ending a call, managing a call (hold/transfer/etc) and has a huge number of extensions, including for sending text messages, notification of left voice messages, etc.

IP phones from different manufacturers may support their own set of extensions. For example, a very common BLF (Busy Lamp Field) is a field of LEDs on a desk phone that can be configured to indicate the status of another employee's phone.

RTP - media protocol. Responsible for transmitting audio and video data. It uses various codecs internally to encode media data.

Both protocols are implemented on top of UDP by default. Almost everyone has TCP as an option, but UDP is better for a variety of reasons. Encryption is also available for both protocols.

The VoIP architecture is based on the idea that the SIP and RTP server are two different servers, so there are server implementations that support only SIP or only RTP. However, FreeSWITCH and Asterisk are open source servers that support both protocols and much more. The choice between them largely depends on personal preferences, requirements and integration tasks, but both allow you to get an office PBX out of the box.

One cannot fail to mention WebRTC. This is the de facto standard for voice calls on the web. WebRTC uses SRTP (secure RTP) under the hood, leaving the implementation of the command layer to the JS code. For p2p calls, we just need to transfer our address and some call parameters to the other party, which can be done based on any protocol. If we need integration with a SIP server, then the SIP over WebSocket protocol is usually used. Implementation - sipjs.com.

FreeSWITCH supports both SIP over WebSocket and its own alternative protocol implemented by the mod_vertoo module. It is designed specifically for WebRTC integration and solves the annoying WebRTC and SIP incompatibility issues that result in 1-5 second delays before a call is created. However, these problems can be solved in another way, on the JS side, but this is a topic for a separate article.

As with regular analog calls, VoIP requires a PBX so that users can find each other using a certain phone number and to circumvent the problem of clients not having white IPs. Once the clients have found each other and agreed on codecs, a connection must be established to stream the media. Depending on the project requirements and network configuration, the data flow can go directly from client to client, or through the server.

The VoIP world tends to favor forwarding traffic through a server. This solves the problem of clients not having white IPs and allows them to centrally eavesdrop/record any conversation.

WebRTC by default offers a direct connection between clients, but this does not work in the case of a symmetric firewall when both clients do not have a white IP. In such cases, we need a STUN/TURN media proxy server. For commercial projects, you will have to install and pay for your own proxy server in order for your WebRTC based application to work. Browsers only provide STUN server services for free - recognizing an external IP client, but proxying traffic (TURN) is already paid.

Note:

There is a technology that, in theory, allows two clients to establish a connection without white IPs with a little help from the server, but in our projects there was no such need. This technology is called “symmetric NAT”.

According to the documentation, Freeswitch supports symmetric NAT. But we recommend testing it thoroughly before using it in production.

Setting up interface card support

If you plan to connect the Asterisk using special interface cards to regular telephone networks, you should ensure that the appropriate drivers are available, implemented as a kernel module. But even if there are no such devices on your computer, it is still recommended to install these drivers. The fact is that all Zaptel devices have a timer, and for the full operation of the IP telephony server it is necessary. But if you don’t have a Zaptel device at hand, you can use a special driver - ztdummy - to emulate it.

From the repository we install the zaptel, zaptel-source packages and assemble modules for our system:

$ sudo apt-get install zaptel zaptel-source $ sudo module-assistant prepare $ sudo ma -t build zaptel

The zaptel-modules-*_i386.deb package will appear in /usr/src, install it using dpkg. After this, we check the operation of the kernel modules:

$ sudo depmod -a $ sudo modprobe ztdummy

And if you need device support:

$ sudo modprobe zaptel $ sudo modprobe wcfxo

To ensure their automatic loading, run the following command:

$ echo 'ztdummy\nzaptel\nwcfxo' >> /etc/modules

Create rules for UDEV:

$ sudo mcedit /etc/udev/rules.d/51-zaptel.rules

KERNEL=”zapctl”, NAME=”zap/ctl” KERNEL=”zaptimer”, NAME=”zap/timer” KERNEL=”zapchannel”, NAME=”zap/channel” KERNEL=”zappseudo”, NAME=”zap/ pseudo" KERNEL="zap0-9*", NAME="zap/%n"

You can also use the source code or the CVS version of the driver. If you compile yourself, you will need the kernel header files (or source code):

$ sudo apt-get install linux-headers-`uname -r`

Let's create a symbolic link so that Asterisk can find the kernel sources:

$ sudo ln -s /usr/src/linux-headers-2.6.20-15-generic /usr/src/linux-2.6

Now we get the latest version of the drivers:

$ cd /usr/src $ wget -c downloads.digium.com/pub/zaptel/zaptel-1.4-current.tar.gz

Compile and install:

$ tar xzvf zaptel-1.4-current.tar.gz $ cd /usr/src/zaptel-1.2.17.1 $ ./configure $ make $ sudo make install

And in order not to manually create configuration files:

$ sudo make config

After this command, a script will be created to automatically launch the modules included in Zaptel, and a config /etc/default/zaptel (or /etc/sysconfig/zaptel) will be created, which will indicate which modules need to be loaded. I recommend leaving only what is necessary in this file. Let's try to load the module:

$ sudo modprobe ztdummy $ lsmod | grep ztdummy ztdummy 6184 0 zaptel 189860 1 ztdummy

Everything is fine. After installation, two more files will appear on the system:

/etc/zaptel.conf – describes the hardware configuration;
/etc/Asterisk/zapata.conf — Asterisk for the Zap channel driver.

Detailed instructions for all types of devices are given in the documentation. In Russian, you can read about this in the document “Zaptel Kernel Driver Configuration”. But we don’t stop there, we still have a lot of work ahead. After configuration, we check the operation with the command ztcfg -vv.

Scaling SIP traffic using OpenSIPs

FreeSWITCH and Asterisk allow you to process a huge number of simultaneous calls on one machine - 500-1000. But what if we need more? Or if the technical specifications immediately include the need for a cloud with load balancing?

This is where a SIP load balancer comes to the rescue. We allocate one machine for a balancer, which selects an end node at the SIP level and redirects media traffic processing there. RTP media traffic does not pass through the balancer, so it can handle a significantly larger number of simultaneous calls. In fact, once the call is created, the load disappears.

The choice is usually between OpenSIPs and Kamailio. Both projects have a common ancestor and a similar module structure. We will leave the comparison and selection of a specific solution for a separate article, but for now we will talk about OpenSIPs.

Many have already worked with FreeSWITCH and/or Asterisk; there are plenty of how-to examples on the web, but setting up a SIP balancer in OpenSIPs or Kamailio is a completely different matter.

FreeSWITCH and Asterisk only require you to specify a list of users and fairly simple call routing rules. Moreover, there are free admin panels that will allow you to configure everything in the web interface.

In contrast, OpenSIPs and Kamailio force the user to understand the SIP protocol at least at the basic level and manually specify what to do with each SIP message.

OpenSIPs configuration is a program in a top-level pseudo-language, which should link 10+ different modules into a single system. The list of modules is much larger, but some duplicate each other, and some may not be needed.

In the program, you specify how to process each incoming SIP message and what to send in response. However, the main work is entrusted to the plug-in modules, and you just need to configure a large switch, when and which module to call.

If you want to add a lot of custom logic, then at some point it may be easier to move the business logic to FreeSWITCH (Asterisk) and/or a separate server, which will manage call routing, queues, etc.

The simplest load balancing script looks like this:

If the incoming SIP message is not the first, and we already know where to route it, we transfer control to the dialog module.
Otherwise, we call the balancer module to search for a less loaded node, put its address in target and transfer control to the dialog module.
Handling errors:
The balancer module may return an error and say that there are no free nodes.

Very important point:

When you write processing programs for OpenSIPs, remember that you must process both the messages coming from outside into your cloud and the message coming from your cloud to the outside. Therefore, the program often breaks into 2-3 huge If blocks, depending on the direction of the SIP packet.

Example:

For brevity we have cut out all error handling, the full version can be found here.

route { if (!has_totag()) { #First message in the dialogue, remember it in the database and don’t exit, we need to find where to route it record_route(); } else { #Not the first message, we route along the already beaten path and exit loose_route(); t_relay(); exit; } // Looking for a free node. 1 — the weight of our request, call resource. // You can make complex logic and allocate different sets of machines for different calls load_balance(“1″,”call”); if ($retcode

Features of OpenSIPs:

Header values do not change during program execution. If you assign something and then try to output it to the log, the original value will be output.
There are 2 different types of user variables and they are not compatible. Some modules use one, some use another.
Many functions do not return anything, but simply do something useful, and what exactly can be found out only by completely reading the documentation and sometimes the source code. For example, load_balance() sets the value of the $du variable. And record_route() stores $du in the message header, but is called before load_balance(). A clear violation of the causal relationship. We assume that the headers are counted at the very end, and record_route() rather puts down some kind of flag or link.
Stateless paradigm. SIP allows you to store all the data necessary for routing directly in message headers. The client is obliged to copy them when responding, which allows the server to be stateless. However, if necessary, you can save the routing table in the database and not notify the whole world about the internal structure of your cluster, but this adds load to the database and delays. The choice is yours.

Installing Asterisk

Unfortunately, it is impossible to give clear instructions about the hardware - there are too many subtleties and nuances, so for approximate computer configurations I refer you to the page of the voip.rus.net website “Performance of Asterisk systems”. If your goal is to get acquainted with Asterisk , you can use one of the distributions that already has a configured and ready-to-use server: Asterisk Now, Trixbox, VoIPonCD.

Asterisk is present in the package repositories of most distributions. So, in Ubuntu, the sudo apt-cache search Asterisk produces a decent list of packages, after installing which you can immediately begin configuration. of Asterisk in it is significantly behind the current one, which can be downloaded from the official website. Therefore, we will consider a universal installation method using the example of Ubuntu, although everything said (with rare exceptions) also applies to other distributions.

Install the packages required for compilation:

$ sudo apt-get install build-essential automake autoconf bison flex libtool libncurses5-dev libssl-dev

Additionally, it is highly recommended to install libpri even if you do not need Primary Rate ISDN support. This can be done either through the repository: sudo apt-get install libpri1.2, or using the sources:

$ wget -c downloads.digium.com/pub/libpri/libpri-1.4-current.tar.gz

Compilation of the library is standard, so we won’t dwell on this.

the Asterisk source code from the website and configure:

$ wget -c downloads.digium.com/pub/Asterisk/Asterisk-1.4.11.tar.gz $ tar xzvf Asterisk-1.4.11.tar.gz $ cd Asterisk-1.4.11 $ ./configure --prefix=/ usr

At the end of the script, we will see the project logo and some information about the settings in the console.

$ make $ sudo make install

Note: if you are installing version 1.2, then to support the mp3 format you should enter “make mpg123” before the make command; version 1.4 no longer responds to this command.

After compilation, among other things, the following executable files will be installed:

/usr/sbin/Asterisk - the Asterisk , which provides all the work;
/usr/sbin/safe_Asterisk - script for starting, restarting and checking the operation of the Asterisk ;
/usr/sbin/astgenkey – a script for creating private and public RSA keys in PEM format, which are necessary for Asterisk .

To install configuration file templates and documentation, type:

$ sudo make samples

Example configuration files will be copied to /etc/ Asterisk . If there are already configuration files in this directory, they will be renamed with the ".old" prefix. To build the documentation you will need the doxygen package; if it is not there, install it:

$ sudo apt-get install doxygen $ sudo make progdocs

We install the package with Asterisk -addons extensions in the same way (this step is optional, you can safely skip it). Many of the modules included in this set are experimental. They should be installed only if you need to record information in the database, support mp3 files and the ooh323c protocol (Objective Systems Open H.323 for C):

$ wget -c downloads.digium.com/pub/Asterisk/Asterisk-addons-1.4.2.tar.gz $ tar xzvf Asterisk-addons-1.4.2.tar.gz $ cd Asterisk-addons-1.4.2 $ . /configure; make; sudo make install; sudo make samples

Asterisk installation is complete. It is recommended to first start the server in debug mode and look at the output for errors:

$ sudo /usr/sbin/Asterisk -vvvgc

If we receive the “ Asterisk Ready” message and a management console prompt, then everything is in order. We leave:

*CLI> stop now

Now you can proceed to further configuration.

Fault tolerance and scaling of VoIP systems

Fault tolerance and scaling are related things. It is impossible to think through a fault tolerance architecture without taking into account the scaling scheme. But you can only implement fault tolerance in its simplest form. First, a little theory.

At the protocol level, VoIP supports the ability to restore a call even if the server fails.

SIP failure. By default, SIP uses the UDP protocol, which, unlike TCP, does not establish a persistent connection. But the client re-registers approximately every minute (configurable). In fact, every packet from a client can be answered by a new server. The only limitation is that the IP address of the response packet should not change, and we need to store the state of the user’s calls (if any at the moment) somewhere so that any server can work with this data.
RTP failure. If for some reason our RTP server, which also uses the UDP protocol, fails, we cannot easily restore the media stream, but we can send a command to recreate the stream (reinvite) via the SIP protocol. Ideally, the client will only notice a few seconds of audio loss. The implementation of fall detection remains questionable. There is no solution out of the box.

Let's consider different architecture options.

Need a VoIP project? Write to us and we will evaluate it for free.

Cancellation of the service

You can cancel the service at any time. In this case, your device registrations will be deleted and the SIP telephony server will be stopped.

Despite the development of various information exchange systems, such as e-mail and instant messaging services, the regular telephone will remain the most popular means of communication for a long time. A key event in the history of telecommunications and the Internet was the advent of voice over IP networks, so the very concept of a telephone has changed in recent years. The use of VoIP is modern, convenient, and cheap, since you can combine remote offices without even resorting to the services of telephone operators. What other reasons are needed to set up your IP telephony server?

SIP server reservation

Implementation details vary greatly depending on the server used, only one thing does not change - the need for a Virtual (floating) IP, which we can dynamically reassign to another server.

The problem is solved at the level of shell scripts, with which we check whether the server is running, and if not, we reassign the IP to another one. The main problem is the balance between false positives and long waits before switching.

But that's not all. The SIP server stores information about connected clients, and in order not to lose it, we must configure our server to save this information to a distributed database. Then, in case of a restart or switch to slave, we will not lose any data. Almost all popular SIP servers support the use of databases out of the box.

Scaling SIP servers

Properly implemented scaling also solves the problem of fault tolerance. One server died - let's take another. The main problem is that the SIP client ignores packets sent from another IP. Example “How it won’t work”:

Client B will simply ignore the incoming call (invite) because its source IP is different from the server on which the client is registered.

How will it work? Depending on the specific project, the available hardware and our capabilities, the problem can be solved in 3 ways.

The best softphone for the best subscribers

Subscribers to the site will not have to be tormented by the dilemma of choice: there is a universal and at the same time simple program available to all users - an IP telephone server - YouMagic Softphone. In addition to the obvious advantages of working with the provider itself, the subscriber will receive the following “bonuses”:

virtual PBX with protection against flooding, spam and DDoS attacks on central nodes, which guarantees comfortable communication without interruptions;
The technical support service will answer any question in detail, and in case of problems, it will promptly solve them;
Each softphone user will be able to use several accounts and financial calculators for each of them, thereby automating the accounting of traffic expenses.

These and many other features make use as comfortable as possible on any platform, including Windows, Android and other OS.
Tags:

IP address spoofing of packets

The simplest option is to install an even simpler (and faster) load balancer, which will replace the source IP with your own for all outgoing packets. And hide all SIP servers behind it. But we need to configure all SIP servers to use a common database.

An obvious advantage is the ease of setup.

Obvious disadvantages:

Single Poing of Failure as a load balancer.
The load balancer may not have enough bandwidth.

Forwarding the Invite team

We set up a farm of SIP servers with different IP addresses, DNS in round robin mode. Clients are randomly assigned to one of many servers.

We program the SIP server to forward invite to the desired server (on which another client is registered), so that it can forward the command to the client on its own behalf.

Plus, everything scales remarkably well.

A small minus - SIP register scales really well, but SIP invite and other call-related commands scale a little worse, because they are likely to pass through 2 nodes. However, register happens much more often.

Sharding

If our application is a SaaS for business, then most likely clients of different companies do not need to call each other, and with the help of a well-chosen hash function we can scatter many of our customers across different SIP servers.

We don't even need to have a common SIP registration database. But you need to somehow tell the SIP client the address of the required SIP server and figure out what to do if it dies (probably switch to another one). If we are talking about a UC solution, then, as a rule, the SIP client is built into the business application and receives all the settings from the server - there is no problem.

It's worse if we use a purely SIP client. Then it makes sense to solve the problem at the DNS and/or routing level. For example, give each company its own VoIP server address and dynamically change DNS tables.

Internet telephony capabilities

Virtual numbers. One subscriber can have several numbers, which are called virtual. For example, three numbers - for Moscow, the Urals and the USA, but all calls are received on one phone.

Multichannel. Allows one number to receive several calls at once. How much exactly depends on the number of lines that the IP provider allocates when connecting the tariff. Omnichannel benefits sales. When clients start calling at the same time, the line will not be busy - calls are distributed among available managers.

Record a conversation. When calling over an IP network, conversations are recorded automatically on the provider's side, if such a function is enabled.

IVR menu. This is the voice menu that the subscriber hears when calling the line. It distributes calls between departments. The robot pronounces the sections and names the numbers of the buttons on the phone that correspond to them. For example: “Press 2 if you want to know if the payment has been credited,” “Press 3 if you want to know the balance.”

As well as additional features that will allow you to:

Automate routine operations through integration with a CRM system and business applications. For example, a customer card will be automatically created when a call is made, and if a call is missed, a callback task will be created.
Analyze managers' conversations.
Maintain statistics on incoming and outgoing calls.
Use short numbers for internal communications.
Organize a conference call with any number of participants.
Create a “black list” of subscribers and configure a voice message for them.

RTP server reservation

In general, we need to track the crash of the RTP server and send a reinvite to both clients.

The task is divided into several sub-points:

Save the SIP dialogue to the database.
Track server death.
Send reinvite for each dialogue.

Implementation details vary greatly depending on the cluster configuration.

As an example: FreeSWITCH has a built-in 'sofia recover' command to recover calls after a server restart. But judging by the reviews, this command will not work in the case of a cluster of FreeSWITCH servers. Here you will probably need custom development.

List of advantages

The Flexisip project is written in C++ and has been successfully used as a SIP server in the Linphone project since 2011. Due to its lightweight and resource-friendly nature, Flexisip Configuration Listing can be deployed on embedded systems, small hardware modules, virtual and cloud platforms. In addition to small platforms, Flexisip supports clustering (using the Redis backend to synchronize nodes) and multiple scaling. Thus, Flexisip is a very flexible and versatile product for various types of tasks.

Let us list the advantages of the Flexisip system:

lightweight and undemanding to hardware resources;
ease of installation and configuration;
modularity, the ability to place modules on various physical servers;
the ability to use as a push notification server;
authentication of subscribers using hash sum or TLS certificates;
possibility of clustering and scaling using built-in tools.

OpenSIPs|Kamailio + RTP proxy

SIP load balancer represented by OpenSIPs or Kamailio manages RTP Proxy or its equivalent. Communication between them is carried out by a certain protocol (not SIP), within which the load balancer must be able to make a request to allocate ports for the RTP protocol in order to send this data to clients. The client opens a new connection using the received IP and port, connecting to the RTP proxy.

All business logic is concentrated inside the SIP load balancer. The solution is suitable if you have very little business logic, and all you need is to switch calls.

In fact, you don’t even have to use a SIP server here. One of our 911 switching projects was implemented in Java: Phone Call Routing.

OpenSIPs|Kamailio + FreeSWITCH|Asterisk

A SIP load balancer represented by OpenSIPs or Kamailio forwards SIP packets through itself to the selected FreeSWITCH|Asterisk server. The closest analogy is http load balancer with sticky sessions. OpenSIPs itself only handles registration. When initializing a new dialogue (call), according to some strategy, it selects the least loaded Freeswitch node and redirects all SIP packets associated with this dialogue there. And Freeswitch itself, processing the client’s SIP packets, allocates an RTP port and sends the port number and its own white IP via OpenSIPs to the client.

It makes sense to move the business logic to FreeSWITCH|Asterisk, which are much more suitable for this and have a large set of different modules out of the box.

This solution is better suited for systems with IVR, Voicemail, Call Center and other features of advanced PBX systems.

Example:

Unified Communications System

OpenSIPs|Kamailio + FreeSWITCH + App server

The solution is similar to the previous one, but all business logic is transferred to the Application server. This approach eliminates the need for a lua script, but adds a potential point of failure.

On the other hand, if you already have an Application server and it is already a vital element of your Unified Communication solution, then why not. FreeSWITCH has a module xml_curl, which allows you to download the entire configuration via http from a third-party server. For every request, be it authorization or an incoming call, FreeSWITCH first requests http xml instructions and then executes them. The rest of the scheme is similar.

Need a VoIP project? Write to us and we will evaluate it for free.