Overview

This article explains the function, benefits, and implementation of hardware offloading. As of EdgeOS firmware version v1.9.1, all EdgeRouter models support some type of hardware offloading.

Table of Contents

1. What is Hardware Offloading?
2. EdgeRouter Features Eligible for Offloading
3. IPsec Offloading Support
4. How to Enable/Disable Offloading
5. Optional Offloading Optimizations and Testing
6. Related Articles

What is Hardware Offloading?

Back to Top

Offloading is used to execute functions of the router using the hardware directly, instead of a process of software functions. The benefit of offloading in EdgeOS is increased performance and throughput by not depending on the CPU for forwarding decisions.

Example setup where the EdgeRouter (ER-4) is forwarding traffic with Hardware Offloading enabled.

There are many processes/features that can take advantage of the offloading engine. One of the most basic examples is IPv4 traffic forwarding. Without offloading enabled, IPv4 traffic will be routed via the CPU and will be limited to around 300Mbps on the EdgeRouter Lite (ERLite-3). With offloading enabled, the throughput will be about 950Mbps.

EdgeRouter Features Eligible for Offloading

Some processes currently cannot take advantage of hardware offloading. This can lead to confusion where it is assumed that offloading is disabled. A more accurate statement is that some features are not eligible to be offloaded and will always depend on the CPU.

There are two platforms that are used by different EdgeRouter models. Each platform has its own offloading support and unique commands to enable the functionality. The platforms are:

MediaTek

Used by the following EdgeRouter models:

• ER-X
• ER-10X
• ER-X-SFP
• EP-R6

Cavium

Used by the following EdgeRouter models:

• ERLite-3
• ERPoE-5
• ER-8
• ERPro-8
• EP-R8
• ER-4
• ER-6P
• ER-12
• ER-12P
• ER-8-XG

The tables below summarizes the features that can be offloaded on each platform.

MediaTek-Based Devices Feature Offload Eligible Command Bridging  hwnat DPI  hwnat NAT  hwnat VLANs  hwnat GRE  hwnat PPPoE  hwnat IPsec  ipsec QoS  -  NetFlow  - Bonding  - Suspend  -

Cavium-Based Devices Feature Offload Eligible Command Bridging  - DPI  ipv4 forwarding NAT  ipv4 forwarding VLANs  ipv4 / ipv6 vlan GRE  ipv4 gre  PPPoE  ipv4 / ipv6 pppoe  IPsec  ipsec  QoS  - NetFlow  -  Bonding*  ipv4 bonding Suspend  -

IPsec Offloading Support

IPsec offload provides significant IPsec performance improvements, increasing throughput for site-to-site and client-to-site tunnels by offloading the ESP (Encapsulated Security Payload) traffic. Not all available ESP hashing/encryption algorithms are compatible with offloading. IKE traffic is not offloaded but this is only used to establish the tunnel and will not affect the performance.

The tables below summarizes the algorithms that can be offloaded on each platform.

MediaTek-Based Devices Feature Offload Eligible Command 3DES AES-128 AES-256 MD5 SHA-1 SHA-256  ipsec SHA-384 SHA-512 AES-128-GCM AES-256-GCM  -

Cavium-Based Devices Feature Offload Eligible Command 3DES AES-128 AES-256 MD5 SHA-1 ipsec SHA-256 SHA-384 SHA-512 AES-128-GCM AES-256-GCM  -

How to Enable/Disable Offloading

That commands that enable and disable offloading are shown below. You should only need to enable offloading for these features if you are using them in your environment. However, enabling offloading for all features will not cause a negative impact if those features are not being used.

MediaTek

Used by the following EdgeRouter models:

• ER-X
• ER-10X
• ER-X-SFP
• EP-R6

1. Enable hwnat and ipsec offloading.

configure

set system offload hwnat enable
set system offload ipsec enable

commit ; save

2. Disable hwnat and ipsec offloading.

configure

set system offload hwnat disable
set system offload ipsec disable 

commit ; save

Cavium

Used by the following EdgeRouter models:

• ERLite-3
• ERPoE-5
• ER-8
• ERPro-8
• EP-R8
• ER-4
• ER-6P
• ER-12
• ER-12P
• ER-8-XG

1. Enable ipv4/ipv6 and ipsec offloading.

configure

set system offload ipv4 forwarding enable
set system offload ipv4 gre enable
set system offload ipv4 pppoe enable
set system offload ipv4 vlan enable
set system offload ipv4 bonding enable

set system offload ipv6 forwarding enable
set system offload ipv6 pppoe enable
set system offload ipv6 vlan enable

set system offload ipsec enable

commit ; save

2. Disable ipv4/ipv6 and ipsec offloading.

configure

set system offload ipv4 forwarding disable
set system offload ipv4 gre disable
set system offload ipv4 pppoe disable
set system offload ipv4 vlan disable
set system offload ipv4 bonding disable

set system offload ipv6 forwarding disable
set system offload ipv6 pppoe disable
set system offload ipv6 vlan disable

set system offload ipsec disable

commit ; save

Optional Offloading Optimizations and Testing

Starting from the v1.10.3 and v1.10.4 EdgeOS firmware releases, additional offloading optimizations are available:

• Flow Flushing Controls whether the offloading flow table is cleared upon routing table changes.
• Flow Lifetime Controls how long it takes before offloaded flows expire.
• Flow Table Size Controls how large the offloading table table is.

The Flow Flushing feature is enabled by default, which means that the flows in the offloading engine are flushed when a routing table change is detected. This means that any routing updates are processed instantly, but it also means that CPU resources are used to process these changes. In environments where the routing table is constantly updated, it can be advisable to disable the Flow Flushing feature which will decrease the load on the CPU and increase the performance. Starting from the v1.10.3 release, the following commands are available to disable the feature:

set system offload ipv4 disable-flow-flushing-upon-fib-changes
set system offload ipv6 disable-flow-flushing-upon-fib-changes

The Flow Lifetime is set to 12 seconds by default on all EdgeRouter models. Increasing the lifetime will lead to flows expiring less frequently and thus less packets will be forwarded using the CPU. But this will also mean that it will take more time before routing table changes are detected if the Flow Flushing feature above is also disabled. Starting from the v1.10.3 release, the following command is available to increase or decrease the lifetime:

set system offload flow-lifetime ?
 <0-4294967295>     Set custom lifetime for flows in offload engine (12 seconds by default)

The Flow Table Size is set to 8192 buckets by default on all EdgeRouter models. Increasing the lifetime will lead to the offloading table being able to store more flows. But this will also mean that more memory is used by the offloading process. Starting from the v1.10.4 release, the following commands are available to increase or decrease the table size:

set system offload ipv4 table-size ?
  8192          memory consumption    1 MB -     6 MB
  16384         memory consumption    2 MB -    12 MB
  32768         memory consumption    4 MB -    23 MB
  65536         memory consumption    8 MB -    46 MB
  131072        memory consumption   16 MB -    91 MB
  262144        memory consumption   32 MB -   182 MB
  524288        memory consumption   64 MB -   364 MB
  1048576       memory consumption  128 MB -   728 MB
  2097152       memory consumption  256 MB -  1456 MB
  4194304       memory consumption  512 MB -  2912 MB
  8388608       memory consumption 1024 MB -  5824 MB
  16777216      memory consumption 2048 MB - 11648 MB

set system offload ipv6 table-size ?
  8192          memory consumption    1 MB -    4 MB
  16384         memory consumption    2 MB -    8 MB
  32768         memory consumption    4 MB -   15 MB
  65536         memory consumption    8 MB -   30 MB
  131072        memory consumption   16 MB -   60 MB
  262144        memory consumption   32 MB -  120 MB
  524288        memory consumption   64 MB -  240 MB
  1048576       memory consumption  128 MB -  480 MB
  2097152       memory consumption  256 MB -  960 MB
  4194304       memory consumption  512 MB - 1920 MB
  8388608       memory consumption 1024 MB - 3840 MB
  16777216      memory consumption 2048 MB - 7680 MB

You can verify the timers and table size with the following command:

show ubnt offload statistics 

 Forwarding cache size (IPv4)
=============================

table_size (buckets)                  16384
table size (bytes)                    2097152
flows_max (bytes)                     9830400

 Flow cache table size (IPv6)
=============================

table_size (buckets)                  8192
table size (bytes)                    1048576
flows_max (bytes)                     2883584

 Flow timers
=============================

cycles                                101066477597
clock_rate                            1000000000
HZ                                    100
timer_ticks                           268415529
new_flow_interval (timer_ticks)       1200
old_flow_interval (timer_ticks)       400

You can also verify the offloading state by running the following command:

show ubnt offload 

IP offload module   : loaded
IPv4
  forwarding: enabled
  vlan      : disabled
  pppoe     : disabled
  gre       : disabled
IPv6
  forwarding: disabled
  vlan      : disabled
  pppoe     : disabled

IPSec offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.480

When high throughput is flowing on the router and the traffic is not offloaded, you will see an increase in CPU usage. When traffic is offloaded the throughput will be higher and CPU usage will remain low. Using a tool like iPerf / iPerf3 is a common way to generate and test throughput. It is important to not use the EdgeRouter as the server or client for iPerf when running the test, as the routers are designed to route/forward traffic and not to generate it.

Related Articles

EdgeRouter - Which EdgeRouter Should I Use?

EdgeRouter - Beginners Guide to EdgeRouter