BGP and Address lists + Mangle. How to implement domain crawling?
Pumping out lists of IP addresses and subnets in the Address List
If for some reason you do not use BGP, then the lists can be downloaded by script and loaded into Address Lists. But in our case, lists that can be obtained via BGP are better received via BGP. By BGP they are added to routes. And the script puts them in address-lists. The fact is that address-list + mangle consumes more resources than routes.
But this method also works, and therefore we will consider it too. The logic here is standard: there is a list of addresses, we mark all traffic that goes to IP addresses from this list, and all marked traffic is sent to the tunnel.
Let’s start by creating a list; we will take the lists from antifilter.network.
To download lists you need a script. On official Mikrotik forum ROS script gurus have been licking scripts for downloading large lists for 4 years now. Let’s take it.
The script is big, so it’s under a spoiler.
{
/ip firewall address-list
:local update do={
:put "Starting import of address-list: $listname"
:if ($nolog = null) do={:log warning "Starting import of address-list: $listname"}
:local displayed true
:local maxretry 3
:local retrywaitingtime 120s
:local retryflag true
:for retry from=1 to=$maxretry step=1 do={
:if (retryflag) do={ :set $retryflag false; :set $sounter 0
:if (retry > 1) do={
:put "Source file changed. Retring after a $retrywaitingtime wait..."
:if ($nolog = null) do={:log warning "Source file changed. Retring after a $retrywaitingtime wait..."}
:delay $retrywaitingtime }
:local fetchResult [/tool fetch url=$url keep-result=no as-value]
:local filesize ($fetchResult->"total")
:local downsize ($fetchResult->"downloaded")
:if ($filesize = 0 && $downsize > 0) do={ :set $filesize $downsize}
:local start 0
:local maxsize 64000; # reqeusted chunk size
:local end ($maxsize - 1); # because start is zero the maxsize has to be reduced by one
:local partnumber ($filesize / ($maxsize / 1024)); # how many chunk are maxsize
:local remainder ($filesize % ($maxsize / 1024)); # the last partly chunk
:if ($remainder > 0) do={ :set $partnumber ($partnumber + 1) }; # total number of chunks
:if ($heirule != null) do={:put "Using as extra filtering: $heirule"} else={:set $heirule "."}
# remove the current list completely if "erase" is not present (default setting)
:if ($noerase = null) do={
:if ($timeout = null) do={:set $timeout 00:00:00; :do {:foreach i in=[/ip firewall address-list find list=$listname] do={/ip firewall address-list set list=("backup".$listname) $i }} on-error={} } else={
:do {:foreach i in=[/ip firewall address-list find list=$listname dynamic] do={/ip firewall address-list set list=("backup".$listname) $i }} on-error={} };
:put ("Conditional deleting all".$dynamic." entries in address-list: $listname")
:if ($nolog = null) do={:log warning ("Conditional deleting all".$dynamic." entries in address-list: $listname")}
} else={:put "Entries not conditional deleted in address-list: $listname"}; # ENDIF ERASE
:for x from=1 to=$partnumber step=1 do={
# get filesize to be compared to the orignal one and if changed then retry
:local comparesize ([/tool fetch url=$url keep-result=no as-value]->"total")
:if ($comparesize = 0 && $downsize > 0) do={ :set $comparesize $downsize}
# fetching the chunks from the webserver when the size of the source file has not changed
# empty array when the source file changed. No processing is done till the next complete retry
:if ($comparesize = $filesize) do={:set $data ([:tool fetch url=$url http-header-field="Range: bytes=$start-$end" output=user as-value]->"data")} else={:set $data [:toarray ""]; :set $retryflag true}
#:if ($ownposix = null) do={
# determining the used delimiter in the list, when not provided in the config
# this only run once and so the impact on the import time is low
:local ipv4Posix "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}"
:local ipv4rangePosix "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}"
:local domainPosix "^.+\\.[a-z.]{2,7}"
:local sdata $data;
# removes any lines at the top of the file that could interfere with finding the correct posix. Setting remarksign is needed
:while ([:pick $sdata 0 1] = $remarksign) do={ :set $sdata [:pick $sdata ([:find $sdata "\n"]+1) [:len $sdata]] }
:while ([:len $sdata]!=0 && $delimiter = null) do={ # The check on length of $sdata is for if no delimiter is found.
:local sline [:pick $sdata 0 [:find $sdata "\n"]]; :local slen [:len $sline];
# set posix depending of type of data used in the list
:if ($sline ~ $ipv4Posix) do={:set $posix $ipv4Posix; :set $iden "List identified as a IPv4 list"}
:if ($sline ~ $ipv4rangePosix) do={:set $posix $ipv4rangePosix; :set $iden "List identified as a IPv4 with ranges list"}
:if ($sline ~ $domainPosix) do={:set $posix $domainPosix; :set $iden "List identified as a domain list"}
:if ($sline ~ $posix) do={:put $iden}
:if ($sline ~ $posix) do={ # only explore the line if there is a match at the start of the line.
:do {:if ([:pick $sline 0 ($slen-$send)] ~ ($posix."\$") || $send > $slen) do={
:set $delimiter [:pick $sline ($slen-$send) ($slen-($send-1))]; :set $result true} else={:set $send ($send+1)}
:if ($result) do={ :set $extra [:pick $sline ($slen-$send) ($slen-($send-1))]
:if ( $extra = " " ) do={ :set $delimiter [:pick $sline ($slen-$send) ($slen-($send-2))] }
:if ( $extra = " " ) do={ :set $delimiter [:pick $sline ($slen-$send) ($slen-($send-3))] }
:if ( $extra = " " ) do={ :set $delimiter [:pick $sline ($slen-$send) ($slen-($send-4))] }
}; # EndIf result
} while (!$result); # EndDoWhile
}; #IF sline posix
:set $sdata [:pick $sdata ([:find $sdata "\n"]+1) [:len $sdata]]; # cut off the already searched lines
:if ($delimiter != null) do={:local sdata [:toarray ""]} ; #Clearing sdata array ending the WhileDo loop
}; #WHILE END $sdata
:local sdata [:toarray ""]
:if ([:len $delimiter] = 0) do={ :set $delimiter "\n"; :set $delimiterShow "New Line" } else={ :set $delimiterShow $delimiter }; # when empty use NewLine 20220529
#} else={:put "User defind Posix: $ownposix"; :set $posix $ownposix } ; # ENDIF ownposix = null
:if ($delimiter != null && $displayed ) do={:set $displayed false; :put "Using config provided delimiter: \"$delimiterShow\""}
:if ($posix = null) do={:set $posix "."}; # Use a match all posix if nothing is defined or found
:if (!retryflag) do={:put "Reading Part: $x $start - $end"}
:if ($timeout = null) do={:local timeout 00:00:00}; # if no timeout is defined make it a static entry.
# Only remove the first line only if you are not at the start of list
:while ( [:pick $data 0 1] = $remarksign) do={ :set $data [:pick $data ([:find $data "\n"]+1) [:len $data]] }; # removes the invalid line (Spamhaus)
:if ($start > 0) do={:set $data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
:while ([:len $data]!=0) do={
:local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
:if ( $line ~ $posix && $line~heirule) do={
:do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$comment timeout=$timeout; :set $counter ($counter + 1)} on-error={}; # on error avoids any panics
}; # if IP address && extra filter if present
:set $data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
# Cut of the end of the chunks by removing the last lines...very dirty but it works
:if ([:len $data] < 256) do={:set $data [:toarray ""]}
}; # while
:set $start (($start-512) + $maxsize); # shifts the subquential start back by 512
:set $end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the
}; # if retryflag
}; #do for x
}; # for retry
:if ($counter < 1) do={:set $resultline "Import was NOT successfull! Check if the list $listname is still being maintained."} else={:set $resultline "Completed reading $counter items into address-list $listname." }
:put $resultline
:if ($nolog = null) do={:log warning $resultline }
:if ($counter > 0) do={:do {/ip firewall address-list remove [find where list=("backup".$listname)]} on-error={} } else={
:do {:foreach i in=[/ip firewall address-list find list=("backup".$listname)] do={/ip firewall address-list set list=$listname $i }} on-error={}
:put "Restoring backup list: $listname"
:if ($nolog = null) do={:log warning "Restoring backup list: $listname"}
}; # if counter restore on failure and remove on success
}; # do
$update url=https://antifilter.network/download/ipsmart.lst listname=vpn-ip timeout=1d nolog=1
$update url=https://antifilter.network/download/subnet.lst listname=vpn-subnet timeout=1d nolog=1
}
# To be used configline settings:
# url= https://name.of.the.list
# listname= name of address-list
# Optinal settings
# timeout= the time the entry should be active. If omited then static entries are created.
# comment= puts this comment on every line in the choosen address-list (default: no comment)
# heirule= this will select on a word on each line if to import or not (default: no heirule)
# noerase= any value, then the current list is not erased (default: erase)
# ownPosix= allow to enter a onw regEX posix to be used (not ative at this moment)
# nolog= any value, then don't write to the log (default: writing to log)
We will use the ipsmart and subnet lists. Each list has its own address-list. If you use other lists, then specify them in $update url.
We place it in System – Scripts – Add new. The script itself is inserted into the source field.
Policy: read, write, test.
We’ll launch it right away Run Script.
You can check that addresses are being added in Firewall – Address Lists.
Add to cron System – Scheduler – Add new.
Policy are the same as the script
Interval – once every time. For example, once every 23 hours: 23:00:00.
On Event – the name of the script that was specified when creating the script.
With one command:
/system scheduler
add interval=23h name=ScheduleGetLists on-event=GetLists policy=\
read,write,test
We create a table, a route to the tunnel for this table, and two marking rules for two lists. You need to substitute the name of your VPN interface instead of $INTERFACE.
/routing table
add disabled=no fib name=vpn
/ip route
add comment=vpn disabled=no distance=1 dst-address=0.0.0.0/0 gateway=$INTERFACE pref-src="" routing-table=vpn suppress-hw-offload=no
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=no dst-address-list=vpn-ip new-routing-mark=vpn passthrough=yes
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=no dst-address-list=vpn-subnet new-routing-mark=vpn passthrough=yes
Do a trace from the client, the route must go through the tunnel.
How to disable block bypass with mangle+address-list?
Just turn off the mangle rule. ip – firewall – mangleDisable button for the rule created above.
List of domains in Address lists
Let’s move on to the most interesting part. I took Mikrotik out of the closet just for this. The resolving of domains built into ROS and adding their IP addresses to lists is implemented simply by adding the desired domain to address-lists.
Add a domain, it will immediately resolve and appear in this list. Sounds good, but there are a couple of problems.
This works well on 5, 10, 20 domains, but if the number of domains exceeds about a hundred, then the brakes begin. This could be as simple as a large list without a configured mangle rule, but with it the router will definitely have a bad time.
Let’s take a list with ~500 domains and use sed to generate commands to add these domains to the address-list.
I warn you, your router will most likely become ill. If you want to repeat it, don’t do it in production.
wget -qO- https://raw.githubusercontent.com/itdoginfo/allow-domains/main/Russia/inside-raw.lst | sed "s/.*/add address=& list=vpn-domains/"
Before this list we will add the command
/ip/firewall/address-list/
I’ve tested this a couple of dozen times and the results of adding 500 domains vary. Tested ROS with default settings.
It happens that everything freezes at once, including the console. And it happens that at first glance everything is okay, but after a little poking, I noticed large lags. And after clearing the list, everything falls into place.
If you need to delete all entries in all lists
/ip firewall address-list remove [find]
Or delete all entries in only one list. Example with vpn-domains
/ip firewall address-list remove [find list~"vpn-domains"]
When I try to delete lists I often get an error no such item (4). Sometimes you need to repeat it several times, but sometimes only restarting the router helps.
At first I thought: “I just have a router of almost the lowest model.”
In my experience, this does not depend on the power of the iron. I tested it on a virtual machine with ROS, giving it 4 Ryzen 5 5600U cores and a gigabyte of RAM. Throwing in the same 500 domains. The result was the same.
But I noticed that the formation of the list of IP addresses from domains is influenced by the DNS server that the router uses. On some DNS servers the list was formed with difficulty, on others it was faster. Apparently, the issue here is the limits on requests. Or maybe it’s a 1Mbit/s limit for ROS Cloud without a license.
But besides my experiments, one person shared his experience: hAP ac2, with a hundred domains, the router got hot and restarted itself.
If you don’t want to experiment with your router, I show what happens in this case in the video, link at the beginning.
I can assume that this is simply the implementation. I couldn’t find any recommendations on the number of domains in the lists in the official documentation.
I don’t understand why no one writes about this. I didn’t see any mention anywhere of what would happen to a couple of hundred domains in address-lists. The same can be achieved by solving the problem with summation.
Besides this, there are a couple more problems:
It can only resolve subdomains, which puts it in an advantageous position compared to the same Dnsmasq (it works only by wildcard). In the case when the main domain should not be in the lists, but a subdomain is needed. But, on the other hand, this is a problem, you need to set all subdomains of the resource. For small resources it is not difficult to collect, but for large ones, where they can be dynamic and depend on the location of the request, this is a problem.
ROS itself resolves these domains. Firstly, we cannot make a wildcard here and we encounter the first problem, and secondly, ROS relies on TTL DNS records. If the TTL is 8 hours, the entry will be updated once every 8 hours, but sometimes the TTL is set near zero, and the router will constantly update the entries. And this is not configurable in any way.
Even if there were no problems with generating a list of IP addresses and the mangle rule miraculously worked quickly, you still run into the problems described above.
Thus, using a large number of domains in the Address list is a bad idea.
How to use resolution in ROS
What can you take from the built-in ROS resolution functionality?
Add the most necessary domains to Address-lists. Most likely, even resources that are not in the provided lists of subnets or they are resolved differently for you.
Stop when the router gets sick.
If the Address List bypass is already used, add another magle
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=yes dst-address-list=vpn-domains new-routing-mark=vpn passthrough=yes
In case of using BGP, the same as for other lists, just substitute your list name in $LIST:
/routing table
add disabled=no fib name=vpn
/ip route
add comment=vpn disabled=no distance=1 dst-address=0.0.0.0/0 gateway=$INTERFACE pref-src="" routing-table=vpn suppress-hw-offload=no
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=no dst-address-list=$LIST new-routing-mark=vpn passthrough=yes
Thus, the best solution for Mikrotik that I came to is to use BGP + mangle by domain.
Crutches
The crutches are based on the fact that you do not use a ROS DNS server, but one raised somewhere at home. It, in one way or another, adds IP addresses to the Address-list or routes via BGP according to a list of domains.
Here is an example article about adding via BGP: Bypassing RKN blocking using DNSTap and BGP
What else can you think of?
OpenWrt as a virtual machine on a router with a DNS server and some kind of script inside
You can install a DNS server in docker on the router if your router has ARM architecture
You can connect some zero PI via USB and deploy it on it.
I haven’t implemented any of this. If you happen to be, yes, it will be interesting to read about your experience.
The ISP interferes with DNS traffic
ROS7 has DoH support, configured in IP – DNS. You just need to specify the DoH server. For example, https://1.1.1.1/dns-query.
Blocking VPN protocols and Mikrotik
Mikrotik does not support XRay and Sing-box combines. Therefore, in this case I see two and a half solutions:
If you are an organization, you can provide the IP addresses of your VPN servers somewhere and forget about such a problem.
Chain of servers. As we know, VPN protocols are not blocked if the packet goes to a Russian IP address, so you can raise WG on a Russian server and redirect traffic from it to a server with a non-Russian IP address. Or even use a server next to Mikrotik for these purposes, which will also redirect traffic on its side to a tunnel that is masquerading as HTTPS.
Someone wrote to me about deploying XRay/Sing-box in docker directly on Mikrotik. Again, you need a router with an ARM processor. This looks strange and unreliable to me, but maybe I’m wrong.
On the other hand, chances are you won’t need it for the next year. Or at least SSTP will suffice until they systematically learn to block it.
Relevant materials
Want to use large lists of domains and Shadowsocks/VLESS/etc directly on the router? I recommend looking towards implementing this in OpenWrt.
If you have any experience on any of the points in the article, it will be interesting to read. Especially about domain lists.
I publish materials about VPN and targeted bypass of blocking, which do not reach a full article, in my telegram channel.
