DNS filtering usually only filters on incoming packets, but for bot stuff that should catch issues.

In general, most routers run everything from a serial flash chip on the board. These are usually 8, 16, or 32 megabytes. They have a simple bootloader like U-Boot. This is what loads the operating system. These devices have a UART serial port on the PCB. You can use a USB to serial UART adaptor to see what is happening in the device. With a proprietary OS, you are still likely to see the pre-init boot sequence that the bootloader prints to terminal. Most operating systems also print information to this interface, at least of the couple dozen junk devices I have been given and messed around with. I make a little mount for a USB to serial adaptor and add it to all of my routers when new, so I only need to plug in USB to get to the internal bootloader and tty terminal interface of OpenWRT. You will need to know the default baud rate of the device, although it is probably listed somewhere online or can be guessed as one of the common high values at or above 9600.

Getting into this further gets complicated. It is probably better to look for any CVE that is relevant to the device or software and work backwards. Look for any software updates that have obfuscated the risk for each CVE. If the issue was not fixed, that is where to look to see if someone has exploited the device. Ultimately, they need clock cycles from the CPU scheduler. So it must be a process or some way of executing code from unregistered memory.

This is getting to the edge of what I have messed around with and understand. There may be a way to get a memory map that includes unused pages, and compare that with a hex dump of the flash memory. This is outside of your scope of a proprietary OS, but hopefully frames the abstract scope of what is possible on this class of device when you have an open source stack. The main advantage of this kind of device and issue is that you can physically remove the flash chip and then see and manipulate every page and memory location. The device likely doesn’t have microcode loaded into the CPU(s) that make it challenging to determine what is going on.

There is probably an easier way, but a hex dump of the current system can be hashed against the factory updated version to see if any differences are present. It is likely that any exploit will include a string with the address to connect to somewhere in flash memory. It could be obfuscated through encryption or a cypher, but a simple check for strings in the hex dump and a grep for “http” is a simple way to looks for issues.

The OpenWRT forum is a good general source. The people behind the bootloaders for these devices are also Linux kernel developers and on the OpenWRT forum.