Day 40 of #100daysofhomelab and the internet is a little more stable… Still not 100%, but “stable”. Speed test results have dropped, as you can see in the graph below, but weirdly, ping times are a little better…
Download speeds. The swap over happened around the 8th Feb, 9th was pretty much a wash, 10th things got a bit better…
Upload Speeds. less spikey upload speeds, but also less upload speed…
ping times went from around 38-40ms to around 28-30ms…
I currently have Observium watching the traffic on the routers, and all logs are being written to an ELK stack. Not correctly (links below on how it *should* work, but I don’t have it fully working… yet) but they are being logged nonetheless.
OpenVPN in LXC – Proxmox VE Need to install Zerotier on an LXC container on proxmox. This is how to get it working.
this post is for day 38 and 39 of #100daysofhomelab… and i have finally moved over to my #RB5009… and, well, it has not gone so well… It has rebooted a few times due to memory issues (too many BGP tables being held, so I shut a few down to start with… some cleanup needed there), then the internet connections are a little unstable, and, well, in the last 48 hours, I have spent more time on LTE than on proper internet… It does seem to be working (ish…) now, but not as fast as it was. I am just using the #Zerotier link, so the #Wireguard links are currently off… Anyway, below are some links… I hope to make things work better tomorrow… And i also hope to have a better write up soon too…
Day 31 of #100daysofhomelab and I am going through the config from my CHR to bring over to my RB5009, and, well, I have no idea what I was doing when I built the original config… Now to try and figure out what the config did, since I want to document it here so I know what I was thinking, but to also possibly help someone else… Mind you, at this stage, it won’t be much help… I also need to figure out how to add my Zerotier Bridge into the mix.
So, as trying to get a high level overview of how this works, lets start with this:
The cable modem comes in at 1Gb/s down, 50Mb/s up. It hands off at 1Gb ethernet and plugs into a switch on VLAN 900. Anything on VLAN 900 can get a public IP from that modem (statically assigned, I have 5 usages, the first being the modem to act as a gateway).
FTTH comes in and goes to my small quad 2.5Gb box, which then, using CHR (we call this DUB1-BK01), hands off a /29 to VLAN 905. Again, any devices on VLAN 905 can get a public IP from there, and use BK01 as a gateway.
For the current CHR (DUB1-BGP01) it being a VM has currently got 3 connections: eth1 is connected to VLAN900, eth2 is connected to VLAN905 and eth3 is connected to VLAN901. VLAN901 has a /27 from my block of /24 addresses, and anything on that VLAN can use an IP from that pool and the IP from DUB1-BGP01 as its gateway.
DUB1-BGP01 connects to both lon1 and fra3 over WireGuard connections. All traffic to lon1 is sent over the Cable Modem link. All traffic to fra3 is sent over the FTTH link. Currently, there is no automatic failover if one link dies… This is where (hopefully) Zerotier comes into play.
I have a VM running on my i7 2.5Gb box that has connections to both VLAN900 and VLAN905, along with VLAN911. I have a bridge on that box that connects VLAN911 to a Zerotier network which is used only for internal peering. It has a /28 Public IP Range and anything on that bridge can use an IP from that network and talk to other machines. Currently that bridge is directly connected to my UDM Pro, and it gets a public IP and uses fra3 as a gateway. Sometimes traffic goes though fra3 but comes back over lon1 (due to asymmetric routing). But because of the way the network is working, all traffic can flow without issues.
The plan is to use that VLAN along with the 2 WireGuard links and give me 2 connections to lon1 and fra3. In theory, if one connection goes down, the traffic should be able to flow the other way…
So, at least that is the theory… How well this will work is anyone’s guess… But more messing with configs is required.
Day 25 of #100daysofhomelab, and not done much in the way of home lab work today, but has tested the bejesus out of the internet connection! I bought a Backblaze License for my Mac Book Pro, which initially has around 2.3Tb to backup. There are my YouTube Videos along with code and other bits… It looks like it has uploaded 290 Gb in the last 24 hours…
I also bought an Xbox Series X, and have downloaded a few games to it too… I previously had an Xbox One S with the Games Pass Ultimate, so those games were downloaded. I think it’s downloaded nearly 200 GB in the last few hours! Finally, my mother got home from the hospital yesterday and found a Netflix TV show she wanted to watch and has binge-watched most of it. That seems to be a bit more sedate 20Gb since last night… Overall, the Zerotier-backed connection seems to be working well!
For the last few weeks, I have been running a Raspberry Pi in my car, along with a small UPS and a Wifi Access point, allowing me to download videos from my dash cam and back them up to my NAS in the house. But I have had some teething issues, and I am currently thinking my way through some fixes…
First, the Pi is connected to both the network in the car (via ethernet) and network in the house (via Wifi). It seems that when the car is parked outside, sometimes the Pi can’t talk to the internet, and sometimes it can’t talk to the dashcam… It’s a routing issue, and it’s starting to annoy me…
I thought the onboard Wifi on the Pi was a little weak… it wasn’t getting much more than about 2-3Mbytes/s (16-24MBit/s) when downloading from the Pi to the House. Given the Pi was serving content from an SSD (not the internal MicroSD) I would have hoped for faster. I tried swapping in an external Wifi dongle with an aerial, but the same kind of speed… must be having issues getting through the metal and glass in the car, plus the metal, glass and brick in the house…
I started running out of disk space on the SSD on the Pi after about 3 or 4 weeks of video… so, I needed to tweak the command for the download script to only keep 14 days on the pi. Resilio Sync, the app I use to sync back to the house, has a “keep deleted files in an archive” folder option, so when the pi does delete the files, they are still stored on the Pi… I would like to find a way of automating that…
While trying to figure out how to fix part 1, I came up with an idea: I have an older Mikrotik RB951G that can be powered via a 12v adapter for the car. I am going to use that, along with a Huawei 4G dongle to act as an internet connection. The onboard Wifi will be in client mode, so when it’s near the house, it will connect to the main network and send traffic through that to the internet (or internal NAS) and when away, use the LTE modem. Then, using the Wifi dongle on the Raspberry Pi, use that as a Wifi AP.
Anything in the car that needs Wifi will connect to the Pi, which will act as a bridge to the Mikrotik. When the script needs to download files from the dashcam, it should have a direct connection to it, plus (hopefully) will be faster… then the Pi is connected to the internet through the Mikrotik. The Pi has both Tailscale and Zerotier on it for remote management, and the Mikrotik can be configured to use Wireguard to connect back to the house directly if required.
I have some of this working on a bench in the house, but it will be a while before I manage to get this running fully… Hopefully, I will have some more stuff sorted this weekend…
A few months back (well, November 2020) I wrote about connecting to my car with Zerotier. In this post, I mentioned using a TP-Link router running OpenWRT and a Huawei LTE dongle to connect to the internet, which allowed me to then connect to my Blackvue Dashcam and watch remotely… But it had some issues I wanted to fix:
The Huawei Wingle was a little slower on 4G than I would have hoped…
When the power in the car went out, everything stopped working immediately (12V sockets in the car run for about 20 min after the engine shut off)
It did not connect to the WiFi in the house when parked
No option for backing up Video…
So, I went digging to find some alternatives… and I realized I had a load of them floating around the house: the Raspberry Pi. Specifically, the 4GB Pi 4. I got my hands on a Pi UPS Hat, a couple of 18650 Cells and an SSD Expansion board with a 512GB (overkill I know) SSD. I also got a BlackVue Power Magic Battery, B112, which will power the Dash Cam (a BlackVue DR750S-2CH). It has 2 USB ports, which allows me to run both the Pi and the new WIFI router, a Netgear Nighthawk M1.
When the car starts, it powers, via the 12V socket in the boot of the car (trunk for my American friends), which powers the Blackvue Battery. Cables run from there to the front of the car where the front camera is. (there is also a rear-facing camera in the boot too… more cables!) This then also starts the Pi and starts charging the 2 18650 batteries. Finally, well, at the same time really, the Nighthawk starts running too. Because the batter on this was running hot, the battery is removed from this.
The Pi is hooked to the Nighthawk via ethernet and the WIFI is set to connect to the house when it sees it. The BlackVue uses the WIFI from the Nighthawk for its internet requirements. When the pi boots, it connects to Zerotier for management via SSH or VNC (I use VNC to remote into the box and watch the live video when the car is parked or when someone else is driving).
There is also a python script that is scheduled to run every 15 min that downloads the videos from the Dashcam. It also downloads any GPS and other info. The folder these files are downloaded to is on the SSD and is shared with my machine at home via Resilio Sync. To make sure I don’t use all my LTE usage, the machine at home is set to only download what I want to download. So, if the car is somewhere else, I can download specific files when I want, or when at home, I can download full days, if required.
It’s been running for a few weeks now, and so far, so good. I haven’t had to do any clean up of the SSD, yet, but I would guess that eventually, I will need to look into that… With the 4G connection and Zerotier, I can then connect to my car and watch the live video whenever it is online, and whenever it is driving, within 15 min it will start downloading videos. I could, in theory, do a LOT more with the Pi in the car… Some ideas that come to mind:
Turn WIFI off on the Nighthawk and use the Pi as a Router, probably adding a second WIFI adapter to get better range… This could then have PiHole running on it for monitoring DNS traffic…
Since I have access to the GPS files in (somewhat) real-time, use it to map the car in somewhat real-time. Though, I do this already using Ruhavik and a TeltonikaFMC-001.
Connecting to the car’s OBDII port (On-Board Diagnostics) and getting data from the car… Technically, again, the FMC001 does most of this, but in theory, it could be replaced with something else…
Keep an eye on the blog for future possible projects with this… Not sure where this project will get me, but we will figure it out at some stage… Leave a comment if you have questions!
I use ZeroTier on my network for a good few things, including internal network peering between BGP VMs, management of machines, and now, connecting to my car over LTE. This is one of those posts that sounds silly, but is very handy! First, the parts list:
Car…
3G/4G/5G modem of some sort. I am using a Huawei Wingle… Can be used without the Router below, but I wanted Zerotier, so I have it in modem only mode…
A router that supports Zerotier. I am using a modified TP-Link TL-WR703N upgraded to 16MB ROM and 64MB RAM. This is required for newer OpenWRT builds
After installing the the latest copy of ROOter on the TPLink (or router of your choice) and getting the modem configured correctly (this took a while) you need to install the Zerotier software though the dashboard. Once installed, I joined my Zerotier network using the CLI (SSH into the router) and the approved it though the my.zerotier.com dashboard. Once its approved and connected, you can now go to the Zerotier IP and get to the router directly. From here, you can either setup a route in Zerotier to point at the internal network behind the router, or, in my case, setup a SSH tunnel to the dashcam. I found the IP given to the dashcam and used SSH forwarding to get to it. Finally, i used the URLs from Digital-Nebula’s hackview repo to get to the different URLs. I use this to download stuff like GPS logs, emergency videos, etc. I have to clean up some scripts at some stage for this, and plan to upload them at some stage.
