colaianni.us

Cyber Lab

This article will detail how I set up the Clemson Cyber lab and got it ready to run CCDC training environments, as described in this article.

As president of Clemson Cyber, I am responsible for managing the cyber lab, a rack of servers in Clemson’s Watt Center. The lab hasn’t operated for two years after an NTP server failure. The lab was overly complex, with VMs running NTP, DNS, and Kerberos services that the host machines depended on. This means that a problem with the VM service or one VM in particular would affect the host machines, which is what I think caused it to go down.

My primary goal was to keep the lab simple and stupid to avoid these problems. I’ve been working over the past 8 months to get it operational again. Per the previous president’s recommendation, I worked with him to remove the machines, firewall, and switch, and reimage them with CentOS. This didn’t last for long, as he soon left the University, and I decided to reimage with Proxmox (if you haven’t heard of Proxmox, it’s fantastic, and I sacrificed my firstborn to it).

After getting the machines operational, I was locked in a three-month-long combat with the Palo firewall and Clemson’s NOC. Various Watt Center network upgrades had left the lab’s VLAN out of date and unable to access the internet, but some back-and-forth with the NOC staff fixed this issue.

With the VLANs in order, I was able to get remote access and stop bothering the Watt employees (thank you to the Watt employees, we love you ❤️, especially you, Mark). I then connected the four Proxmox machines in a cluster and set up a WireGuard VPN for secure remote access (previously, I had connected the lab to my VPN and accessed it through that). Based on my experience, WireGuard is the best VPN available. It perfectly embodies K.I.S.S., and having one config is wonderful. I then initiated a second war with Palo to resolve the port forwarding issue, which was swiftly resolved.

Now that the lab was 90% operational, I gave access to the CCDC team captains so they could work on creating the tryout machines elaborated on in the sister article. I used the D-Link switch to arm-wrestle and create a VLAN for all the VMs. A separate Ethernet cable running between the machines and the switch enabled me to create a VLAN that allowed all the VMs to communicate across machines.

Now, our beautiful lab is complete and operational. I present you with its lovely, logical, well-thought-through design and non-reactive design.

Network Diagram

You’ll notice that all the machines are named after APTs; how clever Adrian is! I forgot the IP address of the KVM and can’t access it anymore, as it seems to operate independently and ignores DHCP. Notably, the servers are connected via ultra-fast, latest-generation 10G cables on the 10.1.1 side for faster resource sharing.

Below is a bullet point summary of the details (this is for me when I forget in two months)

Fall 2025 Update

As the lab is actively used, it’s actively growing and changing to our needs. So far, I’ve added numerous Docker services to facilitate VPN access, secret sharing, and data provision. Additionally, the VM VLAN is no longer just one VLAN, and the ports have been put in hybrid mode to allow us to use several to segment off VM networks. Our primary reason for this is to run a Palo Alto VM with machines under its network, providing our team with a more comprehensive experience.

Better Network Diagram

I’ve added containers over time for the various services we need. A landing page is available at lab.cucyber.net; however, it currently lacks significant content. I’ve used Traefik for HTTP routing, as it’s dead simple and handles SSL for me. Currently, all Docker containers run on equation-group (I need to set up Kubernetes, but that’s a later issue) and are on macvlan networks to appear as typical hosts on the network.

I’ve put a lot of work into the WG containers and the Palo to set up proper traffic forwarding, so traffic from the WG containers doesn’t need to masquerade. This allows VMs to access & ping WG clients directly, allowing us to use reverse shells and better identify different clients. To do this properly, I should set up the WG containers on a different VLAN and configure the Palo to move traffic between them, but I’ve set up the macvlan with the IP range 10.3.1.1/29. This ensures packets moving through WireGuard are sent to our default gateway (the Palo) and from there routed to their proper destination (this is done to allow for symmetric traffic flow, making the Palo happy.

The lab will continue to change and evolve in the future, but I’m quite happy where it’s at. I try to keep in mind that the more I add, the harder it will be for the next Lab Master to maintain. Keeping this lab alive and well for the club’s future is imperative; otherwise, the team will miss out on valuable hands-on experience from VMs at their disposal.

Previous:
NeoVim
Next:
CCDC Training Environments