Blue Teaming for a CCDC event is a harrowing experience. You are thrown into an unknown environment with patchy documentation and some client machines with which to access servers, and the full knowledge that you’ve got mere hours before the much more experienced Red Team starts to poke holes in things. At the same time Blue Teaming is also a fun experience that allows you to learn a lot about project management and teamwork. It forces you to learn to work with something new and gives you valuable insight as to why documentation is so important.
There were several things that worked well for my team whilst participating in Pacific Rim CCDC, and there are several places where we could have done better. In this post, I aim to explain what those things were.
Preparation is a really big part of PRCCDC. Everything that worked really well for my team worked well because of our preparation. From printed command reference guides for various operating systems and programs to a predefined grid of passwords and usernames to use for the servers. We read over various Red and Blue Team debriefs, went over notes from a previous year, and worked hard to paint a picture of how things would look for us during the competition.
Choosing a Team
The hardest decision was choosing who got to be a part of the team. Our previous year at PRCCDC had made us painfully aware that technical skill is perhaps of less importance than having a team that simply works well together. If a highly skilled team cannot cooperate well or even stand each other’s presence, things simply will not work.
At the same time, choosing team leadership and technical roles is just as important. The team leader doesn’t necessarily have to be the most talented person, just the person who works best with other people. In the same token, someone with patience should be answering the phones and interacting with uninvited guests. Having someone good with notes and documentation should be filing incident responses and working with injects. All of this should be decided well in advance of the competition. The final list of roles should be printed and taken with you as reference when things get messy.
Choosing What to Bring
Blue Team members are not allowed to bring their own equipment into the competition space. Backup drives, thumb drives full of tools, or Live CDs are just going to result in you getting turned away at the door; however printed notes, notebooks, books, paper tags, Post-It notes and the like should be a part of everyone’s competition kit. These items are allowed into the rooms as reference materials.
Be careful to avoid information overload. Basic OS commands, SSH and SCP syntax, and networking/firewall commands should be sufficient. There is no need to bring every Linux Man page o
r the entire contents of TechNet. Anything you don’t bring will likely be accessible online, and you will save time when looking for those critical commands by just sticking with the basics.
Random password lists are also valuable. We printed out several identical pages for each team member, each with long, random passwords, generated in such a way as to avoid ambiguous letters/numbers/symbols. Each password was numbered, for example, one to one hundred, and a blank line next to the password was meant for name of the server and the username it was associated with. Once a password was changed or a new account was created, the server name, username and number would be called aloud, and the team would write that information on the correspondingly numbered line for reference. In this way we were not forced into weak passwords or password reuse. The passwords would be rotated periodically or when we realized there was a breach, and this was a key factor in keeping things organized.
Creating a Game Plan
Until the morning of the competition, no one knows what to expect. We don’t know what services there will be, what is considered mission critical, or what holes will need patching. Because of this, it is entirely impossible to assign a machine to everyone; however, this doesn’t mean that there should not be a plan in place as to how to tackle the competition.
For example we could have set up a plan where we first mapped the network, then reviewed user accounts, then reviewed software version numbers and began the process of patching out of date software. However all we had done was pick someone to handle calls and someone to lead the team. These two things are important, but so is a fully fleshed game plan. Instead we walked into the room and ran about like blind chickens trying to organize a concerted effort.
During day one, we were given access to the IT infrastructure, handed bare-bones documentation about what was running on the network and what the importance of each machine was, and were told we had two hours before the Red Team started to poke at us. We quickly decided to split up based on the roles we had been assigned and start hardening servers, and then make backups at the end of the two hours so we had a more solid platform on which to rebuild should the Red Team force us into reverting to backup.
This plan was quickly disrupted when we found out that there was something wrong with our hypervisor preventing us from gaining access, and we were the only team having that problem. Some of our servers were offline or inaccessible, and we could not make any backups. It turned out that the problem we were encountering was not supposed to be a part of the competition, and we ended up escalating the issue to the PRCCDC staff for help. Clearer and more persistent attempts to get it resolved would have gotten us back into the competition much quicker and given us the time to start hardening, but we instead let the matter be. While waiting for them to come solve the issue, we assigned one person to try to fix it while the rest of us began to harden the servers we could access.
Looking back, one thing we did a poor job on that first day was mapping the network. We took the documentation at face value, and decided not to create our own network map and monitor for new hosts. Had we done what we had planned and created a network map, we would have avoided issues that came up later in the competition.
We also should have requested that a list of the (fictitious) current staff and chain of command be provided to us. Unused user accounts were one of our first big mistakes that gave the Red Team a major foothold. This could have been simply fixed by gathering that list and revoking access rights and locking accounts at the start of the competition day. Instead we focused entirely on our own accounts and on hardening services. We were owned very quickly, and when we left for the day the feeling of defeat was hard to push off.
After noting a relatively serious new breach that had apparently occurred overnight, we filed an incident report and began again. This time we had no head-start and were immediately set upon by the Red Team.
Calls and injects also started coming in heavily, and we were made aware that teams could finally access their assigned router. We really started coming into our own but were caught up in the mayhem of getting people started and rebuilding the things that had fallen the previous day. We missed doing most anything on the router, and critically missed a machine that had popped up onto the network that was not there before.
This goes back to assigning responsibilities and watching the network. Had we been certain to give someone the task of hardening the router, we would have been able to make it harder to get into the network, and we would have realized the existence of unnecessary admin accounts with default passwords. Had we mapped the network at the beginning and effectively monitored it, we would have also noticed a rogue machine pop up onto the network. Both of these things are basic practices that were simply forgotten in the heat of the moment.
We discovered that the most targeted system was our website and its associated database, and that they had managed to script the attack in such a way that even if we reverted the website to a snapshot it would quickly be defaced once again. Only after the competition did I discover that a default service account was the Red Team’s path into the database server. This was a stunning realization that once again enforced the "back to basics" mentality.
As we neared the end of the second day, injects started hitting us really hard along with phone calls, and we ended up becoming quickly swamped, and had to choose between focusing on injects or focusing on tracking down the breaches and repairing the holes. In the end we split the team in two and devoted half of our efforts to injects and the other half to keeping up the machines that were still scoring us points. Honestly this appeared to be the most effective tactic for the time, however we would not have had to give up on the other machines had we done what we should have at the start of the competition.
Day Three was something that I had not heard of happening in CCDC before, so I will not ruin the surprise; however, it was on this day that I realized just how simple the Red Team attacks were. Every hole that I thought must have been so complicated for the Red Team to exploit was actually easy to exploit and even easier to prevent. It was almost always a vulnerability that could be easily patched, a default account, unneeded permissions, etc. The Red Team had no time or need to write a new zero day exploit. They were exploiting reliable methods of attack and were successful in doing so.
When all is said and done, as I look back at my experience on a CCDC Blue Team, I realize that had we simply stuck to the basics and come into the competition room with a solid game plan, we would have done far better, and perhaps even won the competition. We had the technical skills to patch machines, dig through user accounts and map the network, we just did not have the proper plan in place. Instead we started running about in a disorderly fashion trying to keep things up and work around each other without a solid plan on how to tackle the challenges that came our way.
Preparation and the appropriate mentality shines through as being the most important part of Blue Teaming at PRCCDC. It was also the place where we failed miserably, and from what I saw, where most other teams failed as well. I came out of the competition with a very simple realization.
Start with the basics.