This year marked ten years since I first attended a Black Hat event. That first one was the USA flagship event in Las Vegas, and since then I’ve attended the European events (held in Amsterdam and London) as well as the Middle East version in Saudi Arabia.

The research is good, and conference experience is usually at a high level, but one thing makes Black Hat stand out - it has a visible Network Operations Centre (NOC). This is an interesting concept: the NOC is constructed, operated and torn down for roughly a week’s work, and in that time the Black Hat network is protected, as well as live workshops, and the front page of the Black Hat website. Therefore, this includes the live registration process.

The Operation

Over the many times I’ve had the pleasure of visiting Black Hat I’ve visited the NOC and been intrigued by what the analysts are doing, who they are, what they see and how much effort goes into this operation. At the recent European conference, I had the chance to step into its hallowed (temporary) walls, and meet NOC lead James Pope. 

Pope has a day job at Corelight - and whom provide the visibility monitoring as well as infrastructure, such as securing the information accessible on the Black Hat badge. He explains that part of the challenge faced is hacking on the network. " “Even at Black Hat, illegal is still illegal and we have people in classrooms hacking other classrooms, whether it is intentional or not.”

There is also the issue of “presenters coming with malware on their machines", and as the registration system is on the home page and public facing, “so anytime you take a web app and stick it on the internet [attacks] are happening, and so we play a lot of defence from an external standpoint.”

Also internally, someone may trying to attack others whilst on the wi-fi, and Pope admits that sometimes “it feels like a little bit of an open range to some people.”

Secure and Reliable

All of this leads to the NOC team ensuring there is a secure and reliable internet service. There was a rumour that the wi-fi is unsecure: when I first attended the U.S. version in 2014 I was told I was ‘brave’ for connecting to the network. I asked Pope what he thought of these beliefs. He says he is on the wi-fi, and “I’m pretty security conscious, I like to believe.”

“I know what we do and I know how much effort we go into doing things,” he says. "If you're not on the wi-fi and you've joined open networks that have no password, your phone's beaconing out. You don't have to be on the Black Hat network for somebody to try to man in the middle you.”

Getting the Job

During my time with Pope and NOC team, I got the chance to solve a few questions I'd had for a while. Firstly, how do you get to be in the NOC team? Pope explains that the NOC leadership has a rare luxury of being able to go to the Black Hat partners, and ask ‘do you want to come and help us secure this network’ from those partners, and provide technology which will be used - and ultimately improve those products by flagging any issues, such as bugs or usability challenges.

The NOC team is drawn from partners depending on geography, availability, familiarity with the products being used, or even just skillset. “We've had some people who are amazing, and then they get in this type of environment and they perform poorly,” he says.

“It's not for everybody to have music and movies playing, a lot of disruptions like something's happening and someone taps on your shoulder asking for help. Some people thrive in it, and love it, and will come back. Some that I think are great analysts who just don't deliver in this type of scenario. It's not for everybody.”

For skills, there are WiFi and network switching specialists, firewall professionals, and MDM experts to handle infrastructure and endpoint security. Also sandbox analysts and CTI professionals work alongside server and open-source tooling specialists to monitor threats, while EDR, NDR, SIEM, and SOAR experts manage advanced detection and response.

Also, seasoned threat hunters and incident responders are key, ensuring things are found. Also adaptive, highly technical individuals who excel under pressure are needed.

“There's an amazing amount of talent and smart caring people and it does seem to attract kind of like the best of the best,” he says “When they bring somebody who's poor and doesn't deliver, they don't get invited back. So the strong ones stay and it ends up as a very smart group of people that you can rely on.”

NOC at the Door

The other thing that interested me is this is pretty unique to Black Hat in its visibility and transparency. The NOC is available to visit and watch, and a report is often presented at the end of the second day.

There is also a NOC at the DEF CON conference in Las Vegas, and similar setups exist at RSA Conference and smaller conferences like BSides, each with varying levels of scale.

The Black Hat NOC started its version 23 years ago, initially to provide students with a secure network for training, and it has expanded alongside the conference to ensure robust, real-time security for all attendees and infrastructure. Pope explains that whilst this is a NOC, there are some security operations going on too, where other conference centres are mostly focused on availability.

Setting Up

Finally I wanted to learn about the setup and configuration of the NOC, as I’ve heard in the aforementioned NOC report that vendors will turn up with donated hardware, during the show. Pope explains that setting up for Black Hat in Europe and Asia typically takes a few days to build, test, and secure, with registration cutover happening about a day before the conference begins.

In the U.S., where the scale is much larger, setup can extend to over a week. Surprisingly, breaking everything down is much faster and usually takes just two to four hours after the conference ends, since packing up is far quicker than the careful process of setting up and securing the infrastructure. 

Also, Corelight’s team has implemented DevOps automations and policies to streamline the setup and monitoring process. For teardown, all drives are wiped, and once confirmed, the appliances are packed back into their shipping boxes for transport.

In London, Pope explains that the firewalls and switches are in a back room, but he admits that “it is a lot of work, but we get better at it.” He says that many logs and DNS logs are going to the Palo Alto Networks’ extended security intelligence and automation management (XSIAM), “and the APIs between all of these things is incredible, as we have got an integration in almost every direction of these tools, and some of that is super easy - we have a SaaS product and it stays up all year round.”

Pope says some companies do deliver products and there are challenges, such as the IP changed, or it didn't work as expected, or there is an API update. He says: “So not only are we looking at it from a security side but a network operations side [to know whether] the other tools doing what they're supposed to be doing.”

The Black Hat NOC leaves quite the impression. From the outside looking in, it is temporary walls, metres of cables, heads focused on screens whilst The Matrix plays on a big screen in the background. However its operation and short life is to be admired, and as Pope says, even if it is only working for a week, “we’re running a proper SOC in some scenarios such as closing tickets and hunting things.”

Dan Raywood Senior Editor SC Media UK

Dan Raywood is a B2B journalist with more than 20 years of experience, including covering cybersecurity for the past 16 years. He has extensively covered topics from Advanced Persistent Threats and nation-state hackers to major data breaches and regulatory changes.

He has spoken at events including 44CON, Infosecurity Europe, RANT Conference, BSides Scotland, Steelcon and ESET Security Days.

Outside work, Dan enjoys supporting Tottenham Hotspur, managing mischievous cats, and sampling craft beers.