Carbon Black: How to set up a threat hunting program

Instead of sitting back passively and waiting for cyber attackers to set off alarms, organisations should be pursuing them like a cheetah hunting for its next meal. We know the attackers are out there – they are perpetually trying to break in, and many are succeeding.

The challenge is to start hunting them to find the shreds of evidence they invariably leave behind. First an organisation needs to build a hunting team. Team members should be knowledgeable about the internals of the operating systems (OS) found in their endpoints. The OS will usually be Microsoft Windows, but also Apple Mac OS and perhaps Linux. Threat hunters need to know how these OSes work at a detailed level, including the following:

✓ OS process tree structure ✓ Files used by the OS ✓ Registry used by the OS (Windows only)

Expertise at this level of detail is important because malware operates within these domains and makes subtle changes to the OS. Threat hunters need to understand what to look for and what 'normal' looks like at the business application and human‐activity level — it's not just about packets on the network and processes in the OS, so anomalies will be more apparent. Those anomalies are the primary sign that malware is lurking in endpoints.

Next it's time to consider who has these expert skills and work on bringing them into the team. Depending on the organisation, its team may be one person or an entire crew.

Making time to threat hunt

It might be necessary to carve out time from the work schedules of existing staff for threat hunting. Depending on an organisation's size, the time spent threat hunting may vary. In part, it depends a lot on security posture and risk tolerance.

Start with two to four person/hours a week dedicated to hunting. When the results emerge,  adjust as needed. It is important to see early results from hunts, to show a return on the time investment.

The chosen threat hunters need to have passion! They must think like predators and have a hunger to hunt adversaries. After that important characteristic comes other trained skills including:

✓ Operating system internals: This skill is critical. Threat hunters need to understand the rules and practices of process management and also the file system operation and network communication in each operating system in use.

✓ Endpoint application behaviour: It's important for threat hunters to understand how any locally used applications function on the organisation's endpoints.

✓ Threat hunting tools: The team needs to understand thoroughly how to use the tools at their disposal, to maximise their effectiveness.

✓ Incident response procedures: They need to know what steps to take when they discover signs of intrusion, then preserve that evidence for potential future legal proceedings.

Put processes in place

Threat hunting needs to be a structured, long‐term effort. There must be a vision for what threat hunting is about and how it works with other IT and IT security processes. This means learning several things, including:

✓ Endpoint baselines: The need to hone continuously threat hunters' knowledge of what constitutes 'normal' in the endpoints, so anomalies can be recognised more quickly.

✓ Improving hunting tools, practices, and skills: Hunts must become more effective over time, and threat hunters must learn quickly from the seasoned warriors on their team. Much of this is tribal knowledge, but the process requires a knowledge base so that each new threat hunter can stand on the shoulders of predecessors.

✓ Improving response: Finding prey requires response that includes containment and remediation. Mainly, this means doing these things more accurately and faster.

✓ Improving skills: Threat hunters need to improve their skills and knowledge, not just from threat hunting itself, but from continuing education on ethical hacking, system and network internals, and incident response. Threat hunters must understand what's 'normal' in an organisation so they can quickly identify anomalies that may signal intrusions. The local context that humans have makes all the difference in detection.

Put the necessary tools in place

Threat hunting is a man‐machine activity — it cannot be done with just people or just machines. Without threat hunting tools, there's no hunt.

Endpoints are today's battleground where intrusions into enterprises begin. Endpoints are the attackers' crown jewels, and they're used to make a landing into an environment. While the data that attackers seek lives on servers, access to servers starts with endpoints.

Endpoint visibility is the ability to capture, in detail, the activities going on inside of every endpoint:

✓ If an organisation allows Bring Your Own Device (BYOD), it has to achieve this visibility on those machines, too.

✓ Include information about every process, including its parents and children, as well as every file that's created, read, written and removed, plus network activity. This information needs to be accessible across the entire organisation, so threat hunters can quickly understand what anomalous activity is going on at any place and time.

✓ Another important aspect of endpoint visibility is known as retrospection, which is the ability to hunt back in time. For example, mine the data for suspicious activity that took place not just yesterday, but last week, last month or even earlier.

In addition to endpoint visibility, having access to network event data is essential. Sometimes the first sign of intrusion is in the command and control (C-C) network traffic from a bot that has already compromised an endpoint. Intrusion prevention systems (IPS), web filtering, firewall logs, and netflow tools are good sources for obtaining this data. Threat hunters must be able to reference one or more of these tools from time to time, to better understand what's going on in the network.

Threat intelligence feeds inform threat hunters of the new tools and techniques that attackers are using against other organisations, as well as the domains and IP ranges they may be using. Threat intel feeds are often high volume and delivered in structured formats such as Structured Threat Information Expression (STIX) and OpenIOC (and Cyber Observable Expression [CybOX]). All these are designed to be fed into an organisation's security information and event management (SIEM) system or other threat management platform.

Remember that threat hunting is a man‐machine activity. In many respects, there is a high volume of information on threats and activities in your environment. To capitalise on this information, the threat team needs to understand what tools they are using and where there might be opportunities to integrate them.

A prime example is the fusion of endpoint data, SIEM data and threat intel feeds. By themselves, they're useful, but when fused together they become invaluable. For instance, threat intel feeds often use STIX, TAXII, or CybOX for structuring this data. APIs for these are available so that threat hunters can consume this data and get it into their other systems.

Because threat and event data arrives from many different places, the tram needs to be able to perform event correlation and analytics to make sense of what's going on in the environment. The tool of choice is SIEM.

SIEM systems are made for event correlation and analytics, and they do a good job. They're often used as a central repository for log and event data from network devices, firewalls, operating systems and applications. It's the storage for everything going on in an environment, together with the ability to make sense of it.

Know the environment

Successful threat hunters need to know as much about their environment as possible, so they can better sense what's normal and what's abnormal. As their hunts progress, they begin to have an intimate familiarity with their environment.

Threat hunters spend much of their time observing and becoming more familiar with normal routing events in their environments. However, they also need to be familiar with the organisation's architecture: networks, systems, tools and applications.

Chiefly they need to understand this independently of their threat hunting, because anything they might observe in the environment may or may not be normal. What they find and consider normal may include things that aren't allowed.

Threat hunters need to know what their goals are. Depending on the attackers and their objectives, this could be information like customer or employee data, or it could be critical assets such as public facing web servers. They need to know all these high value targets (HVTs) – and they need to understand how cyber criminals might go about attacking them.

Threat hunters also need to know how attackers are likely to try to break into their environments. This is part gut feel and part knowing the environment:

✓ Architecture: Attackers will seek out the weak spots in an organisation's architecture and data flows. This helps them discover whatever valuable data they're seeking and how to extract it unnoticed.

✓ Security posture: Attackers will target an organisation's weak spots. They discover these through simple techniques like port scanning to find unpatched and vulnerable systems. So threat hunters need to know where those weak spots are.

✓ People: An organisation's security culture is a great indicator of its vulnerability. While attackers might not have access to security awareness training or other aspects of a security awareness program, the they will be able to gauge how easy it is to lure employees into clever social engineering, phishing or spear phishing campaigns, whether they're purely online or on-site.

✓ Threat intel: Understanding how attackers are targeting other organisations gives threat hunters a better idea of how attacks might target their own organisation. While they will be creative and unpredictable at times, attackers are creatures of habit, apt to use tools and techniques that have worked for them in the past. Just as organisations tend to protect themselves in similar ways, attackers are likely to attack in similar ways.

Finally threat hunters need to know their environment inside and out: How does everything work, where are the gaps and weak spots, and where are the risks? They need to think like attackers, so they can better anticipate threats and stop attacks early.