The application security space has grown up. The focus shifted from security at the perimeter to security where the attacks are actually happening. As attacks on applications proliferate, it’s become abundantly clear that there is a real problem in the software we build. We as modern companies have a lot of applications: legacy applications, applications we don’t even have the source to anymore, and applications we’re unable to modify due to a lack of resources.
Over the past two and a half years, our team has focused not only on building the bleeding edge in application security technology but also on making it easier than ever to integrate into all kinds of applications as seamlessly as possible. Richard Meester and I recently had the opportunity to present at OWASP’s AppSecUSA conference in San Francisco and share with the the attendees how we at Prevoty do what we do. This post summarizes what Richard and I covered in our talk, which details our techniques for using middleware and instrumentation as methods for introducing tooling into applications and improving security.
The Application Security Problem
For every application, we have a vulnerability backlog. While we’re painfully aware of some of the vulnerabilities that exist in these applications, there’s little we can do. Maybe the application has been put out to pasture until it can be replaced, or maybe hasn’t been deemed worthwhile to fix it based on the cost of doing so. At any rate, vulnerabilities still exist and put the organization at risk.
Even developers using best practices and being security-conscious for non-legacy applications will always be at a disadvantage. Good attackers will typically have access to newer and custom tools that the developers have never seen before, new techniques, and will always be ahead of the defense’s signatures. In effect, they will always be fighting a losing battle.
There are a lot of current strategies out there currently being used but few take into account the application itself. Most focus on isolation/containerization -- watching the application from the outside and looking for anomalies or seemingly dangerous behavior. They aim to contain a breach, accepting that it’s going to happen, or watch closely from the outside until some action is taken that is considered bad and sound the alarms.
Some of these techniques are:
- Endpoint protection
- Network Firewalls
Our Simple Solutions
As I mentioned earlier, our team focuses on making our security integrate into applications as seamlessly as possible. In the beginning, we distributed our tools simply as APIs that engineers could use to provide security to their applications. We quickly discovered that this path came with too much friction and we had to build something simpler.
Our goals became:
- Easily able to remediate known and unknown vulnerabilities
- Know when exploitation of a vulnerability is attempted
- Get the Ws (who, what, where, when)
- Introduce a minimal impact to performance
- Introduce no changes to the application itself or recompilation
- Low false positives and negatives
This isn’t to say that teams should depend on these solutions as a replacement for best practices and good engineering. Instead, it should act as a stopgap until a proper solution can be set in place. To do this, we shared two techniques: middleware and instrumentation as methods for introducing tooling into applications.
Many frameworks offer the ability to create middleware (or essentially plugins) that can be used with the framework. Typically they offer callbacks or hook points for various actions (e.g. a request coming in, a response going out, etc.) where you can simply provide a function that will get called when these events happen. The application will then allow that function to take control until yielding control back. This means incoming requests will block while control is transferred, so it’s important to keep in mind the performance impact and time for processing that these functions will introduce. The overall performance impact of the function itself is negligible -- it’s essentially just another function being called -- but the work done within can slow down a request or back up a server. It’s also important to keep in mind the effect these functions will have on the gc (garbage collector). A function that introduces a lot of garbage will likely cause gc pauses much more often.
Use cases we’ve specifically seen for these types of integrations are:
- Adding security headers (CSP, HSTS, XSS, etc.)
- Validating correctness of headers (content-length, mime-type, etc.)
- Inspecting/Modifying GET/POST/etc parameters
- Forcing HttpOnly and secure flags on cookies
- Inserting and validating CSRF (Cross Site Request Forgery) tokens
This is great, but what happens if the framework doesn’t provide nice middleware hooks for the vulnerability classes you’re interested in preventing? Or if you’re not using a modern framework because you’re maintaining a legacy application? In those cases, we can build our own. Most modern languages -- especially ones that run on virtual machines -- expose instrumentation/profiling and debugging APIs that let you do just this. If you’ve ever wondered how a debugger or profiling tool does all the magic it does, this is usually how.
Just like the nice middleware callbacks that can be registered in frameworks, these APIs usually allow listeners to be registered for various events (eg. classes being loaded, objects being instantiated, gc starting/ending, etc.) and typically give access to all sorts of low level functionality such as changing the byte code before JIT (Just in Time compiled). This functionality will differ depending on your platform but should offer a good starting point. But be warned, going down this path opens the doors for some pretty nasty bugs and the ability to cause some serious damage. The documentation is usually not up to par with the rest of the standard library, so you may find yourself experimenting quite a bit until you’re able to get a solution working.
Some of the use cases we’ve found for this functionality are:
- Network access
- File access
- Exec calls
- Database access
- Permission changing
- Content validation
These are behaviors that are traditionally monitored from outside of the application by relying on the kernel to notify of events or watching from within a sandbox as the application attempts to call out to the kernel to perform these actions. The downside to that technique is that so much of the context of what is happening in the application is lost. By doing this in the application, requests can be gracefully terminated, more information can be accessed, and attacks can be caught earlier.
Identifying and Inserting the Hooks
The key part to this technique is identifying what to hook. To do this, you’ll want to identify choke points in the API. For instance, to detect all output going to a console, it’s possible to go through and try to identify every function that will write data to it in the standard library -- but what about the external libraries brought in? It’s easier to look lower: find the low-level library functions or system calls that all the others filter down to. This reduces the amount of modification required and limits the scope. Reflection, or the ability to use the type system to introspect on objects and classes and make decisions from it, should be your best friend. As you traverse the various classes and objects, look at the class names, types, namespaces, and whether a particular class/object implements an interface or subclasses another class that you’re interested in. This can make identification much simpler.
Once you’ve identified your targets, there are two ways to insert the hooks: updating the caller or the callee. Callers are points in code that call the specific function you’re interested in. It’s possible to update all the callers to point to a new function that performs your checks, but this is not only more work, but it also requires updating a lot more code -- which in turn increases the potential to break something. Instead, focus on updating the callee with one of two ways: method modification and method replacement.
Depending on the language, you may be able to modify the code itself if it’s not marked as read only. This allows you to rewrite the function to add additional code. You can either opt for trampolines (inserting a stub that immediately jumps out to the code and performs your intended action, then returns to the original code) or simply insert your code directly in. The more elegant solution, however, is method replacement. Typically, most languages store a pointer to the function that’s being called and all instances of an object refer back to that original pointer. By simply replacing the pointer and pointing it to a shim class that works the same way as the aforementioned trampoline you avoid changing significant amounts of code and potentially avoid issues of read only memory.
As with middleware, these method replacements introduce negligible performance impact into the application where the only additional work is an extra stack frame being allocated for the function call. It is the processing itself that is introduced in these new functions that can introduce serious performance impacts and create issues. Any work done here will be done in a blocking manner unless a new thread is explicitly created and the work is farmed out to it (assuming the processing does not need to be able to stop the call). If this is the strategy you intend to take, make sure you understand the threading model of the language and the impact that an extra thread running concurrently (and potentially in parallel) can have on the application and runtime. Be sure to consider the memory aspect of this processing as well and the effects that it will have. In a garbage collected language will the additional garbage cause more gc cycles that force the runtime to pause to clean up. The one point where there will be a negative potential impact to performance is in startup time depending on the work being performed to facilitate the actual hooking. Depending on the implementation in the language doing the walk of the code, identifying locations that are interesting to hook, and making the actual changes could increase startup time but once this penalty is paid it is done.
Secure at the Source
To wrap up, it’s time to focus on the application itself and the vulnerabilities that it contains. External solutions aren’t enough and will likely not have access to enough context to provide intelligence or make a good choice about specific data entering or leaving applications. If you can, utilize built-in middleware stacks to add your own protection mechanisms in. This is no substitute for high-quality, well-written code but it can help with identifying and stopping threats until a full solution can be written and put into place. If a middleware stack doesn’t provide you will the hooks necessary, build your own and place the checks into the application -- but do so by finding chokepoints to minimize the impact. Focus on changing as little as possible and make sure to test everything. After you, you’re playing with fire.