Research - Gold
National Institute of Standards and Technology - Summer Undergraduate Research Fellowship
Abstract
When designing new methods for software quality analysis, we desire certainty that they are able to detect known classes of bugs. To this end, we have datasets comprised of programs with known vulnerabilities to test new methods on. One such dataset is the Intelligence Advanced Research Projects Activity (IARPA) Securely Taking On New Executable Software of Uncertain Provenance (STONESOUP), a suite of 7770 cases where industry programs written in C or Java had known vulnerabilities injected into them. These vulnerabilities are selfcontained “cysts" of about one page of source code each. One generalized method of analyzing programs like these is static analysis. This method focuses on examining source code for possible bugs without limiting the possible executions to a finite number of runtime tests, and is aided by the use of tools such as Fortify and Frama-C. These tools are examples of static analyzers that, like most, use bug-finding heuristics. As such, they catch or don't catch bugs for seemingly explained reasons. We want to gain some assurance that injecting stand-alone code with known vulnerabilities, as with IARPA STONESOUP, is a good approach to estimate how well an analyzer does in reporting existing, real bugs. Our plan was to first run the tools on all the cysts as standalone code, to get baseline results. Second, we run the tools on all the test cases, which have the cysts injected. Third, we compare the results. We found that running the tools on the full test cases required far more work than expected, and required at least 5 hours per test case. Thus we ran only a couple dozen examples. For these cases, we found that they yielded expected results, indicating that it is reasonable to assume that, given its cysts, IARPA STONESOUP can produce intended results.
Commitment and Connection
My research experience was a summer research fellowship at the National Institute of Standards and Technology (NIST). This project took place over 11 weeks in the summer of 2016, through the Summer Undergraduate Research Fellowship program. My project was in software quality assurance, specifically in the vetting of test cases used to evaluate source code analyzers. The connection to my challenge of securing cyberspace is immediate; it will aid in the development of better tools for finding vulnerabilities in cyberspace in order to patch them before they can be exploited. The project also contributed to ongoing work in automated vulnerability detection that can be used to safeguard vulnerable code before it can be exploited.
Reflection
My research experience is, to my mind, by far the most enlightening experience that I had. It presented an interesting mix of research and industry, allowing me to experience portions of both possible career tracks at the same time. I got to see both how open-ended research is, and how many different components go into a team in industry. Going into this experience I had very different preconceptions of both worlds, imagining research as being somewhat disconnected from practical concerns and industry as being stifling and money-obsessed. All that being said, I think the biggest revelation for me was how much I value a team environment. Previously, I had been always thought of myself as very independent, a self-teacher even; but in this experience, where most of the work was individual but there was a strong community to meet and discuss things with at lunch, it was not hard to tell which I preferred. I guess this is where the benefit of it being industry research comes in for me; more than research, and more than technical skills, this experience taught me the value of having an inviting team environment.
Learning Objectives
Integrity - This was an independent project, largely self-guided and partially self-designed. This lack of oversight necessitated a level of personal integrity to stay focused on my project and put a good faith effort into completing as much of it as I could within the time span.
Perspectivism - As the entire summer was spent on a single project, which even then could only be completed to a certain extent, it put the magnitude of actual industry tasks and changes into focus. This helped to highlight just how many perspectives there will be on this challenge, as well as just how much of a role I'll be able to play in it.
Realistic vision - The project was open-ended to begin with, with any number of different avenues available to explore. This required me to isolate a reasonable portion of my options to actually investigate, and narrow the scope of my project to something realistic to complete within 11 weeks.
Teamwork - I started the project not knowing how to use the necessary software, much as one of my coworkers started her project not knowing the theory behind memory management; over the course of the summer we both were dependent upon contributions from other coworkers to learn how to do our own projects, highlighting the necessity of working as a team to accomplish any new project.
Persistence - As this was an original research project, I was attempting to run an evaluation that had never been done before; not everything worked right the first time, especially given that I was unfamiliar with the technology that I was using, and I had to be willing to adjust and try again as I figure out how to do what I was expected to do.
Flexibility - Not just did the multiple failures require persistence, they also required flexibility as original plans proved to be impossible. Some of the originally planned tests could not even be completed at all, which required modifications of the project in the middle of the internship.
Research Learning
Express ideas in an organized, clear, concise, and accurate manner.
The conclusion of the experience was a colloquium presentation in which I had to give a concise, organized presentation of my work and my results before a professional audience.
Write clearly and effectively.
As my project was one part of a large ongoing framework, I
was required to clearly document all of my work and results in writing so that it could be referenced by the rest of the working group after I left.
Effectively connect multiple ideas and approaches.
The mechanics of the work involved a practical approach, requiring ideas from shell scripting and batch optimization. Meanwhile the interpretation of the results was more theoretical, requiring ideas from formal language theory and compiler methodology. The project involved both; the system management aspects of generating results and the mathematical aspects of interpreting the results.
Formulate questions and hypotheses within the discipline.
While the initial description of the project was provided by my mentor, it was up to me to figure out how to complete it; accordingly, through research and conversations with coworkers, I had to learn how to ask discipline-appropriate questions that would provide answers relevant to my desired results.
Understand how practitioners think within the discipline and view the world around them.
As part of my project I was required to attend weekly working group meetings in which I heard about the progress on different projects throughout the department.
In the course of doing so I learned what was relevant to be studied and to be included in such presentations, and accordingly the modes of thinking and worldview that would motivate such choices.
Predict, recognize, and weigh the risks and benefits of the project for others
In the aforementioned working group meetings I heard presentations about submitting proposals to continue related work, along with the benefits and risk mitigation included in such proposals; I was then able to adapt what I heard from related proposals into potential risks and benefits of my own project.
Demonstrate the ability to work independently and identify when input, guidance, and feedback are needed
As my project was an independent project yet was still dependent upon utilizing the existing research infrastructure, I was required to do most of the work independently while still identifying when I needed guidance about whether or not my ideas were feasible given the existing infrastructure.
Display accurate insight into the extent of their own knowledge and understanding and an appreciation for what isn’t known
As my experience involved weekly team meetings with
both professionals and other interns, I got the chance to see from a variety of perspectives what my background is in relation to my coworkers and the field in general; how far my current knowledge can take me and what there is that I don’t know are immediate corollaries of this.