Systems Engineering Thesis Research Methods
Ronald Giachetti, PhD
Department of Systems Engineering
Naval Postgraduate School
September 13, 2016
Navy curricula sponsors have consistently told the Naval Postgraduate School (NPS) they want stu-
dents to write a thesis. A thesis is the final step toward earning the MS degree and establishes a student
as a professional able to engage and lead others in technical endeavors. The purpose of this document is
to describe what a thesis means in the Systems Engineering discipline.
To start let’s define a thesis as a statement put forward as a premise to be proved based on engineering
research. The NPS proposal asks for a research question rather than a statement, which may be confusing.
However, this is a matter of semantics because a research question is nothing more than the thesis
statement posed as a question. The definition of a thesis as a statement is useful to guide us because
it emphasizes the fact that a thesis is a claim - i.e., the statement made by the author upon which the
remainder of the document argues for or against. A thesis writer needs to approach the task from the
perspective that he is trying to persuade the reader that a thesis is true or false as the case may be.
Importantly, the thesis writer must substantiate the claim with evidence and data generated from the
selected research method, critical thinking, and data analysis as appropriate. The thesis is novel in that
you are addressing a new problem or claim, or you are approaching an existing problem with a different
underlying theory or a different research method. Finally, the thesis statement should be of interest to
the larger community including the Navy, defense, and systems engineering communities.
Appropriate topics for a systems engineering thesis can be almost any topic in the systems engineering
domain. A thesis can address a particular problem faced by the systems engineering community or a subset
of the systems engineering community. A thesis can seek to advance the methods, techniques, or tools
used by system engineers. Regardless of the topic, a thesis should advance some small portion of the body
of systems engineering knowledge.
The thesis is often a student’s most challenging requirement because it is a singular, individual, and
student-led effort. The student must identify a thesis topic, determine an appropriate research method,
conduct the research, and document it. The timeline is demanding. Importantly, a student must leave
sufficient time for iterations of review and feedback from the advisor, second reader, thesis processing
office, and department chair. By thinking ahead of time about the nature of the thesis, thinking about
the research methods available, and envisioning how you will substantiate your thesis claims, then you
are more likely to succeed.
1 Engineering Thesis Research Methods
A thesis writer has a choice about the research method where a research method is the means the
student uses to generate and/or obtain data, results, and evidence to support the thesis statement. The
research method can be experimental, design, empirical study, analysis, or a combination of two or more
of these methods. It is via the research method that a student contributes new knowledge to the systems
engineering body of knowledge. The following subsections describe each research method.
1
1.1 Experimental Thesis
The experimental approach to a thesis is the most traditional research method in the sciences. The
experimental approach essentially applies the scientific method to an engineering question. The thesis
statement in an experimental thesis is a hypothesis. The thesis states a hypothesis, examines the relevant
literature to motivate the hypothesis, describes an experimental procedure to test the hypothesis, analyzes
the results of the experiments, and makes conclusions whether the hypothesis was rejected or not rejected.
Experimental types of thesis are currently rare in systems engineering, but they do not need to be.
Here are some illustrative hypotheses to illustrate the use of the experimental approach to many subfields
within systems engineering:
• Model-Based Systems Engineering (MBSE): Engineers using SysML Activity Diagrams to
model scenarios are able to document more system requirements than engineers using Functional
Flow Block Diagrams to model scenarios.
• Combat Systems: The material xyz can absorb sufficient laser energy to delay damage.
• Logistics Area: Rapid prototyping of spare parts on the ship is more effective than alternative
spare part policies.
• System Control: A unmanned aerial vehicle using abc controller leads to more accurate and
precise control than a unmanned aerial vehicle using xzy controller.
• Cost Estimation: The COSYSMO model can better estimate ship costs than current methods.
• Ship Systems: A xyz hull shape has better sea-keeping ability than a abc hull shape.
In the above examples, each thesis statement is specific, clear, and concise. The thesis statements
are specific to the particular topic or problem being addressed, they are clearly written so the reader
knows what the author sets out to prove, and they are concise since they do have superfluous text or
qualifications.
The experiments to collect data to support the thesis statement can be performed with people, lab-
oratory equipment, models, or simulations. For example, the MBSE hypothesis can be tested by having
people such as other students use SySML and FFBD to model scenarios. The combat systems example
can perform laboratory experiments with a laser. The logistics example can create a simulation model
of logistical policies for spare parts. The system control example could fly UAVs with the different con-
trollers. The cost estimation example can compare cost estimations from COSYSMO vice other cost
estimation methods. The ship system example can test hull shapes in a tow tank. The point is there are
actually many interesting and doable experimental theses in systems engineering.
Experimental thesis can be rather short in terms of the size of the document. The novelty in an
experimental thesis comes from the design, execution and analysis of the experiment. The evidence is
partially in the form of the results and partially a logical argument that the experiment adequately tests
the hypothesis.
1.2 Empirical Thesis
Empirical research is based on posing a research question or hypothesis and collecting data from actual
projects or engineers to accept or reject the hypothesis. Data collection can be via surveys, interviews,
case study, or document collection. The systems engineering discipline is in sore need of empirical research
to substantiate whether our processes, models, and techniques are effective, under what conditions are
they effective, and to answer many other questions about the practice of systems engineering.
As mentioned, there are countless topics for empirical research. As one example in the subfield of
system architecture, a research question could be on the usefulness of required DoDAF products. The
2
student could construct a survey to assess the usefulness of DoDAF products and send it out to the
engineering community. An analysis of the data could justify which DoDAF products are found useful,
why they are found useful, and what they are useful for. The value of such empirical data is obvious - it
justifies whether or not these products should be mandated by the DoD.
Unfortunately for the average thesis writer, the time required to conduct empirical research may be
longer than the time available to complete a thesis. To conduct empirical research, a thesis student needs
to have a research question, prepare an instrument for data collection (e.g., a survey), obtain IRB approval
of the instrument, send out the instrument, wait for responses, and then analyze the data. Much of this
time is non-productive waiting time. A motivated student who identifies an interest early could possibility
do the research question and instrument before getting to the last few quarters, and hence leave sufficient
time to finish.
A more schedule friendly empirical study is a case study. This may be appropriate for students who
worked on a program or students who have access to data. A case study is a single, but rich data point
that contributes to our knowledge of systems engineering. It may be possible to do a case study, in which
you the student do a task in two different ways and remark on what is learned in the process. For example,
two different approaches to analyzing system reliability, and writing a detailed analysis of the pros and
cons of each approach.
The empirical research approach is a great way to collect data because engineering is a human activity
and the empirical research method collects the data from humans performing the engineering. While the
timeline for a thesis does not support large projects, the empirical research approach can be applied in
combination with other research methods. Commonly, some type of survey or interview of subject matter
experts is done in a thesis to support data gather through other research means. In this way, the empirical
research method can greatly enhance a thesis based largely on another research method.
1.3 Design Thesis
Engineers design artifacts, and an acceptable thesis research method is one in which the student designs
a system or a portion of a system. Typically such a design thesis is in support of a larger, ongoing
project at NPS. As an example, we have had students design a UAV launcher to support the multi-year
project in swarming UAVs. The artifact under design could also be a systems engineering tool or model.
For example, a student could design and develop a software program to partially populate a Test and
Evaluation Management Plan (TEMP) from a set of DoDAF models. Methods and tools to partially
generate many of the deliverables in a program from DoDAF products is a fruitful research area since one
of the criticisms of DoDAF is its poor support of program activities. The design artifact can be hardware,
software, or a process.
An important aspect of design research is building a prototype of the design. This is a necessary and
important step in order to do testing and evaluation of the design. In fact, the design research method
is incomplete without some sort of prototype to provide data and/or evidence of its functionality and
performance. The advent of rapid prototyping capabilities for mechanical as well as electrical systems
makes the design type of thesis more practical for a larger number of system types. NPS has the RoboDojo
lab with 3D printing, laser cutting, and many other tools to support the rapid prototyping necessary in a
design type of thesis. A design thesis describes the design problem, describes the design process, describes
what was learned during the design process, and documents the success or failure of the prototypes
developed and built to test the design ideas.
1.4 Analysis Thesis
The analysis research approach is very common in engineering theses. An analysis research method for a
thesis is one in which the bulk of the thesis involves quantitative or computational analysis of a problem.
For example, a research question could be, “Is it technically and operationally feasible to deploy a laser on
a unmanned air vehicle?” The thesis would then try to answer the question by determining the technical
3
feasibility and operational feasiblility. The contribution of an analysis type of thesis is due to the depth
of the analysis conducted and its ability to address the research question. The novelty of an analysis type
of thesis is in the research question, the depth of analysis, and the discussion of the results. Frequently,
a thesis based on the analysis approach can be greatly strengthened within an experimental framework
and/or with statistical analysis of the data.
1.5 Combination of Research Methods
Oftentimes a thesis will use two or more of the research methods. This can be useful because each research
method generates different types of evidence, which collectively can reinforce the overall thesis statement.
2 Thesis Characteristics
This seciton reviews thesis factors or thesis characteristics leading to success. The intent is to help the
student differentiate between those characteristics that lead to a good thesis from those that lead to a
poor thesis.
2.1 Specific and Precise Thesis Statement
A good thesis defines a narrow, specific thesis statement and then explores the topic in depth. Many
students mistakenly believe the broader the scope of the thesis, then the better. These students attempt
to do too much and the result is a thesis that lacks substance. In systems engineering common pitfalls
include when students try to include too many systems or alternatives, they try to analyze too many
aspects of the problem, or they try to do the entire sysetms engineering life-cycle. This is one of those
cases where more is not necessarily better.
2.2 Maintain Focus on Thesis Topic
A thesis that loses focus is related to but subtly different than too broad of a topic. An unfocused thesis
has a clearly defined, concise, and narrow topic, but dilutes the main message of the thesis by addressing
too many ancilliary topics, which are (or should be) out of scope for the thesis. Part of thesis writing is
knowing what to delete as much about what to write in the first place. A good thesis remains focused on
the single claim the thesis is attempting to demonstrate with logic and other collected evidence.
2.3 Topic Your Advisor Has Expertise In
The thesis is an individual student-led effort, which is performed with the guidance, advice, and eventual
approval of a thesis advisor. You should work in an area your advisor has some expertise in. Picking a
topic in an area with which the advisor is not knowledgeable carries risk. The risk is that you work on
solving a problem without knowing how it has been approached before, perhaps your solution approach
is known to be faulty by experts in the area, or perhaps you are missing important information, methods,
or techniques that an advisor with knowledge in the field could have directed you to. Ideally, an advisor
would decline to work with a student on a thesis they were not expert in, but it is the responsbility of
the student to stay within a topic their advisor has the qualifications to advise.
2.4 Avoiding the Conceptual Thesis
There really is no such research method called a conceptual thesis, and this is why it was not mentioned
above. While all thesis should make extensive use of logical arguments to support the thesis, it is unlikely
an entire thesis could be supported soley by logical argument. Moreover, a thesis that solely involves
modeling or developing a conceptual framework without any data is almost always insufficient for a
4
thesis. Some evidence from experiments, design, empirical data, or analysis is needed to help substantiate
the thesis.
2.5 Straying Far from Accepted Systems Engineering
Only the best students with strong convictions about their intended topic should even think about straying
far from systems engineering accepted knowledge, methods, processes, and techniques. In most cases,
students who stray from the larger body of knowledge end up with a poor thesis, and in some cases they
are unable to complete the thesis (perhaps because nobody in the department is willing to sign off on
the thesis). Remember a thesis must be substantiated with evidence derived from one of the research
methods: experimental, empirical, design, or analysis. As an example, it is likely risk management models
and practices are not perfect, but if your thesis idea is a complete departure from current risk management
methods, then it is unlikely anybody will sign off on it unless you are able to provide substantive and
compelling evidence to support it.
2.6 Statistical Thinking or Data Analysis
Most research methods generate data, and just having the data is insufficient. The thesis should use
appropriate experimental design methods, statistical methods, and data analysis methods on the data.
Experimental, empirical, and analysis types of thesis benefit from proper statistical analysis. In a design
thesis, you need to actually build some prototype to test the design and data might come from the
prototype. Again data analysis would be necessary.
2.7 Validation
In a thesis, you make a claim, you perform engineering research to gather data to prove it, and you validate
your method, data, and results. A thesis that makes a claim and offers no validation of the data is a
rather weak thesis. Validation is not a yes or no type of proposition, but instead should be viewed as the
degree of confidence a person has in the research results. For example, if you use simulation experiments,
then the validation question is how well do the simulation results match the data you expect from actual
systems? The most convincing validation technique would be a statistical test such as the student t-test
comparing simulation results to known actual results. However, if actual data is unavailable, there are
other validation techniques such as face validity via subject matter experts, boundary testing, and so
forth. The bottom line is every thesis needs some validation via data and other evidence to support
the claims and evidence. All the thesis research methods have valiation. Experiments generate data for
validation, empirical studies produce data for validation, design thesis create prototypes for validation,
and analysis thesis generate results for validation.
2.8 Lack of Technical Rigor
A thesis should have a rigor at least equal to the level of the analysis done in the coursework. Actually,
if the thesis topic is scoped properly, then the level of detail in the analysis should be greater than what
is done in class. The reason is you have one to two quarters to do the analysis, whereas in class you
probably had only one to two weeks to do something equivalent. A thesis is supposed to be a focused
study so it is expected to have sufficient depth to fully explore a narrowly scoped topic.
2.9 Literature Review
The literature review is an important component of your research. The literature review critically assesses
the scholarly peer-reviewed, archival journal articles, conference proceedings, books, and other publica-
tions in order to establish your knowledge of the research topic, identify the state-of-the-art in the field,
and to put your research in context of the larger body of work. A good literature review is more than a
5
summary of published works. It evaluates and organizes the published research to highlight what has been
accomplished and the open research questions in the field. By doing this, the literature review puts the
thesis research in context of the larger body of knowledge by showing how the thesis relates to previous
published works.
3 Conclusion
In this document, I intended to describe the nature of a systems engineering thesis. A thesis is a claim
supported by data and analysis derived from one or more research methods including the experimental,
design, empirical, and analysis methods. The document explained each type of research method and
provided some examples. Finally, the document mentioned some characteristics of thesis that frequently
lead to unsuccessful results. The purpose is to guide you away from poor approaches to thesis work.
A thesis reflects the critical thinking of the student and all the elements of critical thinking should be
evident. The engineering work in a thesis should be performed with a precision and attention to details
worthy of a student’s best efforts. Statistical and other data analysis techniques are important since the
data students gather or generate in a thesis is almost always drawn from a small sample. Lastly, the
writing should be clear, concise, and direct in order to best convey the thesis contributions.
The document did not discuss many important topics including how to select a thesis advisor, how
to select a topic, when to start the thesis, the NPS policies and procedures for a thesis, the issue of
plagiarism, how to format the thesis, or how to write the thesis. Information and guidance on these topics
are available elsewhere.
The thesis can be a challenge for many students, but it can also be highly rewarding. Theses at
NPS are archived in the library and DTIC, which means that soon after publication the major search
engines find and index them for worldwide consumption. Your thesis work becomes part of NPS’s large
contribution to systems engineering knowledge and application in military systems. Make it a good one.
6