I totally agree, this project looks really promising! I am curious also about the details that Teresa asked about, and additionally would like to hear more about what teachers see (the video used the phrase “automatically analyze data”) and how they use that information. Is there a teacher professional development program that goes along with these virtual internships?

The teachers use a platform to send messages and feedback letters to their interns. For example, when an intern submits the results from one of their design tests, the system evaluates the values and composes a feedback letter that gives the interns suggestions about what’s working and what could be improved.

Later, when the interns have identified their optimal design, they draft a proposal outline prior to completing a final proposal. The teacher uses the platform to evaluate the outline using a rubric tool. The selections the teacher makes generate a prepared feedback letter that can help interns improve their argument and use of evidence for how and why their design is a good solution to the problem.

The engineering units come with several supporting documents to help teachers implement, but at this time there is no formal professional development.

To follow up on the assessment questions, to shed light on engagement, we are collecting event data (clicktream and student submissions) and back-end data from student use within the Design Tool—this will give us things like how long students are using different features of the tool, as well as the specific moves they make within it—the expectation is that this will help us (and help teachers) learn, for example, whether students are making meaningful moves within the digital environments vs just randomly clicking buttons. Also, I’m not sure if this was a part of the engagement question, but we are using the Fascination scale from the Activation Lab suite of measures (see http://www.activationlab.org/ for more detail).
For the “automatically analyze data” question, we are building an analytics framework to draw inferences from all of the data we are collecting (see my response to Jennifer Adams, below, for more on that approach). Our goal is always to provide actionable information to teachers, based on these data—we are building a reporting infrastructure in conjunction with the analytics approach to digital data so that teachers can easily see and make use of the insights from student work in the digital environments.

Jackie,
There are so many wonderful qualities in this program, but I’d like to focus on one that really captured me. Your internships engage students in current societal problems. These experiences are preparing students—creating the state of mind—to contribute to the new, hard questions that society will face in the future. Congratulations!

Thank you, Sue! It has been our goal to enable students to take on a problem-solving role related to real world problems, something that engineering easily affords. Student engagement has been super high. I really liked the comment by one teacher about how her students seemed more hopeful after participating in tackling climate change.

Thanks, Jennifer! Our approach is grounded in Evidence-Centered Assessment Design (from Mislevy and colleagues). To that end, our basic question is, what would count as evidence of understanding in student task performance (in this case, what they’re doing within the digital Design Tool)? This involves careful articulation of the specific practice/understanding we are assessing (e.g., optimizing a design solution), and specifying what moves students could make within the Design Tool environment to show evidence of developing that practice. We are currently working through a combination of top-down (e.g., expert models) and bottom-up (e.g. machine learning with student data) approaches to building an analytic framework that yields useful, formative inferences about student learning for the teacher and student.

Thanks! It will be interesting to compare the outcomes of each (top-down vs bottom-up)

Hi Teresa, We expect these units to be part of a middle school science course.

Hi Helen, We recognize there are a variety of Design Cycles/Processes out there. We’ve simplified the Design Cycle to Plan, Build, Test, Analyze… Repeat.

Thank you, Jane! I like the emphasize on the action words!

Thanks! There’s notable buy-in because we ask students to consider problems that are current and in the news that affect humans around the globe. We’ve seen that some students really engage in the internship fiction, but it depends on how the teacher implements the immersion using internship language, referring to the project director, or modifying other classroom routines.

The Futura Engineering Work Space is browser-based, so students can you any kind of computer device. Students work in pairs on the design challenges, though teachers often have students submit individual proposals at the end of each internship. Those proposals involve students in mounting a design argument: this design is strong because… the trade-offs we made are… etc. Thanks, Isabel!

Thanks! These are designed for grades 6-8. Each internship addresses the NGSS ETS standards and asks students to apply some specific NGSS content in life science, physical science or earth science. There are two internships intended for each scientific domain and thus we expect students could complete two for each grade level. We correlate for both math and ELA CCSS, and because student write a final proposal, there is a lot of language arts work. Each internship has some physical experience, though not always specifically to proof of concept.

Our approach is grounded in Evidence-Centered Assessment Design (from Mislevy and colleagues). To that end, our basic question is, what would count as evidence of understanding in student task performance (in this case, what they’re doing within the digital Design Tool)? This involves careful articulation of the specific practice/understanding we are assessing (e.g., optimizing a design solution), and specifying what moves students could make within the Design Tool environment to show evidence of developing that practice. We are currently working through a combination of top-down (e.g., expert models) and bottom-up (e.g. machine learning with student data) approaches to building an analytic framework that yields useful, formative inferences about student learning for the teacher and student.
As far as the data we are drawing on for our analytics framework, we are collecting event data (clicktream and student submissions) and back-end data from student use within the Design Tool, all in conjunction with data about the digital "stage"and teacher moves at the time we log student events—the expectation is that this will help us (and help teachers) learn, for example, the extent to which students are responding to feedback from the project director and evidence from their design test runs as they iterate toward an optimal solution to the design problem.

Thank you, Eric for this explanation!

Agreed, Jewel, it’s a very challenging and real problem! For all of our internships we designed artificial models based on current scientific understanding, making simplifications so that students can see the cause and effect and apply the foundations of scientific concepts, in the case of malaria, natural selection.

Hah! Yes it probably looks very similar to our Argumentation Tool Kit video for the NSF Showcase last year! I interviewed Eric in the exact same place that I interviewed Suzy ; )

Hi Rinat! Your contributions to the RoofMod design tool still live on! Hope you and your family are doing well!