Dig deeper

Available in Brazilian Portuguese as Cave mais fundo.

A small problem can be a symptom of a complex situation left unaddressed. You should ask yourself:

• What is the problem?
• What is the context of this problem?
• What led your organization to this outcome?
• How can you solve this problem?
• Are there any similar problems you can solve with a similar solution?
• How can you manage this complex situation?

Trigger

Some weeks ago, I noticed that our Feedback 4 email reminders included a message to Outreachy mentors telling them that failure to submit feedback on their intern’s progress in a timely manner would result in a delay to authorize their intern’s final stipend payment — and that is not correct.

Outreachy provides internships in open source and open science to people subject to systemic bias and impacted by underrepresentation in the technical industry where they are living. We run two cohorts per year:

• Our May cohort, which runs from May to August;
• Our December cohort, which takes place from December to March.

Currently, Outreachy organizers collect feedback from mentors and interns four times throughout the internship. Internally, we refer to those feedback cycles as Feedback 1 to 4. Feedback cycles 1 and 3 are tied to stipend payment authorizations¹, whereas feedback cycles 2 and 4 allow us to continue monitoring the progress and successes of our interns.

The addition of a fourth feedback cycle is a recent change. Before March 2022, we would only collect feedback from mentors and interns three times. We would call those feedback cycles initial, midpoint, and final feedback cycles. We decided to increase the frequency of feedback collection for a couple of reasons:

• To follow the progress of interns more closely, and act early when problems arise;
• To decrease the time interns have to wait to receive their second and last stipend payment;
• To allow Software Freedom Conservancy to finish paying most, if not all interns from the December cohorts before the end of their fiscal year.

We send automated emails to coordinators, mentors, and interns all the time; some of the most important are sent directly from our organizer dashboard. Our dashboard is dynamic: it recommends different automation-assisted actions according to the current stage of both cohorts (as they often overlap). One of those emails is a reminder to mentors to submit feedback about their interns if they haven’t done that already. The template we use for that automated email changes a bit if that reminder is sent after the submission deadline — for example, it always includes an [URGENT] flag on the subject of the email, and it may includes a message stating that their intern’s stipend payment may be delayed if that feedback cycle is tied to a payment authorization. As you’ve just learned, Feedback 4 isn’t one of them.

Digging

I decided to take a look at the website folder containing all of our automated email templates, and I realized we had four email templates intended for Feedback 4:

• final-feedback-instructions.txt → Updated 2 years ago by Sage Sharp, one of my fellow organizers, to reflect our current stipend payment authorization policies.
• final-feedback-reminder.txt → Updated 5 years ago, it still includes a stipend payment delay warning.
• feedback4-feedback-instructions.txt → Created 2 years ago, and included accurate instructions for Feedback 4.
• feedback4-feedback-reminder.txt → Created 2 years ago, and included an accurate submission reminder for Feedback 4.

Well, if an updated Feedback 4 reminder exists in our codebase, why isn’t our website fetching it?

Bullseye

Our website code still refers to Feedback 1, 2, and 4 as initial, midpoint, and final feedback cycles. It was much easier and less time consuming for my fellow organizer to add one more feedback cycle as Feedback 3 between our midpoint and final feedback cycles than to reword every mention to every feedback cycle in every file.

The file emails.py fetches email templates based on stage:

def feedback_email(intern_selection, request, stage, past_due, **kwargs):
    emails = []
    if past_due:
        emails.append(organizers)
        for m in intern_selection.mentors.all():
            emails.append(m.mentor.email_address())
        emails = emails + intern_selection.project.project_round.community.get_coordinator_email_list()
        template = 'home/email/' + stage + '-feedback-reminder.txt'
    else:
        emails.append(intern_selection.applicant.applicant.email_address())
        for m in intern_selection.mentors.all():
            emails.append(m.mentor.email_address())
        emails = emails + intern_selection.project.project_round.community.get_coordinator_email_list()
        template = 'home/email/' + stage + '-feedback-instructions.txt'
    send_group_template_mail(template, {
        'intern_selection': intern_selection,
        'current_round': intern_selection.project.round(),
        },
        request=request,
        recipient_list=emails,
        **kwargs)

Reading the file dashboard.py, it becomes clear Feedback 4 is being referred to as final feedback:

class FinalFeedbackInstructions(FeedbackInstructions):
    """
    Send final feedback instructions to mentors and interns.

    When: When final feedback forms open, and again if feedback is missing.
          Emails may be sent with an URGENT subject if feedback is late.
          Emails may be sent multiple times if there is a internship extension.

    Templates: home/templates/home/email/final-feedback-reminder.txt
               home/templates/home/email/final-feedback-instructions.txt
    """
    description = 'Final Feedback Reminder'
    slug = 'final-feedback-instructions'

    @staticmethod
    def due_date(current_round):
        return current_round.finalfeedback

    def generate_messages(self, current_round, connection):
        if not self.request.user.is_staff:
            raise PermissionDenied("You are not authorized to send reminder emails.")

        # Only get interns that are in good standing and
        # where a mentor or intern hasn't submitted feedback.
        interns = current_round.get_interns_with_open_final_feedback()

        for i in interns:
            email.feedback_email(i, self.request, "final", i.is_final_feedback_on_intern_past_due(), connection=connection)

The short term, one file edit solution to this specific problem is to edit final-feedback-reminder.txt to use use the same language used on feedback4-feedback-reminder.txt. But the long term solution is recognizing that our terminology is convoluted and reaching an agreement about the way we refer to feedback cycles in internal and external communications.

For example, Software Freedom Conservancy still uses initial and final denominations when referring to feedback cycles tied to payment authorizations. The JSON file we use to assist them with their automations over RT for some reason refers to Feedback 3 as midpoint feedback (and we have to manually rename it to final feedback every time we send in new payment authorizations). It’s a technical, process, and information architecture debt that will keep haunting us until we address its root causes. That’s why you have to keep digging deeper.

Deeper

When Sage and I had a 1:1 where we talked about the problem with our Feedback 4 reminder, they suggested that I should delete the unused, redundant files feedback4-feedback-instructions.txt and feedback4-feedback-reminder.txt. While doing that, I decided to take a look at the templates we use for other feedback cycles to see if there were any similar problems I could solve with a similar solution. That led me to the following discoveries:

• initial-feedback-instructions.txt still refers to Feedback 1 as “initial feedback”. The subject line of initial-feedback-reminder.txt also refers to our $3000 stipend payment as “initial payment”.
• midpoint-feedback-reminder.txt still refers to Feedback 2 as “mid-point feedback”.
• feedback3-feedback-reminder.txt doesn’t include a warning about payment delays on the subject line or the actual message. feedback3-feedback-instructions.txt doesn’t even mention that this feedback cycle is tied to a payment authorization.

And just like that, my one file edit solution became a multi-file pull request!

Grace

Outreachy is organized by a very small team:

• Karen Sandler, one of the Outreachy co-founders alongside Marina Zhurakhinskaya;
• Sage Sharp, our cultural change agent;
• Myself, Anna e só, as our information and process architect;
• Omotola Omotayo, our community manager;
• Tilda Udufo, our mentor advocate.

We hit our 1,000 interns milestone last year after 14 years of history. We supported 40 mentoring organizations, 191 mentors, and 121 interns in 2023. This incredible work is only possible thanks to the support of our donors, corporate and individual, and Software Freedom Conservancy. (I’m sure I’m not alone in finding that our work is important and impactful. If you feel the same way, please support us!)

We accrue such debt because we were — and we continue to be — under the pressure of limited resources and time constraints. While we have our areas of interest reflected in our titles, in reality, every Outreachy organizer you meet is wearing four or five hats all at once. Sage’s implementation of a fourth feedback cycle is brilliant because, despite having very little time to deploy it, their solution is still effective. It has side effects, but we have the capability to increase its efficiency.

As a process architect, it’s my duty to take a hard look at our organizational structure and find ways to improve it. It is, however, my responsibility to understand that we may choose to do something not because it’s the best solution we can achieve, but because it’s the best we can do at that particular moment.