Anna e só
Systems thinking > Problem structuring

Dig deeper: The art of uncovering messes by investigating seemingly small problems

Technical, process, and information architecture debts that will keep haunting you until you address its root causes. You have to dig deeper. · 5 min read

A small problem can be a symptom of a complex situation left unaddressed. You should ask yourself:

  • What is the problem?
  • What is the context of this problem?
  • What led your organization to this outcome?
  • How can you solve this problem?
  • Are there any similar problems you can solve with a similar solution?
  • How can you manage this complex situation?

Trigger #

Some weeks ago, I noticed that our Feedback 4 email reminders included a message to Outreachy mentors telling them that failure to submit feedback on their intern's progress in a timely manner would result in a delay to authorize their intern's final stipend payment — and that is not correct.

Outreachy runs two cohorts per year:

Currently, Outreachy organizers collect feedback from mentors and interns four times throughout the internship. Internally, we refer to those feedback cycles as Feedback 1 to 4. Feedback cycles 1 and 3 are tied to stipend payment authorizations[1], whereas feedback cycles 2 and 4 allow us to continue monitoring the progress and successes of our interns.

The addition of a fourth feedback cycle is a recent change. Before March 2022, we would only collect feedback from mentors and interns three times. We would call those feedback cycles initial, midpoint, and final feedback cycles. We decided to increase the frequency of feedback collection for a couple of reasons:

We send automated emails to coordinators, mentors, and interns all the time; some of the most important are sent directly from our organizer dashboard. Our dashboard is dynamic: it recommends different automation-assisted actions according to the current stage of both cohorts (as they often overlap). One of those emails is a reminder to mentors to submit feedback about their interns if they haven't done that already. The template we use for that automated email changes a bit if that reminder is sent after the submission deadline — for example, it always includes an [URGENT] flag on the subject of the email, and it may includes a message stating that their intern's stipend payment may be delayed if that feedback cycle is tied to a payment authorization. As you've just learned, Feedback 4 isn't one of them.

Digging #

I decided to take a look at the website folder containing all of our automated email templates, and I realized we had four email templates intended for Feedback 4:

Well, if an updated Feedback 4 reminder exists in our codebase, why isn't our website fetching it?

Bullseye #

Our website code still refers to Feedback 1, 2, and 4 as initial, midpoint, and final feedback cycles. It was much easier and less time consuming for my fellow organizer to add one more feedback cycle as Feedback 3 between our midpoint and final feedback cycles than to reword every mention to every feedback cycle in every file.

The file emails.py fetches email templates based on stage:

def feedback_email(intern_selection, request, stage, past_due, **kwargs):
    emails = []
    if past_due:
        emails.append(organizers)
        for m in intern_selection.mentors.all():
            emails.append(m.mentor.email_address())
        emails = emails + intern_selection.project.project_round.community.get_coordinator_email_list()
        template = 'home/email/' + stage + '-feedback-reminder.txt'
    else:
        emails.append(intern_selection.applicant.applicant.email_address())
        for m in intern_selection.mentors.all():
            emails.append(m.mentor.email_address())
        emails = emails + intern_selection.project.project_round.community.get_coordinator_email_list()
        template = 'home/email/' + stage + '-feedback-instructions.txt'
    send_group_template_mail(template, {
        'intern_selection': intern_selection,
        'current_round': intern_selection.project.round(),
        },
        request=request,
        recipient_list=emails,
        **kwargs)

Reading the file dashboard.py, it becomes clear Feedback 4 is being referred to as final feedback:

class FinalFeedbackInstructions(FeedbackInstructions):
    """
    Send final feedback instructions to mentors and interns.

    When: When final feedback forms open, and again if feedback is missing.
          Emails may be sent with an URGENT subject if feedback is late.
          Emails may be sent multiple times if there is a internship extension.

    Templates: home/templates/home/email/final-feedback-reminder.txt
               home/templates/home/email/final-feedback-instructions.txt
    """
    description = 'Final Feedback Reminder'
    slug = 'final-feedback-instructions'

    @staticmethod
    def due_date(current_round):
        return current_round.finalfeedback

    def generate_messages(self, current_round, connection):
        if not self.request.user.is_staff:
            raise PermissionDenied("You are not authorized to send reminder emails.")

        # Only get interns that are in good standing and
        # where a mentor or intern hasn't submitted feedback.
        interns = current_round.get_interns_with_open_final_feedback()

        for i in interns:
            email.feedback_email(i, self.request, "final", i.is_final_feedback_on_intern_past_due(), connection=connection)

The short term, one file edit solution to this specific problem is to edit final-feedback-reminder.txt to use use the same language used on feedback4-feedback-reminder.txt. But the long term solution is recognizing that our terminology is convoluted and reaching an agreement about the way we refer to feedback cycles in internal and external communications.

For example, Software Freedom Conservancy still uses initial and final denominations when referring to feedback cycles tied to payment authorizations. The JSON file we use to assist them with their automations over RT for some reason refers to Feedback 3 as midpoint feedback (and we have to manually rename it to final feedback every time we send in new payment authorizations). It's a technical, process, and information architecture debt that will keep haunting us until we address its root causes. That's why you have to keep digging deeper. (Remember: This a problem within a mess. Think systemically.)

Deeper #

When Sage and I had a 1:1 where we talked about the problem with our Feedback 4 reminder, they suggested that I should delete the unused, redundant files feedback4-feedback-instructions.txt and feedback4-feedback-reminder.txt. While doing that, I decided to take a look at the templates we use for other feedback cycles to see if there were any similar problems I could solve with a similar solution. That led me to the following discoveries:

And just like that, my one file edit solution became a multi-file pull request!

Grace #

Outreachy is organized by a very small team:

We hit our 1,000 interns milestone last year after 14 years of history. We supported 40 mentoring organizations, 191 mentors, and 121 interns in 2023. This incredible work is only possible thanks to the support of our donors, corporate and individual, and Software Freedom Conservancy. (I'm sure I'm not alone in finding that our work is important and impactful. If you feel the same way, please support us!)

We accrue such debt because we were — and we continue to be — under the pressure of limited resources and time constraints. While we have our areas of interest reflected in our titles, in reality, every Outreachy organizer you meet is wearing four or five hats all at once. Sage's implementation of a fourth feedback cycle is brilliant because, despite having very little time to deploy it, their solution is still effective. It has side effects, but we have the capability to increase its efficiency.

As a process architect, it's my duty to take a hard look at our organizational structure and find ways to improve it. It is, however, my responsibility to understand that we may choose to do something not because it's the best solution we can achieve, but because it's the best we can do at that particular moment.

  1. As of April 2024, Outreachy provides a U$ 7,000 total internship stipend. Feedback 1 is tied to a U$ 3,000 stipend payment authorization, whereas Feedback 3 is tied to a U$ 4,000 stipend payment authorization. ↩︎