Femto-hopper: a tool for improving data portability between mood tracking apps

Posted on April 13, 2020 by Geo in Tech Software Projects Coding Data.

2530 words | 16 minutes


Header image
Photo by Nariman Mesharrafa on Unsplash

Before I begin, if you’d rather let the cold, hard code speak – a perfectly reasonable choice – here’s a quick link to the GitHub Repo:

Code

Otherwise, here’s the whole backstory – but be warned: there’s no TL;DR at the end.
Here it goes.

The backstory

Around the time I began doing meditation, I also started getting into mood trackers, simple apps that really just do… what the name says: keeping track of your general mood over time by asking you how you feel and recording your answers. They’re a very clean and effortless way to keep a bare-bones journal, and some apps even offer analysis functions to highlight correlations between various factors – it isn’t causation, but it doesn’t have to be. It’s just a really neat insight into one’s own mind.

I started out with Moodpath, a great app that complemented its thorough mood tracking features with helpful general mental health resources like articles, podcasts, even meditations and helplines.
After a while, however, its awkward requirement of an internet connection to submit entries became unbearable to me: I was used to making my morning entries while on the subway, and the signal was always too weak if not none – if I tried posting it without appropriate reception, it just failed the submission and the entry was lost forever, so I couldn’t even wait to try again.
I then switched to Daylio, which focused more on the mood tracking and correlation features – which I was almost exclusively using anyway – and allowed for offline entries. I’m still using it to this day, but every time I made a switch – I’d also changed phone meanwhile – I was forced to give up all my progress and restart from scratch: I had a couple of different data backups from Moodpath, which came as a bunch of .csv files that… really just existed for their own sake, as restoring from backups wasn’t even supported by Moodpath itself, let alone any relationship with other platforms like Daylio.

I hadn’t been using any of those apps for a very long time though, so at first I just shrugged it off and accepted having to start over.

However, following the global lockdowns imposed by the COVID-19 outbreak, the developers community promptly responded with a great deal of initiatives like Hack Quarantine: a global, online hackathon aimed at easing the critical situation the world is currently facing.
While everyone is getting used to working remotely, information systems are more than ever vital to keep everything running smoothly, and developing helpful software is a great way for anyone to have a massively positive impact on their local or even global community.

And while I wasn’t ready to take on such a huge challenge, I really did want to start some project given that I had literally nothing else to do, and this really was the perfect chance: so, I revisited my old idea of making some piece of software to enable people to more freely use mood trackers.
By “translating” the data from one app to the other, it would prevent users from losing all their progress and starting over each time they switched to a new app. It would build upon the apps’ common backup-restore functionalities, taking the data coming from the first app’s export, with its own formatting, and reformatting it accordingly to what the second app expects for restoring from a backup file.

Besides the usual "I wanted something and it didn’t exist, so I made it myself" reasoning, here’s more on why I actually felt it was at least somewhat relevant to the hackathon’s topic and tracks, straight from the project’s Devpost submission:

The quarantine and isolation forcibly imposed by the COVID-19 pandemic are a time of particular emotional hardship, as we can all witness first-hand: some are locked inside, away from their friends and family; others are stuck in a foreign country, miles away from their loved ones – but if there’s one thing we can all agree on, is that this is no easy moment for anyone, on a global scale.
In this context, mood trackers are a valuable resource for a much wider and diverse set of people, as the whole world experiences lockdown: for many, myself included, these surreal days tend to merge together in a big grey blob, and keeping track of the few events and our emotional responses with a simple journal or app is a great way to keep a grip on reality; for people suffering from all sorts of mental health conditions, this becomes a fundamental tool for coping with the challenges of every day, magnified by the current situation.

So, I asked a couple of friends if they felt like lending a hand, and set off to make it.

Data trouble

Unfortunately though, it wasn’t that simple: before being able to get our hands on any actual code, we needed to know what kind of data we were dealing with. Remembering the quote by nobody less than Linus Torvalds himself:

Bad programmers worry about the code. Good programmers worry about data structures and their relationships.
(more)

And this project really is all about the data: it doesn’t do anything apart from taking a data structure and relating it to other, different data structures.
So, in my effort of at least trying to be a good programmer, I started scouting for all kinds of mood tracking apps, noting down whether they had the features we were looking for and, if so, documenting what data formats they relied on. The (updated) result of this (ongoing) work is publicly available on this GitHub Project Board.

What I found generally was… a bit of a mess: the sheer number of mood tracker apps available around is staggering, and they all manage their sets of features in different ways: of those which support a backup at all – unfortunately, way too many apps don’t even provide a data export – alas! most use Google Drive’s hidden AppData directory, others rely on Dropbox or proprietary clouds. And the handful that do support a backup that really is owned, handleable and modifiable by the user, aren’t uniform either: some use JSON, some CSV, and a few even SQLite databases.
In short, everything just did its own thing.

But that was exactly the mess we were trying to solve with this project, so we got to work.

Few, for many

The first step was figuring out how exactly to run the conversions.
Given the huge differences between formats, writing a script for each possible combination of input/output apps would have made scaling up this project a nightmare, as the number of individual scripts needed would have increased quadratically:

x = n * (n - 1) = O(n²)

Instead, we opted for a system built around a central format, in which all conversions between apps are broken down into two sub-conversions: one *to* the central format and one *from* the central format. By making all conversion go through a single, standardised structure, we were able to reduce the growth function to a beautiful linear shape:

x = 2 * n = O(n)

Now, this approach undoubtedly meant losing accuracy and thoroughness in the conversions; however, some degree of loss was expected and unavoidable, simply due to the different design choices that give purpose to each app, and pitch-perfect conversion was nothing short of utopian.

Femto-hopper's coral logo
Femto-hopper logo icon by Made x Made on The Noun Project, © 2020 [CC BY 3.0]

The differences between the various apps is something we appreciate, value and promote: they’re all unique in style, features and goal, and each one of them appeals to different people, in different times, with different needs. Everyone’s different and unique, and the growing diversity of apps available on the market beautifully reflects that!

We really do consider it the primary catalyst for this project: using different formats is okay – given the innate differences between them, it’s obvious that the optimal ways of storing the data they record will be multiple and diverse. And although, as we found, they all have a handful of essential features in common, transferring this extremely basic data between the various apps was still pretty much impossible.
But, despite the differences, not all data should be lost: we users shouldn’t be bound to only using one single app just because that’s the one we’ve found out about, only being able to retain their data when changing device – but clinging to the same app – and through an opaque, remote backup they can neither access nor even view; we shouldn’t have to give up all our previous data and start from scratch whenever we try out a new app. Instead, we should be free to switch to a simpler app if longing for more clarity and focus during hardship in life, or to a more complex one if seeking a more thorough insight to our own mood, for instance – all while still being able to retain at least some amount of progress throughout our adventures!

Obviously, optimising this amount of data to make it as accurate and representative as possible is one of the main goals we strive for – but given the inherent uniqueness of each app, we chose to embrace the loss it opposes in the name of elegance, uniformity and scalability.
That’s why, as the foundation of our central standard, we picked the very minimum common denominator of all apps:

  • some kind of “mood out of x” rating
  • an optional note to elaborate on your thoughts and feelings

Sure, the former element is a very, very simple form of data, but even such a basic bit was enough for the first draft – and enough trouble, especially about the different date/time formats. Moreover, the latter proved very helpful, in our design, by acting as a “catch-all” field for any sort of information: even if a piece of data is unavoidably impossible to preserve in a machine-readable way throughout the conversion, that doesn’t stop it from being retained as a simple text field in a humanely meaningful way!
At any rate, we just weren’t willing to come up with 10 new scripts from scratch just to include the 6th app: among the usual business, it would have made continuing to work on this project even harder – instead, we would just refer to the standard set of rules, and add 2 scripts every time. So, pack the data up with a time field in a JSON list of one dictionary per entry, and here’s the very first draft, FEMTO v0.1:

femto = [
  {
    "time": <int: UNIX-epoch timestamp in seconds>,
    "mood": <int: 1=worst ... 5=best>,
    "note": <str>
  },
...]

Intricate coral branches

Photo by NOAA on Unsplash

Code & contact

Reassured by once again having worried about the data, this time we finally started writing some code: the first “subjects” were Daylio, since I know it inside out being my daily driver, and Pixels, since it’s clear and minimal in its backup files; the scripts were simple and basic, just as we wanted, and they even worked!

However, by the time we did get them to work, the hackathon was about to end, and our project was still just a stub: a handful of scripts, even less apps and no interface – but it was never meant to be completed for the hackathon: compared to the other programs competing, it wasn’t even that relevant to the COVID-19 theme.
Moreover, we were met with a paradoxical shortage of apps to include: quite surprisingly, only a few among all the available apps provided the combination of features we were looking for. So we started a second, extended recon sweep: along with the scouting for new apps, I also put together and distributed a survey for gaining a better insight into the actual users’ habits and preferences, and started contacting the app developers, to let them know about our project and get them involved if they’re interested.

This second sweep also revealed an uplifting development: some of the most popular journaling apps (Day One, Diaro, Journal it! – unfortunately not Journey) already offered a suite of functions for importing from each other’s formats! Plus, hey too offer the basic mood tracking features we require for inclusion, which makes them really appealing to us.

How can I help?

If you’re a mood tracker user and you want to help us, the quickest way to help us is by filling out this brief Typeform Survey: this way, you’ll let us know what mood trackers to focus on and provide us with a better insight to the features we’re working on.

If instead you’re a developer, all the updated details for contributing are available on the repo’s README.

At any rate, feel free to contact me about any doubts, questions, offers or requests; you can’t go wrong with that, and we’re always eager to expand our network.

Femto’s future

While the hackathon is now over, we do plan on keeping up the work on Femto-hopper to develop and expand its scope.

Along with more scripts for more apps, an interface is definitely waiting shortly along the roadmap – a text-based, command-line one at least.
And if the project takes off enough, our plan also involves porting it to a mobile app (we’re thinking Kotlin perhaps, for the native cross-platform development which is a staple of our vision), for the most natural, intuitive and user-friendly interaction: export from the old app, fire up Femto-hopper and convert the file, import to the new app.

Let’s see what awaits!