Scrobbling over ActivityPub


#1

I am interested in adding support for scrobbling to Pleroma. Scrobbling is essentially live-updating what you’re listening to in real time (it is a reference to the original name of last.fm, which was audioscrobbler).

However, all of the current scrobbling APIs suck. And, of course, there is the matter of federating the scrobbles themselves.

And so we have the question of how to express a scrobble in terms of AS2 vocabulary. For inspiration, I noticed that Funkwhale has funkwhale:Track objects that describe tracks. But I am not quite sure what is stored in a track object.

And so, I came up with this basic idea:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
       "Track": "funkwhale:Track",
       "funkwhale": "https://funkwhale.audio"
    }
  ],
  "type": "Listen",
  "to": ["https://www.w3.org/ns/activitystreams#Public"],
  "cc": ["https://pleroma.site/users/kaniini/followers"],
  "object": {
    "type": "Track",
    "album": "Metallica",
    "artist": "Metallica",
    "title": "Nothing Else Matters",
    "mimeType": "audio/flac",
    "length": 165000
  }
}

The url property could be set to the track or music video being listened to or watched if the user is using something like the Web Scrobbler extension.

At least in terms of Pleroma, instead of emulating very flawed APIs (home-baked substitute for OAuth lives on in 2019? really?), we would just support this as a native AP C2S message.

Thoughts? Does funkwhale:Track map in the way I think it does?


#2

You could have a look at the last.fm documentation for possible further parameters: https://www.last.fm/api/show/track.scrobble


#3

Ahah we posted the same topic with the same exact title (mine was refused as a result :wink:

Of course we’re interested for Funkwhale, because we plan to federate user activity and starting with scrobbling and having something compatible with other software would be good.

We have a similar idea about the implementation using the Listen activity, which I think is relevant here. However, I think the object value would benefit from being an Audio object instead of a Funkwhale:Track, because that would bring compatibility with a wider range of implementations IMHO.

Funkwhale:Track basically map to MusicBrainz Recordings. It’s an abstract representation of a released work, like when we say “I love this song by XXX”.

On the other hand, Audio (at least in Funkwhale) represent an actual audio file (mp3, ogg, etc.). In Funkwhale, Audio objects are linked to a Track, because multiple users can upload different audio files (with different bitrates, formats, etc.) of the same track. Audio are also linked to Funkwhale user libraries, but it doesn’t matter here.

This is a typical Audio representation in Funkwhale:

{
  "type": "Audio",
  "id": "https://awesome.music/federation/music/uploads/88f0bc20-d7fd-461d-a641-dd9ac485e096",
  "name": "For Whom the Bell Tolls - Ride the Lightning - Metallica",
  "size": 8656581,
  "bitrate": 320000,
  "duration": 213,
  "updated": "2018-10-02T19:49:35.646372+00:00",
  "published": "2018-10-02T19:49:35.646359+00:00",
  "track": "https://awesome.music/federation/music/tracks/82ece296-6397-4e26-be90-bac5f9990240",
  "url": {
    "href": "https://awesome.music/api/v1/listen/82ece296-6397-4e26-be90-bac5f9990240/?upload=88f0bc20-d7fd-461d-a641-dd9ac485e096",
    "type": "Link",
    "mediaType": "audio/mpeg"
  }
}

While a Track looks like that:

{
  "type": "Track",
  "id": "https://awesome.music/federation/music/tracks/82ece296-6397-4e26-be90-bac5f9990240",
  "name": "For Whom the Bell Tolls",
  "position": 3,
  "published": "2018-10-02T19:49:35.822537+00:00",
  "musicbrainzId": "771ab043-8821-44f9-b8e0-2733c3126c6d",
  "artists": [
    // List of Artist objects
  ],
  "album": // Album object
}

For me it makes more sense to use Audio because when you don’t listen to For Whom the Bell Tolls, by Metallica (the Track), you actually listen to the live interpretation of this track, in MP3 format, in this quality (the Audio). And if we are using audio, this also means that for public tracks (like the ones https://open.audio) we will be able to forward the link to the audio itself too!

Also, using Audio instead of Track is probably more generic if we want to support different audio types (like postcast episodes in the future).

I don’t have any strong opinion on the client part (I still have to look at the scroble APIs), and never played with C2S either. This is also easier to make evolve than the federation part.


#4

Audio can work, but would it work well in the case where we cannot actually provide the audio clip being listened to itself? Only the metadata?


#5

For creating statistics I would prefer having something abstract like the track. There a remote system could track this data and could match it with data from other users. Possibly the “track” type could optionally contain some “audio” element.


#6

Possibly the “track” type could optionally contain some “audio” element.

Ok, it could work for us :slight_smile:


#7

Right, that is what I was thinking with my initial proposal. It is better to keep the data separate rather than using an Audio object. For Pleroma’s uses in the short term, having a preformatted string would work well (since we’re just showing it in the UI), but if somebody wanted to do a plugin that offered the same statistics capabilities that last.fm has, welp, they would be screwed with the data not properly split out.


#8

I’m a “big data” fan. I can already think of statistics that could suggest users to other users based upon their musical preferences.


#9

If we go the Track way, do you plan to use Funkwhale’s type for that? (I’d rather not extract data from preformated strings unless I have no choice). I’m asking because internally, Listen activities in Funkwhale are mapped to a Track (we already store them, we just don’t federate them yet).

Based on your exemple, if you used the actual Track type from Funkwhale, your listen activity would look like that:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
       "Track": "funkwhale:Track",
       "funkwhale": "https://funkwhale.audio"
    }
  ],
  "type": "Listen",
  "to": ["https://www.w3.org/ns/activitystreams#Public"],
  "cc": ["https://pleroma.site/users/kaniini/followers"],
  "object": {
    "type": "Track",
    "name": "Nothing Else Matters",
    "position": 3,
    "album": {
      "name": "Metallica",
      "artists": [
        {"name": "Metallica"}
      ]
    },
    "artists": [
      {"name": "Metallica"}
    ],
  }
}

I’d say the mimeType should rather live in an embedded audio, if any. Also we use duration in our Audio type, so maybe we could reuse it here instead of length?

I know this format may be a bit surprising at first (multiple artists, artists on album AND track), but audio (and creation) in general don’t fit in the “1 track = 1 album = 1 artist” paradigm. It’s common to have a different album and track artist, and even to have multiple artists for an album or track. Since I’ve encountered an album with 10 different tracks, all having the exact same name, nothing surprises me anymore, and I tend to leave the most flexibility.

That’s also why I think relying on external databases like MusicBrainz (and include musicbrainzId for Track/Album/Artist) should be considered when possible. It makes everything easier.


#10

I don’t know if it makes sense to use funkwhale:Track or not. Maybe it makes more sense to do this as a litepub specification and put it in the litepub namespace. It really depends on what your take on using funkwhale:Track is.

Agreed about supporting musicbrainz IDs if we have them, that would allow software to do more in depth lookups easily.

Something like this would be OK, I guess, but it would be nice to support a more collapsed format for developer convenience:

    "https://www.w3.org/ns/activitystreams",
    {
       "Track": "funkwhale:Track",
       "funkwhale": "https://funkwhale.audio"
    }
  ],
  "type": "Listen",
  "to": ["https://www.w3.org/ns/activitystreams#Public"],
  "cc": ["https://pleroma.site/users/kaniini/followers"],
  "object": {
    "type": "Track",
    "name": "Nothing Else Matters",
    "position": 3,
    "album": {
      "name": "Metallica",
      "artists": [
        {"name": "Metallica"}
      ]
    },
    "artists": [
      {"name": "Metallica"}
    ],
    "duration": 165000,
    "musicbrainzId": "...",
    "audio": {
      "mediaType": "audio/flac"
    }
  }
}

Note that technically mediaType would be under url as in your original example, but if we’re not sharing a URL to the audio file (for example if the audio file is stored locally, or the user does not choose to share their collection), then it seems inappropriate to store it under url, since there’s nothing to follow.

What do you think?