⏳
4 mins
read time
Music has a significant impact on the world in various ways. Gaining insight into the patterns of popular music can be a fascinating endeavor. In this post, we will demonstrate how to utilize Spotify’s trending music to stay up-to-date with current trends in a self-hosted manner.
Spotify is a widely used music service, and its playlist data is publicly available on the internet. There are several popular trending playlists that reflect current music preferences. By utilizing GitHub Actions, you can automatically fetch this data at regular intervals and store it without needing to manage a database. This data can then be easily accessed using straightforward HTTP requests directly to GitHub. I utilize the project called spotify-downloader to download the playlist data and save it as a file. Here is the follwing code snippet to do it:
def download(key, url):
cmd=f"docker run --rm -v {CWD}/tmpplaylists:/music spotdl/spotify-downloader save {url.strip()} --save-file {key}.spotdl"
p=subprocess.Popen(cmd.split(" "),
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
print(f">>> {line.rstrip().decode('utf-8')}")
Then I parse every playlists file and do simple preprocessing on the artists column in order to obtain a list of every artists that participates in the songs.
def read_data():
appended_data = []
cols = ['name', 'artists', 'album_name', 'date', 'song_id', 'cover_url', 'playlist', 'position']
for f in glob.glob('tmpplaylists/*.spotdl'):
data = pd.read_json(f).assign(
artists=lambda x: x['artists'].explode().str.replace("'","").str.replace("\"", "").reset_index().groupby('index').agg({'artists': lambda y: y.tolist()}),
playlist=f.split("/")[1].split(".")[0],
position=lambda x: x.index + 1
)
assert len(set(cols).difference(data.columns)) == 0, f'Columns: {", ".join(data.columns)}'
assert len(data) > 0, f"Shape {data.shape[0]} and {data.shape[1]} columns"
appended_data.append(data)
(
pd.concat(appended_data, ignore_index=True)
.get(cols)
.to_csv('static/data/data.csv', index=False, header=True, sep=";")
)
By employing periodic GitHub Actions, it is possible to regularly save playlist positions every week, enabling further processing of this data through other tools.
I utilize the Observable framework, which incorporates the D3 JavaScript library for generating swift and adaptable visualizations.
Observable Notebook combines the features of conventional text editors, code editors, and document processors into a unified interface, simplifying the creation of rich and dynamic documents that integrate text, code, data visualization, and other multimedia elements.
Observable employs the concept of “cells” to arrange content within a notebook, where each cell can either contain plain text or executable code written in JavaScript or any other supported language. Cells can be rearranged, grouped, and nested, enabling the creation of hierarchical structures that reflect the logical organization of the document.
One can write a markdown notebook and import data from multiple languages, for example I use a python preprocessing pipeline, then I import the data in the notebook and plot it using the available visualizations functions.
# Playlist details
const commit_date_old = Array.from(new Set(diffData.map(i => i.commit_date)))[1];
const commit_date_recent = Array.from(new Set(diffData.map(i => i.commit_date)))[0];
From ${commit_date_old} to ${commit_date_recent} new songs have been added to the playlist.
const playlistsNames = bestArtists.map(i => i.playlist)
const playlistChoosen = view(Inputs.select(new Set(playlistsNames), {value: playlistsNames[0], label: "Playlists"}));
const artistsNames = bestArtists.map(i => i.artists)
const tableRows = RecentSongAdds(diffData, playlistChoosen, commit_date_old, commit_date_recent)
<div class="card" style="margin: 1rem 0 2rem 0; padding: 0;">
${Inputs.table(tableRows, {
columns: ["position", "artists", "name", "album_name", "attribute"],
align: {"position": "left"},
format: {
attribute: (x) => x == "+" ? "New!" : x == "-" ? "🗑" : x > 0 ? `⬆${x}` : x == 0 ? '--' : `⬇${Math.abs(x)}`
}
})}
</div>
<div class="grid grid-cols-1" style="grid-auto-rows: 560px;">
<div class="card">
${BestArtistsPlot(bestArtists, playlistChoosen)}
</div>
</div>
const mostPopularArtists = view(Inputs.select(mostFrequent(bestArtists.filter(i => i.playlist == playlistChoosen).map(i => i.artists)).slice(0,10), {value: artistsNames[0], label: "Popular artists"}));
<div class="grid grid-cols-1" style="grid-auto-rows: 560px;">
<div class="card">
${BestSongsPlot(bestArtists, playlistChoosen, mostPopularArtists)}
</div>
</div>
The dashboard is hosted on GitHub pages, the link is available at cristianpb.github.io/playlists.
The dashboard allows for the identification of patterns in the development of Spotify playlists over time. The Today Top Hits playlist reflects global music trends, having garnered more than 34 million likes at the time of writing this article.
We can observe artists such as Olivia Rodrigo, who has multiple tracks featured in the “Today’s Top Hits” playlist. Some songs exhibit a consistent pattern, indicating that they have maintained popularity and catchiness over time, for example, “The Vampire Song,” which remained among the top 35 songs for more than four months. Conversely, other tracks like “Catch Me Now” may initially appear in the playlist due to the artist’s popularity but subsequently decline in ranking during subsequent weeks.
One might also observe that artist-specific radio playlists, which are frequently updated, exhibit minimal fluctuations. For instance, “Muse Radio,” “Coldplay Radio,” and “The Strokes” playlists undergo infrequent changes.
Observable is a practical platform for crafting data analyses, offering versatile connectors and support for multiple programming languages. The variety of available visualizations is crucial, and comprehensive documentation plays a significant role in guiding users to create effective visualizations.
However, incorporating reactive filters or reusing variables within an Observable notebook necessitates writing JavaScript code, which may be a drawback for some users. Although the reactivity of Observable notebooks is functional, it might not be the most advanced option available.
The code to process the data and build the dashboard is available at github.com/cristianpb/playlists.
This post shows how to query data from Paris Open Data base, make some geographical plots using folium and also some analysis about the bike utilisation depending on the date
This articles presents how to generate content using large language and vision models. The content is then shared websites and social networks.
This article presents a study on predicting user-generated revenue within a digital application. Using gradient boosting models and a dataset containing user installation features and engagement metrics, the study aims to forecast revenue at day 120. The model's performance and key influencing factors are analyzed, providing insights for optimizing monetization strategies in the digital applications industry.
Using RAG and LLM to provide accurate information about plant care.