Spotify is a widely used music service, and its playlist data is publicly available on the internet. There are several popular trending playlists that reflect current music preferences. By utilizing GitHub Actions, you can automatically fetch this data at regular intervals and store it without needing to manage a database. This data can then be easily accessed using straightforward HTTP requests directly to GitHub. I utilize the project called spotify-downloader to download the playlist data and save it as a file. Here is the follwing code snippet to do it:
def download(key, url):
cmd=f"docker run --rm -v {CWD}/tmpplaylists:/music spotdl/spotify-downloader save {url.strip()} --save-file {key}.spotdl"
p=subprocess.Popen(cmd.split(" "),
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
print(f">>> {line.rstrip().decode('utf-8')}")
Then I parse every playlists file and do simple preprocessing on the artists column in order to obtain a list of every artists that participates in the songs.
def read_data():
appended_data = []
cols = ['name', 'artists', 'album_name', 'date', 'song_id', 'cover_url', 'playlist', 'position']
for f in glob.glob('tmpplaylists/*.spotdl'):
data = pd.read_json(f).assign(
artists=lambda x: x['artists'].explode().str.replace("'","").str.replace("\"", "").reset_index().groupby('index').agg({'artists': lambda y: y.tolist()}),
playlist=f.split("/")[1].split(".")[0],
position=lambda x: x.index + 1
)
assert len(set(cols).difference(data.columns)) == 0, f'Columns: {", ".join(data.columns)}'
assert len(data) > 0, f"Shape {data.shape[0]} and {data.shape[1]} columns"
appended_data.append(data)
(
pd.concat(appended_data, ignore_index=True)
.get(cols)
.to_csv('static/data/data.csv', index=False, header=True, sep=";")
)
By employing periodic GitHub Actions, it is possible to regularly save playlist positions every week, enabling further processing of this data through other tools.
I utilize the Observable framework, which incorporates the D3 JavaScript library for generating swift and adaptable visualizations.
Observable Notebook combines the features of conventional text editors, code editors, and document processors into a unified interface, simplifying the creation of rich and dynamic documents that integrate text, code, data visualization, and other multimedia elements.
Observable employs the concept of “cells” to arrange content within a notebook, where each cell can either contain plain text or executable code written in JavaScript or any other supported language. Cells can be rearranged, grouped, and nested, enabling the creation of hierarchical structures that reflect the logical organization of the document.
One can write a markdown notebook and import data from multiple languages, for example I use a python preprocessing pipeline, then I import the data in the notebook and plot it using the available visualizations functions.
# Playlist details
const commit_date_old = Array.from(new Set(diffData.map(i => i.commit_date)))[1];
const commit_date_recent = Array.from(new Set(diffData.map(i => i.commit_date)))[0];
From ${commit_date_old} to ${commit_date_recent} new songs have been added to the playlist.
const playlistsNames = bestArtists.map(i => i.playlist)
const playlistChoosen = view(Inputs.select(new Set(playlistsNames), {value: playlistsNames[0], label: "Playlists"}));
const artistsNames = bestArtists.map(i => i.artists)
const tableRows = RecentSongAdds(diffData, playlistChoosen, commit_date_old, commit_date_recent)
<div class="card" style="margin: 1rem 0 2rem 0; padding: 0;">
${Inputs.table(tableRows, {
columns: ["position", "artists", "name", "album_name", "attribute"],
align: {"position": "left"},
format: {
attribute: (x) => x == "+" ? "New!" : x == "-" ? "🗑" : x > 0 ? `⬆${x}` : x == 0 ? '--' : `⬇${Math.abs(x)}`
}
})}
</div>
<div class="grid grid-cols-1" style="grid-auto-rows: 560px;">
<div class="card">
${BestArtistsPlot(bestArtists, playlistChoosen)}
</div>
</div>
const mostPopularArtists = view(Inputs.select(mostFrequent(bestArtists.filter(i => i.playlist == playlistChoosen).map(i => i.artists)).slice(0,10), {value: artistsNames[0], label: "Popular artists"}));
<div class="grid grid-cols-1" style="grid-auto-rows: 560px;">
<div class="card">
${BestSongsPlot(bestArtists, playlistChoosen, mostPopularArtists)}
</div>
</div>
The dashboard is hosted on GitHub pages, the link is available at cristianpb.github.io/playlists.
The dashboard allows for the identification of patterns in the development of Spotify playlists over time. The Today Top Hits playlist reflects global music trends, having garnered more than 34 million likes at the time of writing this article.
We can observe artists such as Olivia Rodrigo, who has multiple tracks featured in the “Today’s Top Hits” playlist. Some songs exhibit a consistent pattern, indicating that they have maintained popularity and catchiness over time, for example, “The Vampire Song,” which remained among the top 35 songs for more than four months. Conversely, other tracks like “Catch Me Now” may initially appear in the playlist due to the artist’s popularity but subsequently decline in ranking during subsequent weeks.
One might also observe that artist-specific radio playlists, which are frequently updated, exhibit minimal fluctuations. For instance, “Muse Radio,” “Coldplay Radio,” and “The Strokes” playlists undergo infrequent changes.
Observable is a practical platform for crafting data analyses, offering versatile connectors and support for multiple programming languages. The variety of available visualizations is crucial, and comprehensive documentation plays a significant role in guiding users to create effective visualizations.
However, incorporating reactive filters or reusing variables within an Observable notebook necessitates writing JavaScript code, which may be a drawback for some users. Although the reactivity of Observable notebooks is functional, it might not be the most advanced option available.
The code to process the data and build the dashboard is available at github.com/cristianpb/playlists.
]]>One way to analyze logs is to use a tool like Fluent Bit to collect them from different sources and send them to a central repository like Elasticsearch. Elasticsearch is a distributed search and analytics engine that can store and search large amounts of data quickly and efficiently.
Once the logs are stored in Elasticsearch, you can use Kibana to visualize and analyze them. Kibana provides a variety of tools for exploring and understanding log data, including charts, tables, and dashboards.
By analyzing logs using Fluent Bit, Elasticsearch, and Kibana, you can gain valuable insights into the health and performance of your applications and systems. This information can help you to identify and troubleshoot problems, improve performance, and ensure the availability of your applications.
Traefik, a modern reverse proxy and load balancer, generates access logs for every HTTP request. These logs can be stored as plain text files and compressed using the logrotation Unix utility. Fluent Bit, a lightweight log collector, provides a simple way to insert logs into Elasticsearch. In fact, it provides several input connectors for other sources, such as syslog logs, and output connectors, such as Datadog or New Relic.
To send Traefik access logs to Elasticsearch using Fluent Bit, you will need to:
The configuration file from fluent bit has the following sections:
tail
connector to fetch data from access.log fileThe following is configuration file shows how to collect Traefik logs and send them to Elasticsearch:
# fluentbit.conf
[SERVICE]
flush 5
daemon off
http_server off
log_level info
parsers_file parsers.conf
[INPUT]
name tail
path /var/log/traefik/access.log,/var/log/traefik/access.log.1
Parser traefik
Skip_Long_Lines On
[FILTER]
Name geoip2
Match *
Database /fluent-bit/etc/GeoLite2-City.mmdb
Lookup_key host
Record country host %{country.names.en}
Record isocode host %{country.iso_code}
Record latitude host %{location.latitude}
Record longitude host %{location.longitude}
[FILTER]
Name lua
Match *
Script /fluent-bit/etc/geopoint.lua
call geohash_gen
[OUTPUT]
Name es
Match *
Host esurl.com
Port 443
HTTP_User username
HTTP_Passwd password
tls On
tls.verify On
Logstash_Format On
Replace_Dots On
Retry_Limit False
Suppress_Type_Name On
Logstash_DateFormat all
Generate_ID On
I use an additional filter function to produce a geohash record, which then it’ll be used in kibana in geo maps plot.
# geopoint.lua
function geohash_gen(tag, timestamp, record)
new_record = record
lat = record["latitude"]
lon = record["longitude"]
hash = lat .. "," .. lon
new_record["geohash"] = hash
return 1, timestamp, new_record
end
The parser uses a regex expression to obtain the different fields for each record. By default all fields are process as strings, but you can other types, like integer for fields like request size, request duration and number of requests.
# parsers.conf
[PARSER]
Name traefik
Format regex
Regex ^(?<host>[\S]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?<protocol>\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")? (?<number_requests>[^ ]*) "(?<router_name>[^\"]*)" "(?<router_url>[^\"]*)" (?<request_duration>[\d]*)ms$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Types request_duration:integer size:integer number_requests:integer
Once you have configured Fluent Bit, you can start it by running the following command: fluent-bit -c fluent-bit.conf
or by using docker compose:
# docker-compose.yml
version: "3.7"
services:
fluent-bit:
container_name: fluent-bit
restart: unless-stopped
image: fluent/fluent-bit
volumes:
- ./parsers.conf:/fluent-bit/etc/parsers.conf
- ./fluentbit.conf:/fluent-bit/etc/fluent-bit.conf
- ./geopoint.lua:/fluent-bit/etc/geopoint.lua
- ./GeoLite2-City.mmdb:/fluent-bit/etc/GeoLite2-City.mmdb
- /var/log/traefik:/var/log/traefik
Elasticsearch is a popular open-source search and analytics engine that can be used for a variety of tasks, including log analysis. It is a good choice for log analysis because it can be queried using complex queries, and it provides a REST API to cast queries directly in readable JSON format.
Elasticsearch uses a distributed architecture, which means that it can be scaled to handle large amounts of data. It also supports a variety of data types, including text, numbers, and dates, which makes it a versatile tool for log analysis.
To use Elasticsearch for log analysis, you would first need to index the logs into Elasticsearch. This can be done using a variety of tools, such as Logstash or Fluent Bit. Once the logs are indexed, you can then query them using Elasticsearch’s powerful query language.
Elasticsearch’s query language is based on JSON, which makes it easy to read and write. It also supports a variety of features, such as full-text search, regular expressions, and aggregations.
Elasticsearch creates a mapping for new indices by default, guessing the type of each field. However, it is better to provide an explicit mapping to the index. This will allow you to control the type of each field and the operations that can be performed on it. For example, you can specify that a field is of type ip so that it can be used to filter for IP address groups, or you can specify that a field is of type geo_point so that it can be used to filter by an specific location.
curl -XPUT "https://hostname/logstash-all" -H 'Content-Type: application/json' -d '{ "mappings": { "properties": { "@timestamp": { "type": "date" }, "agent": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "code": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "country": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "geohash": { "type": "geo_point" }, "host": { "type": "ip" }, "isocode": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "latitude": { "type": "float" }, "longitude": { "type": "float" }, "method": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "number_requests": { "type": "long" }, "path": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "protocol": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "referer": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "request_duration": { "type": "long" }, "router_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "router_url": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "size": { "type": "long" }, "user": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } }'
Bonsai.io is a managed Elasticsearch service that provides high availability and scalability without the need to manage or deploy the underlying infrastructure. Bonsai offers a variety of plans to suit different project requirements.
The hobbyist tier is more than enough for this kind of use case, which comes with a maximum of 35k documents, 125mb of data and 10 shards. At the moment of writing this article its a free, you don’t have to enter a credit card to use it.
In order to be compliant with the limits of the hobby tier, I use the following cronjob to remove old documents:
curl -X POST "https://hostname/logstash-all/_delete_by_query" -H 'Content-Type: application/json' -d '{ "query": { "bool": { "filter": [ { "range": { "@timestamp": { "lt": "now-10d" } } } ] } } }'
For my use case, 10 days retention is enough to be compliant with the plan limits.
Bonsai.io also provides a managed kibana service connected to the elasticsearch cluster.
There are certain limitations about the stack management, like there is no possibility to manage the index life cycle or alerting capacity.
Nevertheless, it provides basic functionality to create useful dashboards and discover patterns inside the logs.
Its interesting to see bot request trying to explode vulnerability from services like wordpress and also bot scrapping services.
The following stack provides a simple and cost-effective way to analyze logs. The computational footprint on your server is very low because most of the infrastructure is in the cloud. There are many freemium services, such as Bonsai.io and New Relic, that can be used to ingest and analyze logs.
Observability is important for infrastructure management, but it is also important to have alerting capabilities to detect and respond to threats. Unfortunately, these plugins are not typically included in the free plan, so you will need to upgrade to a paid plan to get them.
]]>I recently received a QuickFeather microcontroller from a Hackster.IO contest. One of the main features of this device is its built-in eFPGA, which can optimize parallel computations on the edge.
This post will explore the capabilities of this little beast and show how to run a machine learning model that was trained using Tensorflow. The use case will be focused for gesture recognition, so the device will be able to detect if the movement correspond to one alphabet letter.
The QuickFeather is a very powerful device with a small form factor (58mm x 22mm). It’s the first FPGA-enabled microcontroller to be fully supported with Zephyr RTOS. Additionally it includes a MC3635 accelerometer, a pressure, a microphone and an integrated Li-Po battery charger.
Unlike other development kits which are based on proprietary hardware and software tools, QuickFeather is based on open source hardware and is built around 100% open source software. QuickLogic provides a nice SDK to flash some FreeRTOS software and get started. There is a bunch of documentation and examples in their github repository.
Since the QuickFeather is optimized for battery saving use cases, it doesn’t include neither Wi-Fi nor Bluetooth connectivity. Therefore, the data can be only transferred using UART serial connection.
The on-board accelerometer is the main sensor for this use case. I use a USB-serial converter in order to read data directly from the accelerometer and transfer it to another host that is connected to the other end of the usb cable.
Data is captured and analysed using another machine. I personally connected a raspberry pi, which has also a small form factor, in order to have flexibility when performing the different gestures.
SensiML provides a web application to visualize and save data. This application is a python application that runs a flask webserver and provides nice functionalities such as capturing video at the save time in order to correlate to saved data. The code is available on github, so one can see how the code works and even propose some modifications, like I did.
I captured data from O, W and Z gestures as you can see in the following picture:
Once data is collected one need to label it so that one can teach a machine learning model how to associate a certain movement with a gesture. I used Label Studio, which is a open source data labelling tool. It can be used to label different kind of data such as image, audio, text, time series and a combination of all the precedent.
It can be deployed on-premise using a docker image, which is very handy if you want to go fast.
Once Label Studio stars, it has to be configured for a label task. For this case, the label task corresponds to a time series data. One can chose a graphical configuration using preconfigured templates or you can customized your self with some kind of html code. Here is the code I use to configure the data coming from X, Y and Z accelerometers.
<View>
<!-- Control tag for labels -->
<TimeSeriesLabels name="label" toName="ts">
<Label value="O" background="red"/>
<Label value="Z" background="green"/>
<Label value="W" background="blue"/>
</TimeSeriesLabels>
<!-- Object tag for time series data source -->
<TimeSeries name="ts" valueType="url" value="$timeseriesUrl" sep="," >
<Channel column="AccelerometerX" strokeColor="#1f77b4" legend="AccelerometerX"/>
<Channel column="AccelerometerY" strokeColor="#ff7f0e" legend="AccelerometerY"/>
<Channel column="AccelerometerZ" strokeColor="#111111" legend="AccelerometerZ"/>
</TimeSeries>
</View>
Label Studio has a nice preview feature, which shows how the labelling task will look with the supplied configuration. The following screenshot shows how the interface looks like for the setup process.
One of the nicest things from Label Studio is the fact that one can go really fast using the keyboard shortcuts. It also provides some machine learning plugins which make predictions with the partial labelled data. The following screenshot shows the interface for some labelled data.
From a machine learning perspective, the exported data should be a csv file with four different columns. Even is Label Studio is able to export in csv, it didn’t have the right format for me, instead it looks like the following:
timeseriesUrl,id,label,annotator,annotation_id
/data/upload/W.csv,3,"[{""start"": 156, ""end"": 422, ""instant"": false, ""timeserieslabels"": [""W""]}, ... ]",admin@admin.com,3
/data/upload/Z.csv,2,"[{""start"": 141, ""end"": 419, ""instant"": false, ""timeserieslabels"": [""Z""]}, ...]",admin@admin.com,2
/data/upload/O.csv,1,"[{""start"": 77, ""end"": 389, ""instant"": false, ""timeserieslabels"": [""O""]}, ...]",admin@admin.com,1
So I decided to export labels in json format and then build a python script to transform and combine them all. The following script transforms three json files from Label Studio into a single file with 4 columns AccelerometerX, AccelerometerY, AccelerometerZ and Label.
import numpy as np
import pandas as pd
df_all = pd.DataFrame()
LABELS = ['W', 'Z', 'O']
sensor_columns = ['AccelerometerX','AccelerometerY', 'AccelerometerZ', 'Label']
for ind, label in enumerate(LABELS):
df = pd.read_csv(f'{label}/{label}.csv')
events = pd.DataFrame(pd.read_json('WOZ.json')['label'][ind])
df['Label'] = 0
for k,v in events.iterrows():
for i in range(v['start'], v['end']):
df['Label'].loc[i] = v['timeserieslabels'][0]
df['LabelNumerical'] = pd.Categorical(df.Label)
df[sensor_columns].to_csv(f'{label}/{label}_label.csv', index=False)
df_all = pd.concat([df_all, df], sort=False)
df_all[sensor_columns].to_csv(f'WOZ_label.csv', index=False)
The resulting data can be directly used as a time series data and a machine learning model can be trained in order to recognise the patterns automatically. The following picture shows data for W, O and Z patterns.
SensiML provides a python package to build a data pipeline which can be used to train a machine learning model. One need to create a free account in order to use it. There is a lot of documentation and examples available online.
Pipelines are a key component of the SensiML workflow. Pipelines store the preprocessing, feature extraction, and model building steps.
Model training can be done using either SensiML cloud or using Tensorflow to train the model locally and the uploading it to SensiML in order to obtain the firmware code to run on the embedded device.
In order to train the model locally, one needs to build a data pipeline to process data and calculate the feature vector. This is done using the following pipeline:
Here is the python code for the pipeline
dsk.pipeline.reset()
dsk.pipeline.set_input_data('wand_10_movements.csv', group_columns=['Label'], label_column='Label', data_columns=sensor_columns)
dsk.pipeline.add_segmenter("Windowing", params={"window_size": 350, "delta": 25, "train_delta": 25, "return_segment_index": False})
dsk.pipeline.add_feature_generator(
[
{'subtype_call': 'Statistical'},
{'subtype_call': 'Shape'},
{'subtype_call': 'Column Fusion'},
{'subtype_call': 'Area'},
{'subtype_call': 'Rate of Change'},
],
function_defaults={'columns': sensor_columns},
)
dsk.pipeline.add_feature_selector([{'name':'Tree-based Selection', 'params':{"number_of_features":12}},])
dsk.pipeline.add_transform("Min Max Scale") # Scale the features to 1-byte
I use the TensorFlow Keras API to create a neural network. This model is very simplified because not all Tensorflow functions and layers are available in the microcontroller version. I use a fully connected network to efficiently classify the gestures. It takes in input the features vectors created previously with the pipeline (12).
from tensorflow.keras import layers
import tensorflow as tf
tf_model = tf.keras.Sequential()
tf_model.add(layers.Dense(12, activation='relu',kernel_regularizer='l1', input_shape=(x_train.shape[1],)))
tf_model.add(layers.Dropout(0.1))
tf_model.add(layers.Dense(8, activation='relu', input_shape=(x_train.shape[1],)))
tf_model.add(layers.Dropout(0.1))
tf_model.add(layers.Dense(y_train.shape[1], activation='softmax'))
# Compile the model using a standard optimizer and loss function for regression
tf_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
The training is performed by feeding the neural network with the dataset by batches of data. For each batch of data a loss function is computed and the weights of the network are adjusted. Each time it loops through the entire training set, then is called an epoch. In the following picture:
The confusion matrix provides information not only about the accuracy but also about the kind of errors of the model. It’s often the best way to understand which classes are difficult to distinguish.
Once you are satisfied with the model results, it can be optimized using Tensorflow quantize function. The quantization reduces the model size by converting the network weights from 4-byte floating point values to 1-byte unsigned int8. Tensorflow provides the following built-in tool:
# Quantized Model
converter = tf.lite.TFLiteConverter.from_keras_model(tf_model)
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.representative_dataset = representative_dataset_generator
tflite_model_quant = converter.convert()
There are more benefits by quantizing the model for Cortex-M processors like the Quickfeather, which uses some instructions that gives a boost in performance.
The quantized model can be uploaded to SensiML in order to obtain a firmware to flash to the QuickFeather. One can download the model using the jupyter notebook widget or in sensiml cloud application. There are two available formats:
The knowledgepack can be customized in order to light the QuickFeather led with a different colour depending on the prediction made. This can be done by adding the following function to the src/sml_output.c file.
// src/sml_output.c
static intptr_t last_output;
uint32_t sml_output_results(uint16_t model, uint16_t classification)
{
//kb_get_feature_vector(model, recent_fv_result.feature_vector, &recent_fv_result.fv_len);
/* LIMIT output to 100hz */
if( last_output == 0 ){
last_output = ql_lw_timer_start();
}
if( ql_lw_timer_is_expired( last_output, 10 ) ){
last_output = ql_lw_timer_start();
if ((int)classification == 1) {
HAL_GPIO_Write(4, 1);
} else {
HAL_GPIO_Write(4, 0);
}
if ((int)classification == 2) {
HAL_GPIO_Write(5, 1);
} else {
HAL_GPIO_Write(5, 0);
}
if ((int)classification == 3) {
HAL_GPIO_Write(6, 1);
} else {
HAL_GPIO_Write(6, 0);
}
sml_output_serial(model, classification);
}
return 0;
}
Finally the model can be compiled using Qorc SDK and flashed again to the QuickFeather.
One can use a Li-Po battery with the battery connector of the QuickFeather in order to have complete autonomy. Then using a nice spoon like the following one can improvise a magic wand 🪄:
The following video shows the recognition system in action, the colours mean he following:
Your browser doesn't support HTML5 video.
QuickFeather is a device completely adapted for tiny machine learning models. This use case provides a simple example to demystify the whole workflow for implementing machine learning algorithms to microcontrollers, but it can be extended for more complex use cases, like the one provided in the Hackster.io Climate Change Challenge.
SensiML provides provides nice tools to simplify machine learning implementation for microcontrollers. They provide software like Data Capture Lab, which capture data and also provides a labelling module. However, for this case I prefer to use Label Studio, which is more generic tool, that works for most use cases.
The notebook with the complete details about the model training can be found in this gist.
]]>Vimwiki makes easy for you to create a personal wiki using the Vim text editor. A wiki is a collection of text documents linked together and formatted with plain text syntax that can be highlighted for readability using Vim’s syntax highlighting feature.
The plain text notes can be exported to HTML, which improves readability. In addition, it’s possible to connect external HTML converters like Jekyll or Hugo.
In this post I will show the main functionalities of Vimwiki and how to connect the Hugo fast markdown static generator.
Vimwiki notes writing in Vim
Markdown notes converted into HTML
With Vimwiki you can:
One of the main Vim advantages is the fact that it’s a modal editor, which means that it has different edition modes. Each edition mode gives different functionalities to each key. This increases the number of shortcuts without having to include multiple keyboard combinations. In Vimwiki this allows to write notes with ease.
When I want to write some notes, I just open Vim and then I use
<Leader>w<Leader>w
to create a new note for today with a name based on the
current date. <Leader>
is a key that can be configured in Vim, in my case it’s comma character (,).
If I want to look at my notes I can use <Leader>ww
to open the wiki index
file. I can use Enter key to follow links in the index. Backspace acts a return
to the previous page.
I use CoC snippets to improve autocompletion. In markdown, I find this plugin very useful to create tables, code blocks and links. You can use snippets for almost every programming language, just take a look at the documentation.
When I want to preview the markdown file, I use <Leader>wh
to convert the current
wiki page to HTML and I added also a shortcut to open HTML with the browser.
In the following video you can see an example of this workflow in action.
Your browser doesn't support HTML5 video.
One of the advantages of digital notes are the fact that you can search quickly in multiple files using queries.
Vimwiki comes with a VimWikiSearch command (VWS
) which is only a wrapper
for Unix grep command. This command can search for patterns in case insensitive mode in all your notes.
An excellent way to implement labels and contexts for cross-correlating information is to assign tags to headlines. If you add tags to your Vimwiki notes, you can also use a VimwikiSearchTags command.
In both cases, when you are searching in your notes, the results will populate
your local list, where you can move using Vim commands lopen
to open the list, lnext
to go to the next occurence and lnext
for the previous occurence.
Vimwiki has a custom filetype called wiki, which is a little bit different from markdown. The native vimwiki2html command only works for wiki filetypes. If you want to transform your files to HTML using other filetypes, like markdown, you have to use a custom parser. Even if I’m not able to use Vimwiki native parser, I prefer markdown format because it’s very popular and simple.
These are some options to use as an external markdown parser:
I started using static website generators because it can also publish easily the notes as static webpages, which I wanted to publish in Github Pages.
My first option was Jekyll, which is the Github native supported static website generator. It’s easy to use and the syntax is very straightforward, but I started to regret it when I accumulated a lot of notes. Then I decided to use Hugo, which is claimed to be faster and since it’s been coded in Go, it has no dependencies. In the following table I show my compiling time results for both:
Compiling time | Jekyll | Hugo |
---|---|---|
10 pages | 0.1 seconds | 0.1 seconds |
150 pages | 48 seconds | 0.5 seconds |
I should say that I used Jekyll Github gems, which includes some unnecessary ruby Gems, so I think Jekyll performance can be increased. It’s a nice software that I use to publish this post, but still Hugo is faster.
The .vimrc
file contains vim configuration and it’s the place where one can
put the definition about Vimwiki syntax and writing directory. As
you can see in my configuration, I use markdown syntax and save my files under
~/Documents/vimwiki/
.
" ~/.vimrc
let g:vimwiki_list = [{
\ 'auto_export': 1,
\ 'automatic_nested_syntaxes':1,
\ 'path_html': '$HOME/Documents/vimwiki/_site',
\ 'path': '$HOME/Documents/vimwiki/content',
\ 'template_path': '$HOME/Documents/vimwiki/templates/',
\ 'syntax': 'markdown',
\ 'ext':'.md',
\ 'template_default':'markdown',
\ 'custom_wiki2html': '$HOME/.dotfiles/wiki2html.sh',
\ 'template_ext':'.html'
\}]
The custom wiki2html file correspond to a script which is executed to transform markdown into HTML. This scripts calls Hugo executable file and tells Hugo to use the Vimwiki file path as a baseurl in order to maintain link dependencies.
# ~/.dotfiles/wiki2html.sh
env HUGO_baseURL="file:///home/${USER}/Documents/vimwiki/_site/" \
hugo --themesDir ~/Documents/ -t vimwiki \
--config ~/Documents/vimwiki/config.toml \
--contentDir ~/Documents/vimwiki/content \
-d ~/Documents/vimwiki/_site --quiet > /dev/null
The complete version of my ~/.vimrc
can be found in my dotfiles repository.
Hugo projects can be easily published to Github using Github Actions. The following script tells GitHub worker to use Hugo to build html at each push and publish the HTML files to Github pages.
name: 🚀 Publish Github Pages
on: push
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Git checkout
uses: actions/checkout@v2
- name: Setup hugo
uses: peaceiris/actions-hugo@v2
with:
hugo-version: 'latest'
- name: Build
run: hugo --config config-gh.toml
- name: 🚀 Deploy to GitHub pages
uses: peaceiris/actions-gh-pages@v3
with:
deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
publish_branch: gh-pages
publish_dir: ./public
force_orphan: true
I like having one part of my notes published on Github Pages, at least the
configuration notes, which can be found in my Github
page. But there is also a part of notes
that I keep private, for example my diary notes, where I may have some sensible
information, so I keep it away from publication just by adding it to the
.gitignore
file. Here you can find my Github notes
repository.
I like that fact that Hugo has no dependencies since it’s written in Go, so it’s very easy to install, just download it from the github project releases page. In addition is also a blazing fast static website converter, you can find benchmarks in the internet.
I have been using Vimwiki very often, it allows me to take notes very easily and also find information about things that happen in the past. When people ask things about last month meeting I can find what I have written easily by searching by dates, tags or just words.
Publishing my notes to github allows me to have a have a place where I can keep track of my vimwiki configuration and also publish simple notes that are not meant to be a blog post, like my install for arch linux or my text editor configuration.
]]>A micro service project typically includes multiple docker containers, where each container includes a separated functionality.
These containers can communicate in a private network and map ports with the external network in order to expose services.
However, not every service includes a security layer, so it’s better to expose a single application that serves as a router which controls every incoming requests and send it to the right service. This avoid exposing a service like a whole database connection. Some applications that are able to act as a router are: Nginx, Apache server, Caddy and Traefik.
In this article I will show how to setup traefik using file system configuration and also how to implement offline metric analysis using GoAccess tool.
Traefik is a reverse proxy, which routes incoming request to microservices. It has been conceived for environments with multiple microservices, where a main configuration is done to set-up Traefik, and then it dynamically detects new services comming from docker, kubernetes, rancher or a plain file system. More information about traefik automatic discovery is available here.
This automatic discovery behaviour was the main thing that attracted me to use Traefik, unlike Nginx, which refuse to start if a declared service is not available. Traefik on the other side, it can run even if a declared service won’t run, and if the docker starts it will be automatically detected by Traefik.
Trafik has a modern web interface to graphically inspect configuration. It shows information about:
Traefik interface can be easily enabled in the configuration file. The following lines tell Traefik to serve the interface in the Traefik entrypoint (8080 by default). The debug option is useful for profiling performance and debugging Traefik.
api:
insecure: true
dashboard: true
debug: true
Here is a screen shot of the web interface, where one can see how one service is configured.
In the following gist you can find the complete configuration file for Traefik. The basic parameters to define are the entrypoints, where Traefik should be listening and the encryption method. The providers configuration can be done in other plain file, or by adding labels to docker, kubernetes, rancher, etc. In any case it dynamically detects changes on providers.
This configuration can be done in plain format if running outside a docker container, but it can also be done by setting labels to Traefik docker container.
By default Traefik will watch for all containers running on the Docker daemon,
and attempt to automatically configure routes and services for each container.
If you’d like to have more refined control, you can pass the
--providers.docker.exposedByDefault=false
option and selectively enable
routing for your containers by adding a traefik.enable=true
label.
Regarding HTTPS security, SSL connections can be easily configured in Traefik, one can use a self signed certificate or connect automatically to Let’s Encrypt in order to get an SSL certificate. The renewal is also taken into account by Traefik. HTTPS redirection is also available into Traefik parameters.
Traefik has been conceived to run as a docker container, but since it’s written in GO, then it’s possible to run the compiled version as a standalone file in several operating systems.
In the docker version, Traefik runs automatically when the container is power on and the logs are scoped to the standard output. However if you run the standalone file, then you have to configure Traefik as a system service. I used the excellent information from this Gerald Pape gist to configure the Traefik service.
I prefer the standalone version in development environments like the raspberry pi or jetson nano, where building docker images can be a little long.
GoAccess is a simple tool to analyse logs. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly. It can generate reports in terminal format, which is nice if your are connecting on SSH, but it can also generate CSV, JSON or HTML reports.
Alternatives for this services are Matomo, which has the advantage of being self hostable and open source. Then you can be sure about how your colected data is being used and that is not being sold to 3rd parties and advertisers. However, Matomo has an extra client side javascript library which is required in order to parse data, which is another dependency that I don’t want for internal off-line environments.
Other popular alternative is Google Analytics, which has very powerful reports and multiple of options that go beyond the scope of this article. The only problem is that it’s not privacy compliant.
What makes GoAccess interesting, is that it generates detailed analytics based purely on access logs from a web server, such as Apache, Nginx or in my case Traefik. It’s written in C, and features both a terminal interface, as well as a web interface. The way it’s designed to be used is by piping the access.log contents into the GoAccess binary and providing any number of switches to customize the output. Switches such as which log format you’re sending it, as well as how to parse Geolocation from IP addresses.
In the following image you can see an example for GoAccess HTML dashboard. On the top there is global information about the number of total requests, the number of unique visitors, the log size, the bandwidth, etc.
GoAccess can be called using the command line, you can configure log format
using a command line parameter or using a configuration file. Default
configuration file can be found at /etc/goaccess.conf
, but you can also pass
other configuration file using --config-file
option.
Default output format is in the command line, but one can configure an html
using a specific output file. This option will create a static html report,
which can be continuously updated using the --real-time-html
option.
The following code shows the systemctl file that I use to configure GoAccess as a service for real time use.
[Unit]
Description=Goaccess Web log report.
After=network.target
[Service]
Type=simple
User=root
Group=root
Restart=always
ExecStart=/usr/bin/goaccess -a -g -f /var/log/traefik/access.log -o /var/www/html/report.html --real-time-html
StandardOutput=null
StandardError=null
[Install]
WantedBy=multi-user.target
GoAccess doesn’t include a static web server, so it can not expose the produced html by himself. But one can easily configure an Nginx static server to expose the static files, as show in the following Nginx virtual server:
server {
listen 8082;
listen [::]:8082;
server_name locahost;
gzip on;
gzip_types text/plain application/xml image/jpeg;
gzip_proxied no-cache no-store private expired auth;
gzip_min_length 1000;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
try_files $uri $uri/ =404;
}
}
Traefik is a static webserver which is well adapted for dynamic configurations. Even if still a young project and is not as performant as Nginx, it has an interesting approach and some nice features. For example in docker applications, it automatically knows the internal IP address of a service to redirect the incoming request.
GoAccess is a very good tool to provide insights from logs in a close environment where you can not share your stats with the exterior. Since it has been written in C, the reading performances are very good, being able to parse 400 millions of hits in 1 hour and 20 minutes, according to GoAccess FAQ.
]]>The historical web interface of Mopidy is MusicBox, which started being developed in 2013. It has the basic functionalities of Mopidy and it’s compatible with several web browsers but the problem is that it doesn’t include support to control Snapcast sources.
There are more recent web interfaces like Iris, with a gorgeous design and support for Snapcast control, but it collects utilization analytics for each client, which I prefer to avoid.
Then, I decided to develop my own web interface using the state of the art web technologies which are described in this post.
I choose the Svelte framework as the core of the web interface. This framework is very light and I like the syntax because I feel that I’m only writing html, css and js but behind the scenes the code is completely optimized at build time.
I also chose Sapper (short for Svelte app maker) to be the glue between the different Svelte components. In the following section I’m explaining how I used those Sapper and Svelte together.
Svelte framework is relatively new (released in 2016) but it’s becoming very popular because it’s very light and fast. This is mainly due to the fact that it optimizes the code to manipulate the DOM directly using vanilla JavaScript, which is different from traditional frameworks like React or Vue, which need a framework in the browser to manipulate the DOM.
One of the advantage of Svelte is the reactivity system. This system is inspired in the philosophy of Excel cells, that are linked one to the other using formulas. When a value in a parent cell changes, all the linked cells are updated automatically. The same happens with reactive variables in Svelte, the variables are updated depending of a relation dependency graph.
I use reactive variables for saving information about the current track that it’s playing, the songs that are present in the tracklist, the songs that are present in a playlists and the results of a search action.
Another advantage of Svelte, is the fact that animation can be easily adapted to components. I add some drag and drop events to order the tracks playlists as you can see in the following video:
Your browser doesn't support HTML5 video.
The order of the items in the tracklists is synchronized with the values in the backend, so when the next song button is pressed, the song that has been dragged is played.
Svelte has a Read–Eval–Print Loop (REPL) page, where people can share their snippets. I think it’s a great way to test small things and share them with everybody. For instance, here is a REPL page for the algorithm that moves the items in theplaylists.
I used the Bulma CSS framework because of it’s simplicity. I added the SCSS version as a global variable using some components. Once the design is stabilized, I can use local CSS in each component to optimize the application.
Sapper has been developed by the Svelte team, so it follows the same principles and simplicity of the Svelte family. However, this framework is not still as popular as Svelte and also not as mature.
I use Sapper as a routing application for the different pages:
Sapper is also in charge of preparing the server side rendering mode, where part of the JavaScript code is executed on the server side, so that the browser can load faster the webpage.
Mopidy core API runs on a backend which recently has reached version 3.0. This
means that it needs to run on python 3.7, which is not installed by default
in all desktop environments. To avoid version compatibility and break internal
dependencies I also created a Dockerfile
with support of Mopidy 3.0, which
helps a lot for local development.
A Makefile is used to inject envrionement variables, like the location of the music and playlist folder. These can be easily tunned depending on the running environment.
Snapcast is a multiroom client-server audio player, where all clients are time synchronized with the server to play perfectly synced audio. It’s not a standalone player, but an extension that turns your existing audio player into a Sonos-like multi room solution.
I have been using it in three Raspberry Pi, to have the same music in different house environments. I works well with Mopidy and initial configuration is very minimalistic. There is one master with the Mopidy instance and Snapcast server and there is a Snapcast client in every other.
Sanpcast also provides a JSON/RPC control API to communicate with the server using websockets. This protocol provides a continuous link between the client and the server. Messages are send when there are changes or events, which allows the client to have a notification when there are change in the server.
I implemented this communication in Muse, so that one can control the volume of the Snapcast clients. The following image shows the Snapcast control panel and the sound from three devices: raspi, raspimov and raspicam.
There are two main components of Muse web client:
The final destination of the python extension is the PyPi repository, where python users can download and install Muse. The publishing action can be made manually using python wrappers like twine. However, there is always the risk of making a human mistake, like publishing the wrong version. This is why I prefer to delegate this action to Github.
Here is a description of the deploying pipeline of the Muse package using Github Actions:
setup.cfg
and package.json
There is still plenty of room for improvement for Muse. I would like to add features like integrating third party services like Discogs or Genius, improving user experience or improving the design. If you think about other nice features to add, you can open an issue on the github page of the project.
I’m also satisfied of having an automatic deploying workflow. I used different github actions developed by others and also contributed to improve one of them.
The code of Muse is available at this github
repository and the python package is
available at PyPi, you can install it
with sudo python -m pip install Mopidy-Muse
, use it without restriction.
These restriction are:
Not having external javascript calls makes really difficult to include some dynamic functionalities like fetching external content or calling API services. It’s the case for a search bar which needs to fetch some content and render the results. This gets more complicated for static web content that is hosted in servers like Netlify, Gitlab or Github Pages. These options don’t provide any backend and thus no dynamic content.
Even if it seems complicated, it’s not impossible. One can adapt a website to AMP format and include dynamic functionalities. For example, this post shows how to implement a javascript call in AMP pages and implement a search bar which allows searching for content in a static website hosted on Github pages.
There are various alternatives to integrate a search function in a static website:
I personally prefer the approach from SimpleJekyllSearch which exposes an endpoint with the posts information and which can be then used in AMP components. The front-matter informs Jekyll that is a special page so it takes into account the defined parameters in order to render it. The front-matter defines the following properties:
/api/github-pages.json
.Jekyll allows looping over the posts using the site.posts
array and get data from each post.
Here’s how my endpoint file looks like:
---
limit: 100
permalink: /api/github-pages.json
---
[{% for post in site.posts limit: page.limit %}{"title": "{{ post.title }}", "description": "{{ post.description }}", "image": "{{ post.image.path }}", "thumb": "{{ post.thumb.path }}", "date": "{{ post.date | date: "%B %d, %Y" }}",{% if post.source %}"source": "{{ post.source }}",{% endif %}{% if post.link %}"link": "{{ post.link }}",{% endif %}{% if post.categories %}"categories": [{% for category in post.categories %}"{{ category }}"{% if forloop.last %}{% else %},{% endif %}{% endfor %}],{% endif %}{% if post.categories == nil %} "categories" : [], {% endif %} "url": "{{ post.url }}", {% if post.tags %}"tags": [{% for tag in post.tags %}"{{ tag }}"{% if forloop.last %}{% else %},{% endif %}{% endfor %}]{% endif %}{% if post.tags == nil %}"tags": []{% endif %}}{% unless forloop.last %},{% endunless %}{% endfor %}]
The content of the file uses the liquid syntax to iterate between the posts. The result is an array that contains the following information for each post:
There are AMP components that can use the static API:
For example, if we define an HTML input element, it’s possible to add a listener that takes the input value and filters the results using some javascript functions (not all ES6 are allowed). The javascript indexOf function returns an int
larger than -1 if the input element is found, so one can use it as a simple search like this:
allArticles.filter(a => a.title.indexOf(event.value) != -1)
For more complex research, in multiple fields, the components looks like the following:
<amp-state id="allArticles"
src="/api/github-pages.json"></amp-state>
<input class="input" type="text" on="input-throttled:AMP.setState({filteredArticles: allArticles.filter(a => ((a.title.toLowerCase().indexOf(event.value.toLowerCase()) != -1) || (a.description.toLowerCase().indexOf(event.value.toLowerCase()) != -1) || (a.tags.join(' ').toLowerCase().indexOf(event.value.toLowerCase()) != -1) || (a.categories.join(' ').toLowerCase().indexOf(event.value.toLowerCase()) != -1) ) ), inputValue: event.value }), top-blog-page.scrollTo(duration=200)" placeholder="🔍 Search" [value]="inputValue">
AMP provides some components to process the JSON object that is binned with amp-bind:
The amp-list component loads the filteredArticles object, the height is controlled dynamically with the number of elements in the object.
By default amp-list expects an object with the format {"items": [obj1, obj2]}
,
it’s possible to configure the location of the items list using the items parameter.
Mustache syntax allows to filter properties of the component using the conditional notation. Here are the definition of the special characters:
Please note that static websites generators like Jekyll need to escape mustache templates using { raw } and { endraw } elements. Otherwise the results would not be rendered. Here is how looks like the final combination of the two elements:
<amp-list height="500"
[height]="(400) * filteredArticles.length"
layout="fixed-height"
items="."
src="/api/github-pages.json"
[src]="filteredArticles"
binding="no">
<div placeholder>Loading ...</div>
<div fallback>Failed to load data.</div>
<template type="amp-mustache">
{% raw %}
<div class="box">
<div class="columns is-multiline">
<div class="column is-3-desktop is-12-mobile">
<a class="image" href="{{#link}}{{ link }}{{/link}}{{^link}}{{ url }}{{/link}}" data-proofer-ignore>
<amp-img
width="300" height="200"
src="{{ thumb }}"
srcset="{{ thumb }} 400w, {{ image }} 900w"
alt="{{ title }}" layout="responsive"></amp-img>
</a>
</div>
<div class="column is-9-desktop is-12-mobile">
<div class="content">
<a class="title is-6" href="{{#link}}{{ link }}{{/link}}{{^link}}{{ url }}{{/link}}" data-proofer-ignore>{{ title }}</a>
<p class="is-size-7">
{{ description }}
</p>
<div class="tags has-addons">
<span class="tag"><i class="fas fa-calendar-alt"></i> {{ date }}</span>
{{#tags}}<a class="tag" href="/blog/tags#{{.}}" data-proofer-ignore> {{.}}</a>{{/tags}}
{{#categories}}<a class="tag is-link" href="/blog/categories#{{.}}" data-proofer-ignore> {{.}}</a>{{/categories}}
{{#source}}
<span class="tag is-danger"><i class="fas fa-external-link-alt"></i> {{ source }}</span>
{{/source}}
</div>
</div>
</div>
</div>
</div>
{{% endraw %}}
</template>
</amp-list>
The final version of the search bar can be seen in the following video. The search bar filters the posts at every key stroke. An additional AMP event is called to place the window at the top of results list and the amp list height is controlled by the number of posts that are shown.
The search bar is always visible using a position: fixed
css property.
The code can be found at the github repository of this page.
Your browser doesn't support HTML5 video.
I have seen different opinions on the effect of using AMP format. For my personal point of view, my SEO has increased since I started using this format and also the page load speed. The AMP rules impose good web format that guaranties the best performances. The traffic data of my website can be found at this github page.
AMP integrates smoothly with API endpoints defined in static website build tools like Jekyll. The endpoint with the contents of the blog is a small json, which can be filtered using simple javascript functions allowed in AMP.
I’m happy with the results since, the json is still very small (less than 19kb) and I think that when I write more posts I can add more advanced functions of amp-list to include a pagination.
]]>Arduino is a big player in the IoT field, their open source platform allows writing software for microcontrollers. These devices can communicate using standard protocols like WiFi, Bluetooth or Low Power Wide Area (LPWA) networks. LPWA protocol is adapted for applications that need to communicate in remote places where there is no WiFi available. Sigfox is a great player in this field.
Sigfox partnered with Arduino to produce the Arduino MKR FOX 1200, which can connect easily with Sigfox network. The Arduino MKR FOX 1200 is a designed to run Arduino code, use Sigfox network to communicate and have a battery which run for long periods of time.
In this article I will show how to set up the device, send messages to the Sigfox network and create a serverless architecture to display the collected messages.
Arduino MKRFOX1200 has been designed to offer a practical and cost effective solution for makers seeking to add SigFox connectivity to their projects with minimal previous experience in networking.
Its USB port can be used to supply power (5V) to the board. It has a screw connector where to attach a 3V battery pack. The board consumes so little that it runs on two 1,5V batteries type AA or AAA for a really long time.
It has 8 I/O pins allows for different applications. For more information about the board, you can see Arduino’s page.
The board can be programmed using C or C++. Arduino provide it’s own IDE to write the software, compile it and send it to the device.
For command line lovers there are other options like platformio.io or ino package, but it involves dealing with complex Makefiles, so I still use Arduino IDE for matter of simplicity.
The first part is to get the device ID and PAC number. Sigfox provide an Arduino sketch file, which connects to the board and shows the ID and PAC number in the serial console.
This information is need in the sigfox backend to configure the device and receive the messages. Sigfox has a very well documentation to register their devices.
For development environment, I connected the device from the micro usb input to my laptop usb port. This allows me to power the device and use the serial communication at the same time.
For production environment, I power the device using the battery connection with 2 AA batteries. The constructor announces at least 6 months of power, which I think might be possible using the low power mode between sending messages.
The following image shows a Fritzing schema, which shows clearly how to connect each wire. This is very useful when you want to recreate the project.
I used the sample code from Antoine de Chassey to get information from the DHT22 and send it to sigfox cloud.
Sigfox allows to send up to 140 messages per day, each message can have a maximum length of 12 bytes. This is enough for cases when you need to send only small amount of information, like data from measurements.
As a reminder, 1 byte == 8 bits. So if we want to sent 1 unsigned byte, it can be from 0 to 255. If we need to add a sign, it can be form 0 to 127. For 2 bytes: the range is 0-65535 for unsigned and the half for signed values.
In Antoine de Chassey’s code, this message is passed using the following C++ structure:
typedef struct __attribute__ ((packed)) sigfox_message {
int16_t moduleTemperature;
int16_t dhtTemperature;
uint16_t dhtHumidity;
uint8_t lastMessageStatus;
} SigfoxMessage;
This message contains the following elements:
Finally, we only used 7 bytes in each message. Smaller message reduces transmission times, and increases battery.
Sigfox API allows to obtain messages from all of the registered devices within
a maximum date history of 3 days and a maximum of 100 messages. The endpoint
correspond to the route /devices/${DEVICE_ID}/messages
, where DEVICE_ID
is
the id of the device. More information about can be found at the Sigfox endpoint
documentation.
I used the library Axios in Javascript to do a get request to the API:
const params = {}
if (limit) {
params.limit = limit
}
if (start) {
params.since = start
}
if (end) {
params.before = end
}
const myurl = `https://api.sigfox.com/v2/devices/${DEVICE_ID}/messages`;
const resulting = await axios.get(myurl, {
params: params,
auth: {
username: UNAME,
password: UPASS
}
});
The variables UNAME and UPASS correspond to authentication credentials, which can be obtained in your Sigfox account page as you can see in the Sigfox API documentation. The parameters limit, since and before allows to control the time range of the response.
The result of from the API seems like 7707d713649c000078c40000, which corresponds to the message in bytes encoded in a hexadecimal string. I used the npm sigfox-parser package to parse this output.
var parser = require('sigfox-parser')
function convertPayload(payload) {
var parsed = parser(payload, format);
var moduleTemp = (parsed.moduleTemp / INT16_t_MAX * 120).toFixed(2);
var dhtTemp = (parsed.dhtTemp / INT16_t_MAX * 120).toFixed(2);
var dhtHum = (parsed.dhtHum / UINT16_t_MAX * 110).toFixed(2);
var heatIndex = computeHeatIndex(dhtTemp, dhtHum, false).toFixed(2);
var result = {
moduleTemp: moduleTemp,
dhtTemp: dhtTemp,
dhtHum: dhtHum,
heatIndex: heatIndex
}
return result
}
The parser
function takes two inputs:
var format = 'moduleTemp::int:16:little-endian dhtTemp::int:16:little-endian dhtHum::uint:16:little-endian lastMsg::uint:8';
Once the result has been parsed, it can be used as a Javascript object. I used ChartJS library to plot the result as you can see in the demo page powered by a free heroku instance. The server keep the token save from the web client.
One of the biggest advantage of this serverless architecture is the fact that it doesn’t involve a database, which reduces the complexity of the web application. The code source of the application can be found on this github page.
The Arduino MKR FOX 1200 allows to easily connect IoT with Sigfox network and that way to the cloud. The device consumes so little that it runs on two 1,5V batteries type AA or AAA for a really long time. The constructor announced at least 6 months, which I’m currently testing right now. This kind of device is completely adapted for outdoor applications that don’t have WiFi connectivity or electrical power.
Sigfox proposes an API which allows to get the device messages and simplifies a dashboard architecture.
I would like to thank Sigfox’s team for organizing the Hackster.io Sigfox Universities Challenge 2019, where I got the top 10 price and obtained an Arduino MKR FOX 1200.
I tried the device on prudction environment and having the debug mode on (which means that the led light is turned every time that there is a message transmission) and the battery lasted 3 months. I guess that the led lights reduce the expected life of 6 months.
Having the debug mode off (no led light) the device battery should last more than 4 months, I didn’t get to know how much more it lasted because my Sigfox subscription year ended, so I could use the network any more.
When debug mode was off, I had some issues to wake up the device after entering to the power saving issue. Hopefully, I found this useful post in the Arduino forum which solved the problem.
]]>