Prometheus exporter for I2C and SPI sensors running on the Adafruit Feather M0 or M4 platforms. Very rudimentary HTTP server that implements minimal amount of the spec, enough to work with Chrome, curl, and Prometheus itself.
I wasn’t sure which platform would work best for this project and picked up a few different Adafruit devices to try, along with a Raspberry Pi Zero W that I already had and various Arduino-compatible devices.
While C has existing HTTP servers that I could use, Python has a great Prometheus client and I was curious to try CircuitPython.
Hardware
The initial candidates included:
The Feather M0 and M4 devices had the best combination of profile, peripherals, and performance for my needs. The code I ended up with can collect and serve sensor data many times per second, while Prometheus only needs to scrape it once each minute.
Server Setup
Using CircuitPython’s socket library,
I implemented very rudimentary connection handling:
conn, addr = self.http_socket.accept()
print('Connection: {}'.format(addr))
req = conn.recv(1024).decode(http_encoding)
req_headers, req_body = self.parse_headers(req)
print('Headers: {}'.format(req_headers))
handler = router.select(req_headers['method'], req_headers['path'])
resp = handler(req_headers, req_body)
There is no keep-alive and properly reading headers is still an open issue,
but the 1024 character limit has been enough to request GET /metrics.
Attaching Sensors
The sensors can be attached to any hardware pins, but input values may need to be converted into Prometheus base units before being passed to the client.
I set up a few I2C sensors on a breadboard and began collecting data from them:
# set up sensors
sensor_bme680 = adafruit_bme680.Adafruit_BME680_I2C(i2c)
sensor_bme680.sea_level_pressure = 1013.25
sensor_si7021 = adafruit_si7021.SI7021(i2c)
# set up prometheus
registry = CollectorRegistry(namespace='prom_express')
metric_gas = Gauge('gas', 'gas from the bme680 sensor', registry=registry)
metric_humidity = Gauge('humidity', 'humidity from both sensors', ['sensor'], registry=registry)
metric_pressure = Gauge('pressure', 'pressure from the bme680 sensor', registry=registry)
metric_temperature = Gauge('temperature', 'temperature from both sensors', ['sensor'], registry=registry)
The main loop reads each sensor and sets the corresponding Gauge, then waits for an incoming connection:
while True:
metric_gas.set(sensor_bme680.gas)
metric_humidity.labels('bme680').set(sensor_bme680.humidity)
metric_humidity.labels('si7021').set(sensor_si7021.relative_humidity)
metric_pressure.set(sensor_bme680.pressure)
metric_temperature.labels('bme680').set(sensor_bme680.temperature)
metric_temperature.labels('si7021').set(sensor_si7021.temperature)
try:
server.accept(router)
except OSError as err:
print('Error accepting request: {}'.format(err))
server = start_http_server(8080, address=eth.ifconfig()[0])
except Exception as err:
print('Unknown error: {}'.format(err))
The endpoint made it very easy to watch the temperature in my office change over the day with the lights on. While
collecting the sensor data on-demand is not the best practice, per the Prometheus docs, setting a timeout on the socket results in an OSError. An issue has since been opened about that particular error.
Benchmark
While writing the HTTP server, I tested with curl and occasionally Chrome. Both sent requests that were, at least for
the kilobyte I read, in line with the spec and both were happy with my minimal responses.
To push things further, I ran the wrk load test tool against the metrics endpoint:
> ./wrk -t 1 -c 1 -d 5s http://10.2.2.255:8080/metrics
Running 5s test @ http://10.2.2.255:8080/metrics
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 82.34ms 21.98ms 108.68ms 67.24%
Req/Sec 12.04 4.43 20.00 75.56%
58 requests in 5.01s, 45.04KB read
Socket errors: connect 0, read 1, write 21, timeout 0
Requests/sec: 11.59
Transfer/sec: 9.00KB
Without the (method, path) tuple routing, things were substantially faster:
> ./wrk -c 1 -d 60s -t 1 http://server:8080/
Running 1m test @ http://server:8080/
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.64ms 485.57us 12.81ms 97.75%
Req/Sec 111.60 5.21 121.00 71.00%
2222 requests in 20.61s, 671.83KB read
Socket errors: connect 0, read 1, write 2743, timeout 0
Requests/sec: 107.82
Transfer/sec: 32.60KB
Since my Prometheus instance is configured to scrape every 60s, even the slower result with routing is more than sufficient, but that part could use some serious optimization. It worked, though, and served valid-enough responses.
Socket Errors
While load testing, I started running into occasional OSErrors from the socket:
Connection from ('client', 8080)
Accepting...
Connection from ('client', 8080)
Accepting...
Error accepting request: [Errno 128] ENOTCONN
Binding: server:8080
Traceback (most recent call last):
File "code.py", line 90, in <module>
File "code.py", line 87, in main
File "code.py", line 87, in main
File "code.py", line 55, in bind
File "/lib/prometheus_express/http.py", line 11, in start_http_server
OSError: 4
...
Connection from ('client', 8080)
Accepting...
Error accepting request: 7
Binding: server:8080
Accepting...
These are passed on from the OS by CircuitPython, so I opened an issue looking for more information and found that there is an open issue around better handling. Until then, catching the errors and rebinding seems to work reliably for days at a time, the longest I’ve tested so far.
Results
These Feather devices are more than sufficient for serving Prometheus metrics, even with my questionable HTTP server.