Problems staying connected with MQTT/TLS

[pmeskoni]

I am working on what I thought was a simple implementation of an auto reconnect from my MQTT secure client and I have run into 2 problems that I am having trouble fixing.

I am using Mongoose 7.11 with OpenSSL 3.0.12. I cannot yet go to Mongoose 7.12 or greater because other parts of my code rely on the removed CHUNK event. Otherwise I would try the latest Mongoose with my issue. My application can have many individual MQTT clients, all being managed by same Mongoose manager instance. Mosquitto is my MQTT broker.

I was able to get things working for the reconnect.

Issue 1: I then wanted to get more detailed TLS information, so I found the mg_tls_cb implementation in Mongoose 7.13 and patched my Mongoose to include the handling of that callback. I then sent the error message back to my application via a MG_EV_USER event initiated from within the OpenSSL callback. This seems to be working ok, but I have noticed that I don't always get the OpenSSL error for a failed connect attempt.

For example, I have one client trying to pass a bad certificate. The connection will fail, I will wait a few seconds then try the connection again. Sometimes I will get an appropriate message from OpenSSL, sometimes I will get nothing, the callback doesn't get invoked. It seems to depend on where in Mongoose it figures out that the connection failed. Sometimes I just get a 10054 or 10035 socket error, other times a full SSL reported error.

Issue 2: Things get even stranger when I run a stress test with 8 clients. 5 try to connect with assorted TLS issues, 3 connect just fine and start publish/subscribe operations. At random times, the connected MQTT clients will disconnect with the OpenSSL errors that other clients are intentionally failing with. If I get the 5 failing clients to all fail with non-TLS issues I don't seem to have this problem. This has me really baffled.

Any hints for what to investigate? I've already spent more than a week investigating this.
Thanks for any assistance.

[scaprile]

I'm sorry, I don't really get what the problem is or might be.
If the MQTT connection drops, you start a new one, be it TLS or not. Our tutorials suggest how to deal with that https://mongoose.ws/documentation/tutorials/mqtt-client/#timer-function https://mongoose.ws/documentation/tutorials/error-handling/ and we also have a full example https://github.com/cesanta/mongoose/tree/master/examples/mqtt-dashboard/device
Basically, you just try to connect and wait a reasonable time.

[pmeskoni]

Yes, I can handle the reconnecting. My question is, why does my connection, which is up and functioning normally, get disconnected simply because I am making other connections?

[scaprile]

It should not, if you follow the guidelines in the documentation, examples and tutorials available, have enough stack and heap, and don't do multithreading

NOTE: Since Mongoose's core is not protected against concurrent accesses, make sure that all mg_* API functions are called from the same thread or RTOS task.

[pmeskoni]

I have plenty of stack, heap and all connection activity is on the same thread. If I make each of my connections non-secure connections i do not see this problem. I thought some experts would have some advice for me. Seems not. I'll continue to debug and review again all of the documentation, examples and tutorials you have pointed me to.

[pmeskoni]

Let me explain it this way.
I have 3 clients, all set to retry if they are disconnected or fail to connect:

• Conn 1 has a bad CA and gets this error (most of the time) from a connect attempt:
  14AC0000:error:0A000418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl\record\rec_layer_s3.c:1586:SSL alert number 48
• Conn 2 has a bad certificate and gets this error (most of the time) from a connect attempt:
  14AC0000:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:ssl\record\rec_layer_s3.c:1586:SSL alert number 116
• Conn 3 has no issues and connects properly

When I run conns 1 and 3 at the same time, conn 3 will randomly get disconnected with "alert number 48" errors
When I run conns 2 and 3 at the same time, conn 3 will randomly get disconnected with "alert number 116" errors

I guess I'm just asking if anyone has seen an issue like this.
My application also supports mbedTLS, so I will be trying this with mbedTLS today.

[scaprile]

Please note that if anyone would have seen an issue like this, we'd have already fixed it or there would be an open issue and we'd be working on it.
We can't work out of thin air, if there is an actual issue, please open an issue, honor the issue template so we can know your scenario and reproduce the issue if necessary, and let us know exactly what you are doing. Attach a smallest minimum possible example to reproduce your issue.
But please, use 7.13 because we do not support legacy releases.
If you are a paying customer, please contact by email as per contract.