MQTT TLS on Google cloud - invalid JWT generated

I'm trying to connect to our thing on Google Cloud IoT Core, but I get an error when trying to connect using  

status = nxd_mqtt_client_secure_connect(&g_mqtt_client0, &broker_address, NXD_MQTT_TLS_PORT,

I figure out the cause is in nx_secure_tls_session_start.c, function _nx_secure_tls_handshake_process that Fails with 56 (0x38) error (NX_NOT_CONNECTED 0x38)

Then looking at the JWT that is generated, I've checked on (a website) and gets me invalid JWT.

Note that I have RootCA certificate, device certificate and device private key (in pem format) and they work just fine with Python (just checking certificates and key are correct, and server address/ TLS is working), in fact the JWT generated in python gets me "genuine" under the same website validator.

I dont' get any errors while loading certificates and key on netx secure, so I'm guessing is the way I pad or calculate the JWT is not correct, I'm using the same base64 and JWT create functions that you use in the MQTT_tls google cloud example project.

Root certificate and device certificate are converted first to .der (using openssl) and then to binary array using the hexy.exe tool.

Private key is converted first to pkcs1.pem, then to .der and finally to binary array.

All the CA certificate, device certificate and private key are loaded into some header files and included in my project.

Some code, after I establish an IP layer through Wifi, following:

//> SNTP gets time, I save timestamp and call get_time() when I need it (to generate JWT) - no errors here

//> DNS to get IP address of Google Cloud endpoint - no errors here

//>Init of mqtt packet pool and client - all good

Then init and store of certificates, I get no errors in the following:

status = nx_secure_tls_session_create(&g_mqtt_client0.nxd_mqtt_tls_session,
(NX_SECURE_TLS_CRYPTO *) &nx_crypto_tls_ciphers_synergys7,

status = nx_secure_tls_session_packet_buffer_set(&g_mqtt_client0.nxd_mqtt_tls_session,


char *temp_str = "unused";

for(UINT i = 0; i < sizeof(temp_str); i++)
username[i] = temp_str[i];

//Password is the JWT generated using realtime timestamp (epoch) and IoT thing private key
//rsa_private_pkcs1, RSA_PRIVATE_LEN
jwt_expiration = JWT_EXPIRATION_SECS;

/* Generate JWT */
if (jwt_create(password, sizeof(password), &jwt_expiration,
rsa_private_pkcs1, RSA_PRIVATE_LEN, (CHAR*) &project_id))
HSP_APP_LOG(HSP_LOG_LEVEL_ERROR,"Unable to create JWT. Aborting.\r\n");

HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"JWT token created successfully %s\n============", password);
//set login
status = nxd_mqtt_client_login_set(&g_mqtt_client0, username, strlen(username),
(char *)password, strlen((char *)password));
HSP_APP_LOG(HSP_LOG_LEVEL_ERROR,"tnxd_mqtt_client_login_set error %d", status);

//---------------------------------- after this, I get back the error described above.

///> In nx_secure_tls_session_start.c
///> function _nx_secure_tls_handshake_process Fails with 56 (0x38) error
///> (NX_NOT_CONNECTED 0x38)

status = nxd_mqtt_client_secure_connect(&g_mqtt_client0, &broker_address, NXD_MQTT_TLS_PORT,

HSP_APP_LOG(HSP_LOG_LEVEL_ERROR,"nxd_mqtt_client_secure_connect error %d", status);


I include my project file, so maybe somebody can address me on how to properly generate the JWT, as the following is not getting me a proper JWT.

Is the certs/private key conversion and handling correct?

Do I need to change padding/ crypto functions in the JWT header/payload/signature generation?

Unfortunately I can not find any more info on this topic, as normally a jwt library is used to generate hash and create the token, but I want to make use of the hardware secure engine.

Any info/hints very welcome. 

I can share certs/keys if you want to check, but basically If I get a valid JWT generated I think we are half way there.

Generated JWT:


(note the == characters generated, I suspect shouldn't be there)

and so: [0000002235][new_thread0_entry.c:0395] - nxd_mqtt_client_secure_connect error 65541

I'm on S5D9, SSP1.7.8 e2Studio. NetX Duo.

Thanks a lot.


  • Hi S,

    Have you try modifying  the Base64urlEncoder() in Base64Decode.c to strip the padding, "="? Would it pass checking

    The hash/signature would definitely be different without '=', As the Header/payload is available, the receiver could have use it to verify the signature,   

  • Hi,

    Yes I've tried but still doesn't pass the check. Is there a valid Base64url encoder library that actually works with Google IoT Core?

    Also, it has something to do with the pkcs1 padding in the jwt_create() ?


  • Have you taken a Wireshark trace of the network activity to see what is going on in the communication between the Synergy board and the Google servers?

  • Yes, I did, is my uBlox NINA Wifi DHCP address. Asus is our wifi router, external activity is captured from the internal wifi network but I think you can extract the big picture.

    Please find it here:

    I did also 2 additional captures with just my laptop (wireshark running) and uBlox connected to my WiFi hotspost.

    One using ECC certificate and the nx_secure_tls_ecc_initialize() (with the proper endpoint for ECC cert) and the RSA cert version.

    Interestingly, in these two last captures, I can't see the DNS request, even though I get it and resolve the IP of google cloud endpoint correctly from Synergy, also SNTP time query is not showing, and again I get the time correctly from the SNTP server.

    I think something is going on when resolving uBlox external IP, gateway IP, but again, I can perform SNTP queries, DNS queries, and HTTP Post successfully with the same stack.

  • I don't see the DNS query of the Synergy device looking up the address of the Google mqtt server:-




            /* Look up IPv4 addresses to record  IPv4 addresses in record_buffer and return the IPv4 address count.  */
            status = nx_dns_ipv4_address_by_name_get(&g_dns0, (uint8_t *)GCP_IO_SERVER, &gcp_io_ip_address[0], sizeof(gcp_io_ip_address), &record_count, 500/*NX_IP_PERIODIC_RATE*/);

    in any of the the wireshark logs. The only DNS queries in any of the logs you posted are in the file "nina-mqtt-tls-3.pcapng :-

    Also, Have you tried using Google's minimal root CA set (<1 KB) for The minimal root CA set includes a primary and backup certificate.

  • Yes, the minimal set root CA is the one originally used, but as Yep said that is an ECC certificate. And yes, the original address 8883 with the minimal CA (ECC) is the one I was using. Also, I tried adding the nx_secure_tls_ecc_initialize() (when using the minimal cert, ECC) and tried also with the RSA cert version changing the broker accordingly ( 8883).

    Same error comes out.

    Also I tried to manually set the google brokers IP (pinging from my laptop just before compiling) and same happens.

    Is there some IP routing going on in the wrong way? Like, is Google trying to send back packets to my uBlox public addres but it can not find me?

    I don't use any DCHP ssp module, I think I don't need to do that.

    when you say "Also, Have you tried using Google's minimal root CA set (<1 KB) for The minimal root CA set includes a primary and backup certificate.", that means I need to load both certificates into netx secure? Because in python I just use the primary and all goes well as expected.

    Thanks for your help, is becoming very complex to use netx duo framework on IoT Core/ TLS.

    At the moment we are stuck and we need to make sure we can proceed the development on Synergy with this platform.

    Is there a working example using Syergy to interact (connect/publish/receive) with MQTT on IoT Core with these certificates (either minimal root certs or the regular version) through a mqtt tls using JWT?

  • I've noticed a strange thing. When synergy goes to ask ARP resolution, gets no answer.

    I think the whole IP chain down to the synergy stack is not correct/ routed correctly.

    Also. the MAC address I get when I query directly uBlox module is different from the one I see in the wireshark capture (ends with a different number).

    How can I setup the whole uBlox <-> PPP <-> IP in order to work?

    Strangely though, even in this weird configuration, I can get the time from SNTP and google IP from DNS, but I don't see it in the tcp/ip capture...

    Please find here my configuration, the IP I get assigned by my router is, and this is the Peer IPv4 on PPP interface I set up, but I'm not sure about the IP interface configuration.

    Intel core(my laptop) gets an answer, uBlox ARP request gets lost..

    Full extract of the log, you can see the full AT command sequence, MAC address of uBlox, Wifi station init and PPP mode init..

    [0000000331][hsp_wble.c:0418] - WBLE > +STARTUP

    [0000000331][hsp_wble.c:0458] - WBLE < ATE0

    [0000000331][hsp_wble.c:0418] - WBLE > ATE0

    [0000000332][hsp_wble.c:0418] - WBLE >

    [0000000332][hsp_wble.c:0418] - WBLE > OK

    [0000000342][hsp_wble.c:0458] - WBLE < AT+UWAPMACADDR

    [0000000342][hsp_wble.c:0418] - WBLE >

    [0000000342][hsp_wble.c:0418] - WBLE > +UWAPMACADDR:6009C3E05737

    [0000000342][hsp_wble.c:0418] - WBLE > OK

    [0000000352][hsp_wble.c:0458] - WBLE < AT+UWSC=0,0,1

    [0000000352][hsp_wble.c:0418] - WBLE >

    [0000000352][hsp_wble.c:0418] - WBLE > OK

    [0000000362][hsp_wble.c:0458] - WBLE < AT+UWSC=0,2,"----SSID----"

    [0000000362][hsp_wble.c:0418] - WBLE >

    [0000000362][hsp_wble.c:0418] - WBLE > OK

    [0000000372][hsp_wble.c:0458] - WBLE < AT+UWSC=0,5,2

    [0000000372][hsp_wble.c:0418] - WBLE >

    [0000000372][hsp_wble.c:0418] - WBLE > OK

    [0000000382][hsp_wble.c:0458] - WBLE < AT+UWSC=0,8,"----password----"

    [0000000382][hsp_wble.c:0418] - WBLE >

    [0000000382][hsp_wble.c:0418] - WBLE > OK

    [0000000392][hsp_wble.c:0458] - WBLE < AT+UWSC=0,100,2

    [0000000392][hsp_wble.c:0418] - WBLE >

    [0000000392][hsp_wble.c:0418] - WBLE > OK

    [0000000402][hsp_wble.c:0458] - WBLE < AT+UWSC=0,107,1

    [0000000402][hsp_wble.c:0418] - WBLE >

    [0000000402][hsp_wble.c:0418] - WBLE > OK

    [0000000412][hsp_wble.c:0458] - WBLE < AT+UWSC=0,301,1

    [0000000412][hsp_wble.c:0418] - WBLE >

    [0000000412][hsp_wble.c:0418] - WBLE > OK

    [0000000422][hsp_wble.c:0660] - Wifi init SUCCESS
    [0000000422][hsp_wble.c:0500] - WBLE < AT+UWSCA=0,3

    [0000000423][hsp_wble.c:0418] - WBLE >

    [0000000423][hsp_wble.c:0418] - WBLE > OK

    [0000000652][hsp_wble.c:0418] - WBLE >

    [0000000652][hsp_wble.c:0418] - WBLE > +UUWLE:0,FAAC838145DA,1

    [0000000652][hsp_wble.c:0679] - Wifi attach SUCCESS
    [0000000652][hsp_wble.c:0545] - WBLE < AT+UPPPC=101,

    [0000000653][hsp_wble.c:0418] - WBLE >

    [0000000653][hsp_wble.c:0418] - WBLE > OK

    [0000000663][hsp_wble.c:0545] - WBLE < AT+UPPPC=102,

    [0000000663][hsp_wble.c:0418] - WBLE >

    [0000000663][hsp_wble.c:0418] - WBLE > OK

    [0000000673][hsp_wble.c:0545] - WBLE < AT+UPPPC=104,

    [0000000673][hsp_wble.c:0418] - WBLE >

    [0000000673][hsp_wble.c:0418] - WBLE > OK

    [0000000683][hsp_wble.c:0545] - WBLE < AT+UPPPC=105,

    [0000000683][hsp_wble.c:0418] - WBLE >

    [0000000683][hsp_wble.c:0418] - WBLE > OK

    [0000000693][hsp_wble.c:0545] - WBLE < AT+UPPPC=107,1

    [0000000693][hsp_wble.c:0418] - WBLE >

    [0000000693][hsp_wble.c:0418] - WBLE > OK

    [0000000825][hsp_wble.c:0418] - WBLE >

    [0000000825][hsp_wble.c:0418] - WBLE > +UUNU:0

    [0000000835][hsp_wble.c:0545] - WBLE < ATO3

    [0000000835][hsp_wble.c:0418] - WBLE >

    [0000000835][hsp_wble.c:0418] - WBLE > OK

    [0000000845][hsp_wble.c:0697] - PPP init SUCCESS
    [0000000845][hsp_wble.c:0699] - Wifi station link UP and PPP mode active
    [0000001045][hsp_wble.c:0714] - Initialised NX common
    [0000001095][hsp_wble.c:0832] - WBLE module init_connection
    [0000001099][hsp_wble.c:0865] - Initialised IP
    [0000001099][hsp_wble.c:0891] - Initialised PPP
    [0000001099][new_thread0_entry.c:0135] - Interface index 0, IP: 2886994442, NMASK: 4294967295, MTU: 1500, MSW: 0, LSW: 0
    [0000001099][new_thread0_entry.c:0144] - SNTP init and get time
    [0000002154][new_thread0_entry.c:0149] - TIME/DATE >> Sep 4, 2021 16:19:16.796 UTC <<
    [0000002154][new_thread0_entry.c:0150] - EPOCH >> 1630772356 <<
    [0000002154][new_thread0_entry.c:0158] - DNS get MQTT broker IPv4 - attempt #1
    [0000002162][new_thread0_entry.c:0186] - is at
    [0000002174][new_thread0_entry.c:0222] - JWT token created successfully eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJoNG5oNDQtMSIsImV4cCI6MTYzMDc3OTU1NiwiaWF0IjoxNjMwNzcyMzU2fQ.rRBufIU9bdx0gQzckMW-QfIRqzfLruCst27_mIPviDyYkZbIgG10RM3xcKv-n8XY6fKgZrsa56MuoBdg0FX61mi3e9Qbh7f9lD53YDZgienf38qCmArXBgD3O2zwJyt1dnDRqc87fhDdBIsvoYfqnI4EPZyUUElu8UVAX8gVQcorpciHbgoWJjDP1qDK-P7K_ViLNcirj8oXQ7GOabFoEAcLJ3Nnw21FRw19w2ddbvR-h4HuIUC4QYop2CljGMMyRKUeRzGaxD9vQ7x3r56xy_fHPKeRyugnp6BvZl8KVy0yN9dxdLutbCn9p4rJn6VdtthY4YPO2q7DY7xiE1-teg

    [0000002175][new_thread0_entry.c:0237] - Secure TLS session created!
    [0000002175][new_thread0_entry.c:0244] - Secure_tls_ecc_initialize returned : 0x0
    [0000002175][new_thread0_entry.c:0263] - MQTT Client login set successful
    [0000002275][new_thread0_entry.c:0278] - nx_ip_status_check got status 0x0 and actual_status 0x4
    [0000002326][new_thread0_entry.c:0291] - MQTT Client secure connect : ERROR 0x10005
    [0000002326][new_thread0_entry.c:0307] - Interface index 0, IP: 2886994442, NMASK: 4294967295, MTU: 1500, MSW: 0, LSW: 0

  • I think your development network setup you are using to do Wireshark captures is not actually allowing you to see most of the packets going to and from the Synergy device.

    Most networking componets these days are switches that do packet routing, so only send the network packets out on the network interface they need to go out on, rather than a dumb hub that just broadcasts all packets on all network interfaces.

    For this reason I have to use a Network switch that does port mirroring, to send all the network traffic between the Synergy board and the Internet router, to my development PC, to allow me to Capture the packets with Wireshark. I would expect to see something like (this is from an MQTT connection io.adafruit but it shows the DHCP IP allocation, DNS lookup, NTP and TLS handshake for an ethernet connection) : -

  • Hi Jeremy, I'm using SSP 1.7.8, S5D9, and I got all my pieces of code almost from that project you're talking about.

    I though the same about packet capture, indeed I can not see (ARP replies, and other packets) data directed other than the interface I'm running Wireshark from (my laptop). I read about monitor mode would do the trick (instead of promiscuous that is the one I'm using) but not all wifi interfaces allows that.

    Do you think I need to check further the IP assignment, and maybe set up manually the uBlox MAC address into the msw, lsw of my ip_interface? 

    But since I can resolve DNS, get time from the SNTP server and perform HTTP posts, I think the whole IP stack is actually working. Not sure.

    Is there any simple way to test if I can make a TLS connection to the google server, using one of the certs (reduced ECC or standard RSA)?

    I need to make some progress on this, because If I can't use the NetX Duo stack to connect to MQTT/TLS of IoT Core I need to change my direction of development.

    How could I set up something like your switch to mirror the packets from the Wifi router to my laptop? 

    Thanks a lot


  • Hi S,

    For RSA, you use the Baltimore Cybertrust Root (roots[] = ). Could you use the GTS CA 101 (its also given in the project shared by Jeremy).

    For port mirroring, I have use TL-SG105E | 5-Port Gigabit Easy Smart Switch | TP-Link Singapore with port mirroring feature.

  • Hi Yep, 

    Yes, I loaded the certificate from that project (gcloud_rootCA.pem) and it worked, also managed to publish some data on our GCP device.

    The problem still remains though, as we need a long term certificate (this one expires in december) and a reduce size one because of the data that will be used at every connection. 

    > openssl x509 -enddate -noout -in .\gcloud_rootCA.pem
    notAfter=Dec 15 00:00:42 2021 GMT

    Could you please explain/investigate why we can not use the ECC (reduced) google certificate?

    It is of vital importance that we can make use of the ECC suite and this certificate to create our TLS connection.

    Are you able to try yourself with the ECC reduced one and see if you can connect (using the same projects you developed)?

    Thanks for you help, at least now we got something working!

  • Hi Yep, 

    Yes, I loaded the certificate from that project (gcloud_rootCA.pem) and it worked, also managed to publish some data on our GCP device.

    The problem still remains though, as we need a long term certificate (this one expires in december) and a reduce size one because of the data that will be used at every connection. 

    > openssl x509 -enddate -noout -in .\gcloud_rootCA.pem
    notAfter=Dec 15 00:00:42 2021 GMT

    Could you please explain/investigate why we can not use the ECC (reduced) google certificate?

    It is of vital importance that we can make use of the ECC suite and this certificate to create our TLS connection.

    Are you able to try yourself with the ECC reduced one and see if you can connect (using the same projects you developed)?

    Thanks for you help, at least now we got something working!

  • Hi Rusty,

    When using, GTS LTSR (primary) you need to set r_sce_0 SCE COMMON Driver on r_sce to CRYPTO_WORD_ENDIAN_LITTLE.  While verifying an ECC cert, I found it wont pass in ENDIAN_BIG. For RSA certs, both BIG/LITTLE are OK. I dont quite figure out where problem when converting endian.

    The (backup) can't be used. It has a ECC 384bit keys. So any cert ecdsa-with-sha384 is not possible.

    I ask to use RSA, cause your JWT. uses { "alg" : RS256 ...} You may have to change your device cert to ECC as well.

    Also strongly recommend you switch to SSP 2.0.0.. 

  • For big endian, before verification, swapping for input parameters is done, in-situ. If the certificate is in ROM (is a const), bytes cannot be swapped, so operations will fail. You cert need to be in RAM. 

    Sorry my Google account is in a mess, I couldn't set it up to do a proper test.. 

  • I can get the AE CLoud2 version of this project :-

    to connect to using the minimal root CA set's primary certificate, if I add the TLS SNI :- 

        static NX_SECURE_X509_DNS_NAME dns_name;

        status = nx_secure_x509_dns_name_initialize(&dns_name, (UCHAR *)p_cfg->p_endpoint, (USHORT)(strlen(p_cfg->p_endpoint)));
        if (NX_SUCCESS != status)
            NX_MQTT_log_msg(p_ctrl, "Unable to initialise SNI DNS name, ret=0x%x\r\n",status);
            return status;

        status = nx_secure_tls_session_sni_extension_set(&(p_ctrl->p_secure_client->nxd_mqtt_tls_session), &dns_name);
        if (NX_SUCCESS != status)
            NX_MQTT_log_msg(p_ctrl, "Unable to add SNI, ret=0x%x\r\n",status);
            return status;

    The requirements to connect to an LTS MQTT domain are here :-

  • Thanks Jeremy, I've updated my e2Studio and SSP to 2.0.0 and trying again now to tls using the primary certificate (ECC) to connect to

    Couple of questions if you don't mind:

    1. Do I still need to initialise ECC suite with nx_secure_tls_ecc_initialize(&g_mqtt_client0.nxd_mqtt_tls_session,
      nx_crypto_ecc_curves_synergys7) ?
    2. Can I use the converted certificate from PEM->DER->Byte array as I have it at the moment or do I need to convert from the .crt?
    3. At which point do I need to add and set the SNI Dns? In the NX_MQTT_tls_setup() or just before/after the nx_secure_tls_session_create()?
    4. I'm getting the following error now with the new SSP2.0.0, have the function name/arguments changed? 

    ./synergy/ssp/src/framework/sf_el_nx_crypto/nx_crypto_synergys7_ciphersuites.o:(.data.crypto_method_aes_128_gcm_16_sce+0x1c): undefined reference to `_nx_crypto_method_aes_gcm_sce_operation'

    collect2.exe: error: ld returned 1 exit status

    Thanks a lot,


  • I had success using SSP 1.7.8 with the google cloud example project. I have migrated the project to SSP 2.0.0 however I have been unsuccessful in connecting using the LTS certificates.

    In answer to your questions :-

    1) In the application code, the API nx_secure_tls_ecc_initialize() must be called
    after TLS session is created  ( nx_secure_tls_session_create() ). This API notifies the TLS session of the type of
    curves used in the system. During the TLS handshake phase, if ECC
    algorithm is selected, the client and server exchange ECC curve-related parameters, so ECC can be used for the session.

    2) If the certificate is in the correct format in a byte array then it is fine. I have used a similar process before (converted certificate from PEM->DER->Byte array ).

    3) Again, for the SNI, the API to add the SNI information needs to be called after the TLS session is created (nx_secure_tls_session_create()  )

    4) For the compilation issue with SSP 2.0.0, when you enable ECC Cipher, you also need to enable AEAD Cipher for the compilation to be successful. As I said, I have yet to successfully connect to Google with SSP 2.0.0, using the ECC LTS certificates, as there have been some additions to NetX secure, so the configuration is slightly different.

  • Thanks Jeremy, copied all your suggestions:

    • tls_ecc_initialise called after tls_session_create()
    • SNI called after tls_ecc_initialise() (and so after tls_session_create()
    • Enabled AEAD (both options) and now compiles correctly (is that a new feature for SSP2.0.0? This error didn't appear in SSP1.7.8

    As you stated though, now I fail again on the mqtt_secure_connect. Is quite frustrating that this function doesn't return a more precise error, is a generic NX_NOT_CONNECTED : 0x10005.

    Do I need to rollback to 1.7.8 or is there a workaround you guys are working on for the new SSP?

    I'm also trying to bring up another laptop with a Wifi card that should allow monitoring mode, so maybe I'll be able to see what's going on during the handshake.

    Thanks again,


  • I am currently trying to work out why SSP 2.0.0 will not connect to Google using the ECC certificates.

  • I have done a bit more digging as to why the SSP 2.0.0. based project won't connect to the Google cloud MQTT Long term domain, and it appears to be an issue with NetX DUO TLS version in SSP 2.0.0, and the integration of the SCE Crypto HW integration that is used.. There have been alot of changes in NetX DUO TLS between SSP 1.7.8 and SSP 2.0.0. In the Wireshark trace when trying to connect I see an Alert :-

    and the Alert is labelled as an internal error, and the Alert came from the Synergy device.

    After a bit of debugging on the Synergy side, I see the error is coming from a Crypto operation not being successful (the Crypto operation is using the SCE Crypto HW) :-

    It is because of this Crypto operation error, that NetX DUO TLS in SSP 2.0.0 signals the Alert.

    This issue is not going to be fixed quickly, I have raised the issue with the SSP development team, for them to look at.

    As I said, with SSP 1.7.8 I could successfully connect to the Google cloud MQTT Long term domain.

  • I am still stuck RSA certs, SSP1.7.8/2.0.0.  The server FIN after receiving first Application data (41931).

    Double check settings project-id etc, No luck.

  • This my edited version of thr Google Cloud project for the AE-Cloud2 board, for SSP 1.7.8 that I used.

    I used an ethernet connection.