MQTT TLS on Google cloud - invalid JWT generated

Hi S,

Have you try modifying  the Base64urlEncoder() in Base64Decode.c to strip the padding, "="? Would it pass checking jwt.io?

The hash/signature would definitely be different without '=', As the Header/payload is available, the receiver could have use it to verify the signature,   

Hi,

Yes I've tried but still doesn't pass the jwt.io check. Is there a valid Base64url encoder library that actually works with Google IoT Core?

Also, it has something to do with the pkcs1 padding in the jwt_create() ?

Thanks

Hi,

Could you get a wireshark logs? Hope to understand why server terminates this tcp connection.

Just to double check, is the size of your certificate correct?  You mention you convert this certificate into byte array. Could you do a memory dump of this and check if it's correct?

Hi Yep, 

Yes I've done a capture of my wifi network from my PC while application running, the DHCP address assigned to my uBlox NINA is 192.168.1.12, something is going on but I'm not sure at what to look at.

Following are my google certificate in PEM format (that I use from Python with no problems) and the corresponding byte array used in my FW. Size seems the same. 

CA_cert.zip

Should I use a different certificate or check if is loaded correctly into netx secure?

Can I send the capture file in private to you? 

Also, would it be possible to create a TLS session project (client) and use the same certificate just to see if it's loaded and checked correctly? I tried also an openssl s_client -connect with this certificate and broker endpoint and status returns 0 from openssl (verified, OK).

 Thanks,

S

edit: Looking here:AE-CLOUD2. Google cloud. Could not create tls session. - Synergy - Forum - Renesas Synergy Platform - RenesasRulz

I have confirmed that I can connect to Azure and Amazon using the 2.1.0 SECT binary, but I cannot connect to Google. There has been a change in the Google certificate chain, that requires a different intermediate CA certificate to be used for Google. This would require a new SECT binary firmware.

This answer maybe is a clue? Not sure how I load a certificate and an intermediate certificate, as a chain?

Hi Rusty,

The CA_cert_der[] is an ECC cert.  Can you check if ECC is added to nx_crypto_tls_ciphers_synergys7. 

Could you find/use RSA cert instead? You have install a root/intermediate cert as trusted cert.

Other intermediate certs in the chain will be downloaded as needed to verify the server cert.

Hi,

Yes all the suites are enabled, including ECC. 

Do I need to specify that is an ECC cert in tls init?

That certificate has been given by google, not sure If/how I can find an RSA version of it

Do you call nx_secure_tls_ecc_initialize() in the mqtt tls setup function?

extern const USHORT     nx_crypto_ecc_supported_groups_synergys7[];
extern const USHORT     nx_crypto_ecc_supported_groups_synergys7_size;
extern const NX_CRYPTO_METHOD *     nx_crypto_ecc_curves_synergys7[];

 nx_secure_tls_ecc_initialize(p_tls_session,
                                          nx_crypto_ecc_supported_groups_synergys7,
                                          nx_crypto_ecc_supported_groups_synergys7_size,
                                          nx_crypto_ecc_curves_synergys7);

in the mqtt tls setup function?

Hi Jeremy,

No, I call x509 for both CA_cert and device cert. should I substitute the ca_cert one?

Or add nx_secure_tls_ecc_initialize() at the beginning/end of the mqtt_tls_setup?

Here's my mqtt tls setup function:

static UINT NX_MQTT_tls_setup(NXD_MQTT_CLIENT *p_client, NX_SECURE_TLS_SESSION *p_tls_session,
NX_SECURE_X509_CERT *p_cert, NX_SECURE_X509_CERT *p_trusted_cert)
{
UINT status = NX_SUCCESS;
UINT index = 0;

SSP_PARAMETER_NOT_USED (p_client);

/* Need to allocate space for the certificate coming in from the remote host. */
memset(&remote_cert_buffer, 0, sizeof(remote_cert_buffer));
memset(&remote_certificate, 0, sizeof(remote_certificate));

status = nx_secure_tls_remote_certificate_allocate(p_tls_session, &remote_certificate,
remote_cert_buffer, sizeof(remote_cert_buffer));
if (NX_SUCCESS != status)
{
HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unabe to allocate memory for the certificate buffer %d", status);
return status;
}

memset(remote_int_ca_cert_buffer, 0, sizeof(remote_int_ca_cert_buffer));
memset(remote_intermediate_ca, 0, sizeof(remote_intermediate_ca));
for (index = 0; index < ARRAY_SIZE(remote_intermediate_ca); index++)
{
status = nx_secure_tls_remote_certificate_allocate(p_tls_session, &(remote_intermediate_ca[index]),
&(remote_int_ca_cert_buffer[index * CERTIFICATE_BUFFER_SIZE]),
CERTIFICATE_BUFFER_SIZE);
if (NX_SUCCESS != status)
{
HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to allocate memory for interemediate CA certificate %d", status);
return status;
}
}

/* Add a CA Certificate to our trusted store for verifying incoming server certificates. */
status = nx_secure_x509_certificate_initialize(p_trusted_cert,
(UCHAR *) CA_cert_der, (USHORT) CA_CERT_DER_LEN,
NX_NULL, 0, NULL, 0,
NX_SECURE_X509_KEY_TYPE_RSA_PKCS1_DER);
if (NX_SUCCESS != status)
{
HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to initialize CA certificate %d", status);
return status;
}

status = nx_secure_tls_trusted_certificate_add(p_tls_session, p_trusted_cert);
if (NX_SUCCESS != status)
{
HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to add CA certificate to trusted store, %d", status);
return status;
}

if (dev_cert != NULL)
{
/* Add the local certificate for client authentication. */
status = nx_secure_x509_certificate_initialize(p_cert, (UCHAR *) dev_cert, (USHORT) DEV_CERT_LEN,
NX_NULL, 0, rsa_private_pkcs1, RSA_PRIVATE_LEN,
NX_SECURE_X509_KEY_TYPE_RSA_PKCS1_DER);
if (NX_SUCCESS != status)
{
HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to initialize device certificate, %d",status);
return status;
}
status = nx_secure_tls_local_certificate_add(p_tls_session, p_cert);
if (NX_SUCCESS != status)
{
HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to add device certificate to trusted store, %d",status);
return status;
}
}

/* Add a timestamp function for time checking and timestamps in the TLS handshake. */
nx_secure_tls_session_time_function_set(p_tls_session, get_time);

return status;
}

Thanks a lot

Do you call nx_secure_tls_ecc_initialize() anywhere? Try nx_secure_tls_ecc_initialize() adding after you call nx_secure_tls_session_create();

I've called secure_tls_ecc_initialize after session_create(), it returns 0 (I guess is NX_SUCCESS), but still I can not connect with the same 0x10005 error.

I also tried to load the google certificate in RSA format (and changed broker to "mqtt.googleapis.com" as advised by IoT Core cloud.google.com/.../mqtt-bridge), but still I get the same error from mqtt connect. Also tried to use port 8883 and 443.

www.filemail.com/.../cyvrwfffjslgbzp

Have you taken a Wireshark trace of the network activity to see what is going on in the communication between the Synergy board and the Google servers?

Yes, I did, 192.168.1.12 is my uBlox NINA Wifi DHCP address. Asus is our wifi router, external activity is captured from the internal wifi network but I think you can extract the big picture.

Please find it here: www.filemail.com/.../mojofybpwdtrpuk

I did also 2 additional captures with just my laptop (wireshark running) and uBlox connected to my WiFi hotspost.

One using ECC certificate and the nx_secure_tls_ecc_initialize() (with the proper endpoint for ECC cert) and the RSA cert version.

www.filemail.com/.../heenqzenkxqxpbx

Interestingly, in these two last captures, I can't see the DNS request, even though I get it and resolve the IP of google cloud endpoint correctly from Synergy, also SNTP time query is not showing, and again I get the time correctly from the SNTP server.

I think something is going on when resolving uBlox external IP, gateway IP, but again, I can perform SNTP queries, DNS queries, and HTTP Post successfully with the same stack.

Yes, I did, 192.168.1.12 is my uBlox NINA Wifi DHCP address. Asus is our wifi router, external activity is captured from the internal wifi network but I think you can extract the big picture.

Please find it here: www.filemail.com/.../mojofybpwdtrpuk

I did also 2 additional captures with just my laptop (wireshark running) and uBlox connected to my WiFi hotspost.

One using ECC certificate and the nx_secure_tls_ecc_initialize() (with the proper endpoint for ECC cert) and the RSA cert version.

www.filemail.com/.../heenqzenkxqxpbx

Interestingly, in these two last captures, I can't see the DNS request, even though I get it and resolve the IP of google cloud endpoint correctly from Synergy, also SNTP time query is not showing, and again I get the time correctly from the SNTP server.

I think something is going on when resolving uBlox external IP, gateway IP, but again, I can perform SNTP queries, DNS queries, and HTTP Post successfully with the same stack.

I don't see the DNS query of the Synergy device looking up the address of the Google mqtt server:-

"mqtt.2030.ltsapis.goog"

or

"mqtt.googleapis.com"

        /* Look up IPv4 addresses to record  IPv4 addresses in record_buffer and return the IPv4 address count.  */
        status = nx_dns_ipv4_address_by_name_get(&g_dns0, (uint8_t *)GCP_IO_SERVER, &gcp_io_ip_address[0], sizeof(gcp_io_ip_address), &record_count, 500/*NX_IP_PERIODIC_RATE*/);

in any of the the wireshark logs. The only DNS queries in any of the logs you posted are in the file "nina-mqtt-tls-3.pcapng :-

Also, Have you tried using Google's minimal root CA set (<1 KB) for mqtt.2030.ltsapis.goog. The minimal root CA set includes a primary and backup certificate.

cloud.google.com/.../mqtt-bridge

Yes, the minimal set root CA is the one originally used, but as Yep said that is an ECC certificate. And yes, the original address mqtt.2030.ltsapis.goog 8883 with the minimal CA (ECC) is the one I was using. Also, I tried adding the nx_secure_tls_ecc_initialize() (when using the minimal cert, ECC) and tried also with the RSA cert version changing the broker accordingly (mqtt.googleapis.com 8883).

Same error comes out.

Also I tried to manually set the google brokers IP (pinging from my laptop just before compiling) and same happens.

Is there some IP routing going on in the wrong way? Like, is Google trying to send back packets to my uBlox public addres but it can not find me?

I don't use any DCHP ssp module, I think I don't need to do that.

when you say "Also, Have you tried using Google's minimal root CA set (<1 KB) for mqtt.2030.ltsapis.goog. The minimal root CA set includes a primary and backup certificate.", that means I need to load both certificates into netx secure? Because in python I just use the primary and all goes well as expected.

Thanks for your help, is becoming very complex to use netx duo framework on IoT Core/ TLS.

At the moment we are stuck and we need to make sure we can proceed the development on Synergy with this platform.

Is there a working example using Syergy to interact (connect/publish/receive) with MQTT on IoT Core with these certificates (either minimal root certs or the regular version) through a mqtt tls using JWT?

I've noticed a strange thing. When synergy goes to ask ARP resolution, gets no answer.

I think the whole IP chain down to the synergy stack is not correct/ routed correctly.

Also. the MAC address I get when I query directly uBlox module is different from the one I see in the wireshark capture (ends with a different number).

How can I setup the whole uBlox <-> PPP <-> IP in order to work?

Strangely though, even in this weird configuration, I can get the time from SNTP and google IP from DNS, but I don't see it in the tcp/ip capture...

Please find here my configuration, the IP I get assigned by my router is 172.20.10.4, and this is the Peer IPv4 on PPP interface I set up, but I'm not sure about the IP interface configuration.

Intel core(my laptop) gets an answer, uBlox ARP request gets lost..

Full extract of the log, you can see the full AT command sequence, MAC address of uBlox, Wifi station init and PPP mode init..

[0000000331][hsp_wble.c:0418] - WBLE > +STARTUP

[0000000331][hsp_wble.c:0458] - WBLE < ATE0

[0000000331][hsp_wble.c:0418] - WBLE > ATE0

[0000000332][hsp_wble.c:0418] - WBLE >

[0000000332][hsp_wble.c:0418] - WBLE > OK

[0000000342][hsp_wble.c:0458] - WBLE < AT+UWAPMACADDR

[0000000342][hsp_wble.c:0418] - WBLE >

[0000000342][hsp_wble.c:0418] - WBLE > +UWAPMACADDR:6009C3E05737

[0000000342][hsp_wble.c:0418] - WBLE > OK

[0000000352][hsp_wble.c:0458] - WBLE < AT+UWSC=0,0,1

[0000000352][hsp_wble.c:0418] - WBLE >

[0000000352][hsp_wble.c:0418] - WBLE > OK

[0000000362][hsp_wble.c:0458] - WBLE < AT+UWSC=0,2,"----SSID----"

[0000000362][hsp_wble.c:0418] - WBLE >

[0000000362][hsp_wble.c:0418] - WBLE > OK

[0000000372][hsp_wble.c:0458] - WBLE < AT+UWSC=0,5,2

[0000000372][hsp_wble.c:0418] - WBLE >

[0000000372][hsp_wble.c:0418] - WBLE > OK

[0000000382][hsp_wble.c:0458] - WBLE < AT+UWSC=0,8,"----password----"

[0000000382][hsp_wble.c:0418] - WBLE >

[0000000382][hsp_wble.c:0418] - WBLE > OK

[0000000392][hsp_wble.c:0458] - WBLE < AT+UWSC=0,100,2

[0000000392][hsp_wble.c:0418] - WBLE >

[0000000392][hsp_wble.c:0418] - WBLE > OK

[0000000402][hsp_wble.c:0458] - WBLE < AT+UWSC=0,107,1

[0000000402][hsp_wble.c:0418] - WBLE >

[0000000402][hsp_wble.c:0418] - WBLE > OK

[0000000412][hsp_wble.c:0458] - WBLE < AT+UWSC=0,301,1

[0000000412][hsp_wble.c:0418] - WBLE >

[0000000412][hsp_wble.c:0418] - WBLE > OK

[0000000422][hsp_wble.c:0660] - Wifi init SUCCESS
[0000000422][hsp_wble.c:0500] - WBLE < AT+UWSCA=0,3

[0000000423][hsp_wble.c:0418] - WBLE >

[0000000423][hsp_wble.c:0418] - WBLE > OK

[0000000652][hsp_wble.c:0418] - WBLE >

[0000000652][hsp_wble.c:0418] - WBLE > +UUWLE:0,FAAC838145DA,1

[0000000652][hsp_wble.c:0679] - Wifi attach SUCCESS
[0000000652][hsp_wble.c:0545] - WBLE < AT+UPPPC=101,172.20.10.4

[0000000653][hsp_wble.c:0418] - WBLE >

[0000000653][hsp_wble.c:0418] - WBLE > OK

[0000000663][hsp_wble.c:0545] - WBLE < AT+UPPPC=102,255.255.255.0

[0000000663][hsp_wble.c:0418] - WBLE >

[0000000663][hsp_wble.c:0418] - WBLE > OK

[0000000673][hsp_wble.c:0545] - WBLE < AT+UPPPC=104,172.20.10.1

[0000000673][hsp_wble.c:0418] - WBLE >

[0000000673][hsp_wble.c:0418] - WBLE > OK

[0000000683][hsp_wble.c:0545] - WBLE < AT+UPPPC=105,172.20.10.1

[0000000683][hsp_wble.c:0418] - WBLE >

[0000000683][hsp_wble.c:0418] - WBLE > OK

[0000000693][hsp_wble.c:0545] - WBLE < AT+UPPPC=107,1

[0000000693][hsp_wble.c:0418] - WBLE >

[0000000693][hsp_wble.c:0418] - WBLE > OK

[0000000825][hsp_wble.c:0418] - WBLE >

[0000000825][hsp_wble.c:0418] - WBLE > +UUNU:0

[0000000835][hsp_wble.c:0545] - WBLE < ATO3

[0000000835][hsp_wble.c:0418] - WBLE >

[0000000835][hsp_wble.c:0418] - WBLE > OK

[0000000845][hsp_wble.c:0697] - PPP init SUCCESS
[0000000845][hsp_wble.c:0699] - Wifi station link UP and PPP mode active
[0000001045][hsp_wble.c:0714] - Initialised NX common
[0000001095][hsp_wble.c:0832] - WBLE module init_connection
[0000001099][hsp_wble.c:0865] - Initialised IP
[0000001099][hsp_wble.c:0891] - Initialised PPP
[0000001099][new_thread0_entry.c:0135] - Interface index 0, IP: 2886994442, NMASK: 4294967295, MTU: 1500, MSW: 0, LSW: 0
[0000001099][new_thread0_entry.c:0144] - SNTP init and get time
[0000002154][new_thread0_entry.c:0149] - TIME/DATE >> Sep 4, 2021 16:19:16.796 UTC <<
[0000002154][new_thread0_entry.c:0150] - EPOCH >> 1630772356 <<
[0000002154][new_thread0_entry.c:0158] - DNS get MQTT broker IPv4 - attempt #1
[0000002162][new_thread0_entry.c:0186] - mqtt.2030.ltsapis.goog is at 74.125.133.206:8883
[0000002174][new_thread0_entry.c:0222] - JWT token created successfully eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJoNG5oNDQtMSIsImV4cCI6MTYzMDc3OTU1NiwiaWF0IjoxNjMwNzcyMzU2fQ.rRBufIU9bdx0gQzckMW-QfIRqzfLruCst27_mIPviDyYkZbIgG10RM3xcKv-n8XY6fKgZrsa56MuoBdg0FX61mi3e9Qbh7f9lD53YDZgienf38qCmArXBgD3O2zwJyt1dnDRqc87fhDdBIsvoYfqnI4EPZyUUElu8UVAX8gVQcorpciHbgoWJjDP1qDK-P7K_ViLNcirj8oXQ7GOabFoEAcLJ3Nnw21FRw19w2ddbvR-h4HuIUC4QYop2CljGMMyRKUeRzGaxD9vQ7x3r56xy_fHPKeRyugnp6BvZl8KVy0yN9dxdLutbCn9p4rJn6VdtthY4YPO2q7DY7xiE1-teg

[0000002175][new_thread0_entry.c:0237] - Secure TLS session created!
[0000002175][new_thread0_entry.c:0244] - Secure_tls_ecc_initialize returned : 0x0
[0000002175][new_thread0_entry.c:0263] - MQTT Client login set successful
[0000002275][new_thread0_entry.c:0278] - nx_ip_status_check got status 0x0 and actual_status 0x4
[0000002326][new_thread0_entry.c:0291] - MQTT Client secure connect : ERROR 0x10005
[0000002326][new_thread0_entry.c:0307] - Interface index 0, IP: 2886994442, NMASK: 4294967295, MTU: 1500, MSW: 0, LSW: 0

Which version of the SSP are you using? There is an example of connecting to Google Cloud for Synergy, but it has not been updated since Nov 2019.

https://www.renesas.com/document/apn/synergy-mqtttls-google-cloud-connectivity-solution-application-project?language=en&r=1054626

https://www.renesas.com/eu/en/document/scd/synergy-mqtttls-google-cloud-connectivity-solution-application-project-r11an0335eu0105-rev105-sample?language=en&r=1054626

I think your development network setup you are using to do Wireshark captures is not actually allowing you to see most of the packets going to and from the Synergy device.

Most networking componets these days are switches that do packet routing, so only send the network packets out on the network interface they need to go out on, rather than a dumb hub that just broadcasts all packets on all network interfaces.

For this reason I have to use a Network switch that does port mirroring, to send all the network traffic between the Synergy board and the Internet router, to my development PC, to allow me to Capture the packets with Wireshark. I would expect to see something like (this is from an MQTT connection io.adafruit but it shows the DHCP IP allocation, DNS lookup, NTP and TLS handshake for an ethernet connection) : -

Hi Jeremy, I'm using SSP 1.7.8, S5D9, and I got all my pieces of code almost from that project you're talking about.

I though the same about packet capture, indeed I can not see (ARP replies, and other packets) data directed other than the interface I'm running Wireshark from (my laptop). I read about monitor mode would do the trick (instead of promiscuous that is the one I'm using) but not all wifi interfaces allows that.

Do you think I need to check further the IP assignment, and maybe set up manually the uBlox MAC address into the msw, lsw of my ip_interface? 

But since I can resolve DNS, get time from the SNTP server and perform HTTP posts, I think the whole IP stack is actually working. Not sure.

Is there any simple way to test if I can make a TLS connection to the google server, using one of the certs (reduced ECC or standard RSA)?

I need to make some progress on this, because If I can't use the NetX Duo stack to connect to MQTT/TLS of IoT Core I need to change my direction of development.

How could I set up something like your switch to mirror the packets from the Wifi router to my laptop? 

Thanks a lot

S

Hi S,

For RSA, you use the Baltimore Cybertrust Root (roots[] = ). Could you use the GTS CA 101 (its also given in the project shared by Jeremy).

For port mirroring, I have use TL-SG105E | 5-Port Gigabit Easy Smart Switch | TP-Link Singapore with port mirroring feature.

Hi Yep, 

Yes, I loaded the certificate from that project (gcloud_rootCA.pem) and it worked, also managed to publish some data on our GCP device.

The problem still remains though, as we need a long term certificate (this one expires in december) and a reduce size one because of the data that will be used at every connection. 

> openssl x509 -enddate -noout -in .\gcloud_rootCA.pem
notAfter=Dec 15 00:00:42 2021 GMT

Could you please explain/investigate why we can not use the ECC (reduced) google certificate?

It is of vital importance that we can make use of the ECC suite and this certificate to create our TLS connection.

Are you able to try yourself with the ECC reduced one and see if you can connect (using the same projects you developed)?

Thanks for you help, at least now we got something working!

Hi Rusty,

When using, GTS LTSR (primary) you need to set r_sce_0 SCE COMMON Driver on r_sce to CRYPTO_WORD_ENDIAN_LITTLE.  While verifying an ECC cert, I found it wont pass in ENDIAN_BIG. For RSA certs, both BIG/LITTLE are OK. I dont quite figure out where problem when converting endian.

The (backup) can't be used. It has a ECC 384bit keys. So any cert ecdsa-with-sha384 is not possible.

I ask to use RSA, cause your JWT. uses { "alg" : RS256 ...} You may have to change your device cert to ECC as well.

Also strongly recommend you switch to SSP 2.0.0.. 

For big endian, before verification, swapping for input parameters is done, in-situ. If the certificate is in ROM (is a const), bytes cannot be swapped, so operations will fail. You cert need to be in RAM. 

Sorry my Google account is in a mess, I couldn't set it up to do a proper test..