• State Not Answered
  • Replies 6 replies
  • Subscribers 893 subscribers
  • Views 3060 views
  • Users 0 members are here
Options
Related

NetX Duo HTTPS Client receive file larger than 4094 bytes

Ghost

Hi,

I have a project that I am working on that makes https tls connection to aws api with asp.net core 3.1, the api requires 3 header keys added to the get request and if those match our security requirements then the connection is made and I can run the following code to get the response from the server.

if(status == NX_SUCCESS){
bytes_copied = 0;
j = 0;

do
{
statusRX = nx_web_http_client_response_body_get(&g_web_http_client0, &receive_packet, HTTP_CLIENT_WAIT_OPTION);

for(i = 0; i < RECEIVE_BUFFER_SIZE; i++){
g_receive_buffer[i] = 0;
}

if ((NX_SUCCESS == statusRX) || (NX_WEB_HTTP_GET_DONE == statusRX))
{
bytes_in_packet = receive_packet->nx_packet_length;
bytes_offset = 0;
do{
/* Process response packets */
status = nx_packet_data_extract_offset(receive_packet, bytes_offset, g_receive_buffer, RECEIVE_BUFFER_SIZE, &bytes_copied);
// status = nx_packet_data_retrieve(receive_packet, g_receive_buffer, &bytes_copied);

// Process the file to QSPI
// processRXData((char *)g_receive_buffer);

// Reset all variables and move markers accordingly
bytes_in_packet -= bytes_copied;
bytes_offset += bytes_copied;

for(i = 0; i < RECEIVE_BUFFER_SIZE; i++){
g_receive_buffer[i] = 0;
}
bytes_copied = 0;
}while(bytes_in_packet > 0);

/* Regardless if an error occurs extracting the data, release the packet. We are done with it. */
nx_packet_release (receive_packet);
}else{
break;
}
j++;
}while(statusRX != NX_WEB_HTTP_GET_DONE);
}

now all works perfect with connection and I do get data from the server, with one HUGE issue, if the response data has more than 4094 bytes in it then I don't get the response from the server with this call

statusRX = nx_web_http_client_response_body_get(&g_web_http_client0, &receive_packet, HTTP_CLIENT_WAIT_OPTION);

the statusRX has the value 0x01 in it, which seems to relate to the following error - 

#define NX_NO_PACKET                               0x01

Added info for those that will try and assist me finding the issue - 

e2studio 7.5.1 Synergy with SSP 1.7.0 running a custom board with S5D9 - Framework used is XWare - NetX Duo Web HTTP/HTTPS Client.

I have tried changing packet pool packet size, as well as number of packets, no joy, I have tried turning on the HTTP Web common fragmentation option, also tried the ip instance fragment option, but none of this helps....

Now from the manual I have this - 

UINT nx_web_http_client_response_body_get(
NX_WEB_HTTP_CLIENT *client_ptr,
NX_PACKET **packet_ptr, ULONG
wait_option);

This service retrieves the next packet of content of the resource
requested by the previous nx_web_http_client_get_start() or
nx_web_http_client_get_secure_start() call. Successive calls to this
routine should be made until the return status of
NX_WEB_HTTP_GET_DONE is received.

This means I should be able to get the entire contents from the server until I get NX_WEB_HTTP_GET_DONE... and this is not the case.

I want to add that before I switched to HTTPS, I had this test code with no security to check if I could receive large files, I used miniweb as the web server to test as its small and efficient at just hosting a file for testing on local PC, this is the code I used and it worked to receive a file with over 400 KB of data - 

    if(status == NX_SUCCESS){
        /* Start GET request. */
        status = nx_http_client_get_start(&g_http_client0, HTTP_CLIENT_HOST_IP_ADDRESS,
                                          HTTP_CLIENT_PATH, NX_NULL, 0, NX_NULL,
                                          NX_NULL, HTTP_CLIENT_WAIT_OPTION);

        if(status == NX_SUCCESS){
            do{
                for(i = 0; i < RECEIVE_BUFFER_SIZE; i++){
                    g_receive_buffer[i] = 0;
                }

                /* Get a response packet. */
                statusRX = nx_http_client_get_packet(&g_http_client0, &receive_packet, HTTP_CLIENT_WAIT_OPTION);

                /* Check to see if we have a packet. */
                if (status == NX_SUCCESS)
                {
                    /* Retrieve the data from the packet. */
                    status = nx_packet_data_retrieve(receive_packet, g_receive_buffer, &bytes_copied);

                    processRXData((char *)g_receive_buffer);

                    /* Release the packet. */
                    status = nx_packet_release(receive_packet);
                }
            }while(statusRX != NX_HTTP_GET_DONE);
        }
    }
 
so please keep in mind the new code does not use the nx_web_http_client_get_secure_start version of the above code, it uses the -
nx_web_http_client_secure_connect, then
nx_web_http_client_request_initialize, then
nx_web_http_client_request_send
and the reasons for this according to the manual is the nx_web_http_client_get_secure_start is used if no custom headers are needed, and the above method uses the custom header method, the receive code would or should stay the same.....
 
So I appologise for the long question, but I want to make sure all question are answered to be able to get the help I need to understand why the HTTPS wont read data past the 4094 mark.
 
Regards,
Ronald Hare
  • 0 in reply to Jeremy
    This is not my part of the code, got this from a thread on this site, but this is the tls_setup.c file

    #include "fw_download_thread.h"
    #include "AmazonRootCA1.h"
    #include "tls_setup.h"

    typedef struct TRUSTED_CERTIFCATE_INFO
    {
    UCHAR * pointer_to_cert;
    USHORT sizeof_cert;
    }TRUSTED_CERTIFCATE_INFO_STRUCT;

    const TRUSTED_CERTIFCATE_INFO_STRUCT g_trust_certs[] =
    {
    [0]
    {
    .pointer_to_cert = (UCHAR *)(AmazonRootCA1),
    .sizeof_cert = sizeof(AmazonRootCA1),
    },
    };

    /* Allow up to 4 certificates from the Server */ // jlc not sure of this...
    #define NO_OF_REMOTE_CERTS (4)

    /* Typical certificates with RSA keys of 2048 bits
    using SHA-256 for signatures are in the range of 1000-2000 bytes. The
    buffer should be large enough to at least hold that size, but depending on
    the remote host certificates may be significantly smaller or larger. Note
    that if the buffer is too small to hold the incoming certificate, the TLS
    handshake will end with an error. */
    #define SIZE_OF_REMOTE_CERT_BUF (2 * 1024)

    /* nx_secure_tls_session_packet_buffer_set() associates a packet re-assembly buffer to a TLS session.
    In order to decrypt and parse incoming TLS records, the data in each record
    must be assembled from the underlying TCP packets. TLS records can be
    up to 16KB in size (though typically are much smaller) so may not fit into a
    single TCP packet.
    If you know the incoming message size will be smaller than the TLS
    record limit of 16KB, the buffer size can be made smaller, but in order to
    handle unknown incoming data the buffer should be made as large as
    possible. If an incoming record is larger than the supplied buffer, the TLS
    session will end with an error.*/
    #define SIZE_OF_TLS_PACKET_BUF (16 * 1024)

    /* This RSA Module size is defined in the TLS Common property Maximum RSA module size (bits). The default value is 4096, this is used
    * to establish the size of the encryption buffer (meta data processing). */
    #define SIZE_OF_META_BUFFER (12192) /* calculated with nx_secure_tls_metadata_size_calculate() for nx_crypto_tls_ciphers_synergys7, RSA Modulus 4096 */


    /* Define memory area for processing remote certificate(s) */
    UCHAR g_remote_cert_buffer[NO_OF_REMOTE_CERTS][SIZE_OF_REMOTE_CERT_BUF];

    NX_SECURE_X509_CERT g_remote_cert[NO_OF_REMOTE_CERTS];

    /* A TLS client requires a minimum of one "trusted" certificate to verify
    the incoming server certificate. This is usually the "trusted root CA"
    certificate that issued the server's certificate. */
    static NX_SECURE_X509_CERT trusted_certificate;

    /* Define the memory area for processing remote certificate data. */
    UCHAR g_tls_packet_buffer[SIZE_OF_TLS_PACKET_BUF];

    /* Define buffer area for encryption. */
    CHAR g_tls_session_meta_data[SIZE_OF_META_BUFFER];

    /* This crypto table is defined in and exclusively used by the Synergy cryptography driver */
    extern const NX_SECURE_TLS_CRYPTO nx_crypto_tls_ciphers_synergys7;

    /* Define the callback for establishing a secure socket connectioon. Pass a pointer to a TLS session instance,
    * which this funtion will create for the caller, and a pointer to the HTTPS client to register the session with. */
    UINT tls_setup_callback(NX_WEB_HTTP_CLIENT *p_client_ptr, NX_SECURE_TLS_SESSION *p_tls_session)
    {
    UINT status;
    uint32_t i;

    SSP_PARAMETER_NOT_USED(p_client_ptr);

    /* Create a TLS session */
    memset(g_tls_session_meta_data, 0, sizeof(g_tls_session_meta_data));
    status = nx_secure_tls_session_create (p_tls_session, &nx_crypto_tls_ciphers_synergys7, g_tls_session_meta_data, sizeof(g_tls_session_meta_data));
    if (NX_SUCCESS != status)
    {
    return status;
    }

    /* Clear memory space for packet re-assembly */
    memset(g_tls_packet_buffer, 0, sizeof(g_tls_packet_buffer));

    /* Register this buffer space with the session. */
    status = nx_secure_tls_session_packet_buffer_set(p_tls_session, g_tls_packet_buffer, sizeof(g_tls_packet_buffer));
    if (NX_SUCCESS != status)
    {
    return status;
    }

    /* Initialize and register a trusted CA Certificate to our trusted store for verifying incoming server certificates. */
    status = nx_secure_x509_certificate_initialize(&trusted_certificate, &AmazonRootCA1[0], sizeof(AmazonRootCA1), NX_NULL, 0, NULL, 0, NX_SECURE_X509_KEY_TYPE_NONE);
    /* Check for error. */
    if (status)
    {
    return status;
    }

    // status = nx_secure_tls_local_certificate_add(p_tls_session, &trusted_certificate);
    status = nx_secure_tls_trusted_certificate_add(p_tls_session, &trusted_certificate);

    /* Check for error. */
    if (status)
    {
    return status;
    }

    /* Allocate buffer space for the specified number of remote certificates. */
    for( i = 0; i < NO_OF_REMOTE_CERTS; i++)
    {
    /* Allocate buffers for in-coming certificates */
    memset(&g_remote_cert_buffer[i][0], 0, SIZE_OF_REMOTE_CERT_BUF);
    memset(&g_remote_cert[i], 0, sizeof(NX_SECURE_X509_CERT));
    status = nx_secure_tls_remote_certificate_allocate(p_tls_session, &g_remote_cert[i], &g_remote_cert_buffer[i][0], SIZE_OF_REMOTE_CERT_BUF);
    if (NX_SUCCESS != status)
    {
    return status;
    }
    }

    return NX_SUCCESS;
    }

    Regards, Ronald Hare
  • 0 in reply to Ghost
    What size are the packet pools in the system?
  • 0 in reply to Jeremy
    Packet pool has packet bytes 1568, and number if packets is 16
  • 0 in reply to Ghost

    can i get code for the https client and how to insert the server certificate on to the device..?