AllBASIC

BASIC Developer & Support Resources => Scripting Languages => Topic started by: John on August 15, 2021, 01:26:33 pm

Title: cURL Windows
Post by: John on August 15, 2021, 01:26:33 pm
I spent more time debugging the URL issue than writing the script.  :(

If I ever find the time, I'm going to replace JOKER with MATCH. A side affect of English not being Peter's native language.

Title: Re: cURL Windows
Post by: AIR on August 15, 2021, 06:59:52 pm
You might want to alias it instead so you don't break existing/old code....
Title: Re: cURL Windows
Post by: John on August 15, 2021, 08:50:52 pm
Good point AIR. An alias makes a lot of sense.

It looks like SB can't OPEN a socket connection for a HTTPS URL even using 443 as the port.
Title: Re: cURL Windows
Post by: AIR on August 15, 2021, 09:54:21 pm
If I remember, SB sockets don't use ssl, so no TLS support.  That would have to be added for this to work.

Title: Re: cURL Windows
Post by: John on August 15, 2021, 11:21:05 pm
Thanks AIR.

Your code challenge has created list of issues not working with SB.

Title: Re: cURL Windows
Post by: AIR on August 16, 2021, 12:26:15 pm
Are you using MinGW to build on Windows (I don't remember)?

If so, I think if you add these two flags to your CFLAGS for the Curl module, it should enable the Windows native ssl support, eliminating the need for OpenSSL:

-DUSE_WINDOWS_SSPI
-DUSE_SCHANNEL

You might also need to change the init call, not sure.  Check the Curl documentation....

AIR.
Title: Re: cURL Windows
Post by: John on August 16, 2021, 12:47:47 pm
I'm using TDM-GCC-32 latest 9.2 I think. I'm using your MAKE (slightly modified) for SB you created. I now create two SB versions. SBC for console and SBW for GUI based programs. Both support Windows styles and themes.

Getting cURL HTTPS to work on Windows has been a challenge.

I will give your suggestions a try and see if I can get it going.

Title: Re: cURL Windows
Post by: John on August 16, 2021, 01:10:45 pm
Here is the make file I have been using. I just downloaded the cURL source but lost how to get this compiled. Any direction or guidance would be appreciated.

Code: C
  1. all : bin\mod\lib\curl.a bin\mod\dll\curl.dll
  2.  
  3.  
  4. bin\mod\lib\curl.a : bin\mod\obj\curl\s_interface.o
  5.         ar -r ..\..\bin\mod\lib\curl.a ..\..\bin\mod\obj\curl\s_interface.o
  6.  
  7. bin\mod\dll\curl.dll : bin\mod\obj\curl\interface.o
  8.         gcc -Wno-write-strings -static-libgcc -pipe -shared -w -s -O2 -m32 -Wl,--add-stdcall-alias,--output-def,"..\..\bin\mod\dll\curl_imp.def",--out-implib,"..\..\bin\mod\dll\curl_imp.a" -s -o ..\..\bin\mod\dll\curl.dll ..\..\bin\mod\obj\curl\interface.o -lm -lpthread -lcurl
  9.  
  10. bin\mod\obj\curl\interface.o : interface.c
  11.         gcc -w -D_WIN32 -m32 -c -o ..\..\bin\mod\obj\curl\interface.o interface.c
  12.  
  13. bin\mod\obj\curl\s_interface.o : interface.c
  14.         gcc -w -D_WIN32 -DSTATIC_LINK=1 -m32 -c -o ..\..\bin\mod\obj\curl\s_interface.o interface.c
  15.  
  16.  
Title: Re: cURL Windows
Post by: John on August 16, 2021, 02:08:24 pm
I was able to get HTTPS working on Windows using these old files. I have no idea where the source is.  The LIBCURL.DLL is in the bin directory.

Your RSS code challenge on Windows works with these files.

I'm going to download the binary form (DLL) of LIBCURL and compile against it. I think that is how what works was built. What I have must be trying to create a standalone extension module.



Title: Re: cURL Windows
Post by: John on August 16, 2021, 10:30:01 pm
AIR,

I got everything setup with the latest libcurl libraries and include. This is the errors I'm getting trying to build the extension module. I have seen this before but I forget what you had me do to fix it.

Code: Bash
  1. C:\sbgcc\source\extensions\curl>mingw32-make -B -f Makefile
  2. gcc -w -D_WIN32 -DSTATIC_LINK=1 -m32 -c -o ..\..\bin\mod\obj\curl\s_interface.o interface.c
  3. In file included from interface.c:27:
  4. ../../basext.h:837:41: error: static declaration of '_init' follows non-static declaration
  5.   837 | #define besSUB_PROCESS_START static int _init(){
  6.       |                                         ^~~~~
  7. interface.c:2214:1: note: in expansion of macro 'besSUB_PROCESS_START'
  8.  2214 | besSUB_PROCESS_START
  9.       | ^~~~~~~~~~~~~~~~~~~~
  10. ../../basext.h:978:7: note: previous declaration of '_init' was here
  11.   978 |   int _init(void);\
  12.       |       ^~~~~
  13. interface.c:2212:1: note: in expansion of macro 'besDLL_MAIN'
  14.  2212 | besDLL_MAIN
  15.       | ^~~~~~~~~~~
  16. ../../basext.h:843:42: error: static declaration of '_fini' follows non-static declaration
  17.   843 | #define besSUB_PROCESS_FINISH static int _fini(){
  18.       |                                          ^~~~~
  19. interface.c:2219:1: note: in expansion of macro 'besSUB_PROCESS_FINISH'
  20.  2219 | besSUB_PROCESS_FINISH
  21.       | ^~~~~~~~~~~~~~~~~~~~~
  22. ../../basext.h:979:7: note: previous declaration of '_fini' was here
  23.   979 |   int _fini(void);\
  24.       |       ^~~~~
  25. interface.c:2212:1: note: in expansion of macro 'besDLL_MAIN'
  26.  2212 | besDLL_MAIN
  27.       | ^~~~~~~~~~~
  28. Makefile:14: recipe for target 'bin\mod\obj\curl\s_interface.o' failed
  29. mingw32-make: *** [bin\mod\obj\curl\s_interface.o] Error 1
  30.  
  31. C:\sbgcc\source\extensions\curl>
  32.  
Title: Re: cURL Windows
Post by: John on August 16, 2021, 11:54:46 pm
I got it to compile remarking out the following.

Code: C
  1. besDLL_MAIN
  2.  
  3. besSUB_PROCESS_START
  4.   INIT_MULTITHREAD
  5.   return 1;
  6. besEND
  7.  
  8. besSUB_PROCESS_FINISH
  9.   FINISH_MULTITHREAD
  10. besEND
  11.  

Compile
Code: Bash
  1. C:\sbgcc\source\extensions\curl>mingw32-make -B -f Makefile
  2. gcc -w -D_WIN32 -DSTATIC_LINK=1 -m32 -c -o ..\..\bin\mod\obj\curl\s_interface.o interface.c
  3. ar -r ..\..\bin\mod\lib\curl.a ..\..\bin\mod\obj\curl\s_interface.o
  4. gcc -w -D_WIN32 -m32 -c -o ..\..\bin\mod\obj\curl\interface.o interface.c
  5. gcc -Wno-write-strings -static-libgcc -pipe -shared -w -s -O2 -m32 -Wl,--add-stdcall-alias,--output-def,"..\..\bin\mod\dll\curl_imp.def",--out-implib,"..\..\bin\mod\dll\curl_imp.a" -s -o ..\..\bin\mod\dll\curl.dll ..\..\bin\mod\obj\curl\interface.o -lm -lpthread -lcurl
  6.  
  7. C:\sbgcc\source\extensions\curl>
  8.  

When I try to run the RSS challenge I get the following extension error I had with the old curl extension module I compiled. This error code isn't defined in the source. The error happens executing the curl::perform() function. Since it works with http URLs it has to be something unique to this version of libcurl.dll and SSL.


(0): error 0x0008113c:Extension specific error: %s


On a positive note the new curl extension module works with http URLs.

These gcc arguments didn't help.

-DUSE_WINDOWS_SSPI
-DUSE_SCHANNEL

I replace the new libcurl.dll that came in the distribution with the one I posted earlier and it works.  Version: 7.50.1.0

The new version is 7.78.0.0 and fails.

I compiled the new curl.dll extension module using the new includes and libcurl.a.

The new curl.exe works with the RSS challenge HTTPS URL.
Title: Re: cURL Windows
Post by: John on August 17, 2021, 02:48:28 am
Success!

Adding the following curl options allowed the new libcurl.dll to work.

Code: Script BASIC
  1. ch = curl::init()
  2. curl::option(ch, "URL", "https://s.ch9.ms/Events/MIX/MIX11/RSS/mp4high")
  3. curl::option(ch, "SSL_VERIFYPEER")
  4. curl::option(ch, "CAINFO","C:\\ScriptBasic\\bin\\curl-ca-bundle.crt")
  5. rss = curl::perform(ch)
  6. curl::finish(ch)
  7.  

This problem of Windows not finding the certificate store may be due to when OpenVPN wiped out my Windows system path and I had to rebuild it from what info I could find on the web. I may be missing that path. (guess) Why the earlier version of the libcurl.dll works is a mystery to me. Outlook keeps asking me to verify my certificate but won't save it.

Title: Re: cURL Windows
Post by: John on August 17, 2021, 03:35:19 pm
I tried the new curl extension module on my older laptop running Windows 10 Pro. I use this laptop for testing and not development. It won't run unless I have he the new lines I added to the rss.sb challenge script. It seems things have changes since 7.5 and a it checks for a certificate before connecting to a HTTPS site.

Title: Re: cURL Windows
Post by: AIR on August 17, 2021, 07:53:25 pm
Something must be 'broken' on your side.

This works on both my Mac and my Linux server:

Code: C
  1. #include <curl/curl.h>
  2. #include <stdio.h>
  3. // #include <stdlib.h>
  4. #include <string.h>
  5.  
  6. long write_data (char *content, long size, long nmemb, char *stream) {
  7.   strcat(stream,content);
  8.   return size*nmemb;
  9. }
  10.  
  11.  
  12.  
  13. int main(int argc, char **argv) {
  14.     char buffer[128000][10];
  15.     CURL    *curl;
  16.     CURLcode res;
  17.  
  18.     curl_global_init(CURL_GLOBAL_DEFAULT);
  19.     curl = curl_easy_init();
  20.  
  21.     if (curl) {
  22.         curl_easy_setopt(curl, CURLOPT_URL, "http://channel9.msdn.com/Events/MIX/MIX11/RSS/mp4high");
  23.         curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
  24.         curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
  25.         curl_easy_setopt(curl, CURLOPT_WRITEDATA, buffer);
  26.  
  27.         res = curl_easy_perform(curl);
  28.  
  29.         if(res != CURLE_OK) {
  30.             printf("\ncurl_easy_perform() failed: %s\n",curl_easy_strerror(res));
  31.         } else {
  32.             printf("%s\n", *buffer);
  33.         }
  34.  
  35.     }
  36.  
  37.     curl_easy_cleanup(curl);
  38.     curl_global_cleanup();
  39.     return 0;
  40. }

Note that I'm using the 'http' address as specified in the challenge.

"curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);" is what tells cURL to process redirects.

AIR.
Title: Re: cURL Windows
Post by: John on August 17, 2021, 09:58:47 pm
I didn't have any issues on Linux either. Windows cURL is causing the problem. Are you using the latest release of LIBCURL?

The URL in your last example is not the URL in the original post.
Title: Re: cURL Windows
Post by: AIR on August 17, 2021, 10:42:16 pm
It's a Windows specific issue.  Just tried with latest binary build (7.78), same issue...
Title: Re: cURL Windows
Post by: John on August 17, 2021, 10:53:12 pm
Thanks AIR for the validation. At least there is a solution.
Title: Re: cURL Windows
Post by: AIR on August 18, 2021, 04:35:10 am
The URL in your last example is not the URL in the original post.

Neither is yours.  ;D

Changed it in mine.
Title: Re: cURL Windows
Post by: AIR on August 18, 2021, 04:52:40 am
cURL on Windows is stupid.  By default, it doesn't use the Windows Certificate Store.

Instead of passing the crt file, John, try this single flag instead to force it to use the Windows Certificate Store:

Code: C
  1.         curl_easy_setopt(curl, CURLOPT_SSL_OPTIONS , CURLSSLOPT_NATIVE_CA);
  2.  

AIR.
Title: Re: cURL Windows
Post by: John on August 18, 2021, 08:54:54 am
Not sure if the SB curl ext. module has that option. Checking now.

It doesn't look like that option is available. I guess I'm stuck with passing the cert.
Title: Re: cURL Windows
Post by: John on August 18, 2021, 10:19:29 am
An option would be to distribute the 7.5 LIBCURL.dll which works with HTTPS. I don't know if this a bug or how they have chosen to move forward?
Title: Re: cURL Windows
Post by: AIR on August 18, 2021, 11:06:15 am
Another option would be to use the Windows Native SSL implementation instead of OpenSSL.

See attached archive which contains both static and dynamic libraries that I built a little while ago from the current released cURL source.

AIR.
Title: Re: cURL Windows
Post by: John on August 18, 2021, 11:18:09 am
Great!

I give this a try.
Title: Re: cURL Windows
Post by: John on August 18, 2021, 12:07:44 pm
Your libcurl.dll doesn't work and returns an extension module error. (my OPTIONS remarked)

It doesn't work even if I use the two cURL OPTIONS I added.
Title: Re: cURL Windows
Post by: AIR on August 18, 2021, 12:10:26 pm
did you rebuild the module using the libs, or did you simply try to plug them in?
Title: Re: cURL Windows
Post by: John on August 18, 2021, 12:13:21 pm
I just copied your libcurl.dll into my BIN directory.

I'll try rebuilding the extension module with your code. Do I need to modify my Makefile?
Title: Re: cURL Windows
Post by: John on August 18, 2021, 12:25:07 pm
I rebuilt the SB curl.dll extension module using your libcurl.a and used your libcurl.dll. I get an ext. module error trying to use it.
Title: Re: cURL Windows
Post by: AIR on August 18, 2021, 12:55:17 pm
They are both the same library, one static one dynamic.  Use one or the other.
Title: Re: cURL Windows
Post by: John on August 18, 2021, 01:19:34 pm
I tried your static and dynamic and neither work. I don't have libz.a either.

I'm going with the 7.5 LIBCURL.DLL that works with no additional options. If someone wants to use the latest libcurl.dll for some reason they can pass the cert.
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 06:42:58 am
Thanks, John.

Would it be possible for you to clean up this topic by splitting off the posts that centered around cURL?

AIR.
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 10:09:07 am
John,
Which C package is the windows version of SB built with?  VC or MinGW?  Just curious because the dll was built with MinGW....
Title: Re: cURL Windows
Post by: John on August 19, 2021, 02:03:20 pm
The  Windows SB core release was built with TDM-GCC-32.

Can you try building the cURL extension module for Windows 32 and see if you can get it working with your cURL source build?
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 05:28:05 pm
I don't have a 32bit environment set up.
Question:  Were you trying my cURL build on 32bit SB?  What I built is 64bit......
Title: Re: cURL Windows
Post by: John on August 19, 2021, 08:04:55 pm
The SB core build I just released is 32 bit. I'm not doing anything in 64 bit on Windows. COM/OLE is a big part of what I use it for.

I'm surprise the extension module compiled being that libcurl.a was 64 bit.
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 08:36:18 pm
Can you post your makefile for windows?
Title: Re: cURL Windows
Post by: John on August 19, 2021, 08:40:59 pm
I already did. It's part of this thread.

I would like to do a COM/OLE challenge but afraid I would lose your interest. :(
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 09:11:16 pm
I meant for the entire thing.
It's okay, I pulled the old 64bit archive from a backup.
Title: Re: cURL Windows
Post by: John on August 19, 2021, 09:15:21 pm
AIR,

I would be happy to zip up my Windows 32 SB source directory and send it to you.
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 09:22:37 pm
No Need.
Title: Re: cURL Windows
Post by: John on August 19, 2021, 09:24:41 pm
I just sent it to your Gmail account before reading this.


This will give you a peek at the Makefile you created and I modified.
Title: Re: cURL Windows
Post by: John on August 19, 2021, 09:30:21 pm
It bounced. Here is the Makefile.
Title: Re: cURL Windows
Post by: AIR on August 19, 2021, 10:06:58 pm
Don't need it, I don't have 32bit windows or dev environment.

See attached for 64bit scriba with 64bit curl module using Windows Native SSL and test file.  Keep it all together for now in same folder.

Code: [Select]
--------------------
Title: Riding the Geolocation Wave
Presenter: Tatham Oddie
Date: Tue, 15 Mar 2011 20:26:00 GMT
Description: It&rsquo;s pretty obvious by now that geolocation is a heavy player in the next wave of applications and APIs. Now is the time to learn how to take advantage of this information and add context to your own applications. In this session we&rsquo;ll look at geolocation at every layer of the stack &ndash; from open protocols to operating system APIs, from the browser to Windows Phone 7. Building a compelling geo-enabled experience takes more than simple coordinates. In this session we will introduce the basics of determining a user&rsquo;s location and then delve into how to make geo data a useful addition to your application instead of a bolt-on &ldquo;me too&rdquo;.
Video Link: http://video.ch9.ms/ch9/7ce8/f849553d-644e-4dad-8deb-9ea700dd7ce8/MIX11EXT04_high_ch9.mp4
Duration: 00:47:52
--------------------
Title: Flickr API: Tap Into Billions of Photos for Windows Phone 7
Presenter: Markus Spiering
Date: Tue, 15 Mar 2011 20:26:00 GMT
Description: Do you want your app to integrate with one of the world&rsquo;s leading photosharing site? The Flickr for Windows Phone 7 app is a full featured mobile application that is pushing the boundaries of UX design and is 100% built on Flickr's API methods that are available for developers. This talk will introduce the mobile app, highlight some of the complex design decisions, and provide insights into the Flickr API methods. Attendees will learn how to enrich their applications using Flickr's rich content via the Flickr APIs.
Video Link: http://video.ch9.ms/ch9/7BAE/1F7A8CCB-5580-47BF-9371-9EA700DD7BAE/MIX11EXT01_high_ch9.mp4
Duration: 00:50:32
--------------------
Title: Fonts, Form and Function: A Primer on Digital Typography
Presenter: Robby Ingebretsen
Date: Tue, 15 Mar 2011 20:26:00 GMT
Description: Typography in digital experiences is unavoidable, and for years it was a fight we mostly lost. Today, however, technology is on our side! High resolution screens, an expanding library of open fonts and new flexibility in nearly every UI technology have made digital typography more fun and more interesting than ever. We'll cover all aspects of working with digital type: everything from choosing complimentary typefaces to licensing, rendering and a system for layout and sizing. We all love type. Come to this talk to learn why!
Video Link: http://video.ch9.ms/ch9/7bc8/133b3b99-1949-4ef4-a27c-9ec6010c7bc8/MIX11EXT02_high_ch9.mp4
Duration: 01:03:27

AIR.
Title: Re: cURL Windows
Post by: John on August 28, 2021, 05:02:23 pm
AIR,

I'm trying to compile the cURL module but don't have the work you did to create the module. Can you send me your curl extension module directory and anything else I need to build it?

Thanks!