AllBASIC Forum

BASIC Developer & Support Resources => Compilers => Topic started by: JRS on September 23, 2010, 01:53:58 AM

Title: Alternative O2 Builder
Post by: JRS on September 23, 2010, 01:53:58 AM
Charles,

I have been sifting through your source code and from what I see, FreeBASIC is being used as an scripting engine to build a ASM source code program. Instead of using an external assembler, O2 embeds it's generated source as inline ASM statements which FreeBASIC assembles and does any linking that needs to be done.

For grins, lets say I was able to convert the FreeBASIC version of O2 to ScriptBasic and used the gcc's inline ASM (http://ibiblio.org/gferg/ldp/GCC-Inline-Assembly-HOWTO.html#s2) feature to create the O2 compiler. Is this theoretically possible?

Wouldn't that offload all that work you have to do to be C friendly? (read C headers, let C be your generic interface helper shell, expand to cross platform, ...)

John

Quote from: Charles - FreeBASIC Forum
Hi everyone!

Thank you very much for all your interesting comments on Oxygen Basic. I was about to announce the project on this forum, but John got in there first :)

I am more than satisfied with FreeBasic as a development language. While I have not made use of its more advanced features, it has proved to be very reliable, and interfaces nicely with assembly code. Thank you Joshy for posting the compatibility fix with FB 0.21

O2 was conceived nearly three years ago as a machine script, in essence machine code with a linker. It was an experiment to see how minimal a compilable language could be. Then came the Assembler shortly followed by a macro language. Eros Olmi (of thinBasic) suggested more people would be able to understand it if I added Basic-like syntax. So I thought okay, I'll have this ready in a few weeks. Little did I know how much work was involved in producing a high level compilable language. A compiler has to spend much of its time doing invisible things like automatic type conversion and operator precedence and one is not fully aware of these complexities until you try to encode them.

Even before adding Basic, I had OOP capability in place. I think retrofitting it to a straight procedural language would have been much more difficult which I think is why many Basics have got stuck in their development.

One very importand and most recent feature in the development of o2 is the ability to read C headers as well as Qbasic headers. This requires O2 to understand quite a large chunk of C itself. So far, this fusion of C and basic has not been too problematic. A Few directives are needed. For instance partial case sensitivity: #case capital which distinguishes fully capitalised keywords.

I use the term JIT because O2 compiles directly to an executable memory image which may be used immediately without first storing it to an EXE file. For programs up to about 40-50k of source code, the compilation time is barely noticeable. This is good for ad-hoc programming where you are not building a static application. The source code is the only item that needs to be stored.

Currently O2 compiled executables depend on Oxygen.dll for the run-time library, but I am working on the 'independent' mode where the run-time library is included. This is of course essential when generating 64bit code with a 32 bit compiler.

I am still devoted to GOTOs. But I have only used them very sparingly 372 times :D. Many of these are error trapping points, and many GOTOs are directed towards a small number of nodes in the program.

It will take about three more months to approach Beta quality releases. Much of the testing at this stage involves exposing Oxygen to as many different situations as possible and testing with various APIs.


Charles

http://oxygenbasic.sourceforge.net
 

Title: Re: Alternative O2 Builder
Post by: cevpegge on September 23, 2010, 03:52:26 AM

John,

Oxygen makes intensive use of FreeBasic Inline Assembler (which is slightly preprocessed GNU GAS using Intel notation). The Oxygen Assembly language is largely compatible with this. We also have a high degree of compatibility between FreeBasic and OxygenBasic language. Then there is the evolving C compatibility. And operating system dependencies are absolutely minimal at this stage so I hope this is sufficient to keep Oxygen's porting options open.

However porting to SB, being a scripting language would pose a few problems like being unable to tightly couple Assembly code with the host script. But theoretically any Turing machine will do.

Looking back over the last 35 years. What has been the most stable language? X86 Assembler!  ;D Who would have thought it! But so many other processors, despite their more elegant architecture, are rapidly heading for extinction, and the main rival is the ARM - whose cores are embedded in the processors of many mobile devices. ARM Assembler is a joy to code. I would love to develop an ARM version of Oxygen should the opportunity arise.

Charles


Title: Re: Alternative O2 Builder
Post by: JRS on September 23, 2010, 04:16:15 AM
Quote
However porting to SB, being a scripting language would pose a few problems like being unable to tightly couple Assembly code with the host script.

<Start Theory>ScriptBasic would only act as an interactive C/ASM source code generating tool. It outputs C / C inline ASM text to be compiled by a C compiler. Once you have a OxygenBasic compiler compiled, SB & gcc would no longer be required.<End Theory>

Maybe I don't fully understand the dependency on FreeBASIC when generic inline ASM is the outputted result. With no real development going on with FreeBASIC in years, it seems a bit risky to bet the farm on it.
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 23, 2010, 05:41:43 AM

Yes I see. I think I would either go for C or self-compile. But the worst that could happen to FB is going into stasis and that largely depends on its major users. The most useful thing about basic is good string manipulation and easy access to assembler. Compilers don't really require more than that.

I spend much of my time refactoring the source code (It can be Draconian sometimes.)  and FB is a comfortable and familiar environment to do this. And FB compiles O2 in less than 2 seconds. It is almost interactive. The only thing I really miss is GOSUB. I often need subroutines that have access to all the parental variables. This would reduce my global list considerably.

Charles
Title: Re: Alternative O2 Builder
Post by: JRS on September 23, 2010, 09:57:40 AM
Your right. C or BCX makes more sense.

Quote from: BCX Docs
$ASM directive

Purpose: The $ASM directive allows inline assembly instructions to be used with BCX. An $ASM directive must be placed before and after the assembly statements.

 Syntax:

 $ASM
  /** Assembly statements go here */
 $ASM

Code: Text
  1.  DIM str1$
  2.  DIM str2$
  3.    
  4.  str1$ = "123456789"
  5.    
  6.  str2$ = AsmLeft$(str1$, 3)
  7.  PRINT str2$
  8.    
  9.  str2$ = AsmMid$(str1$, 4, 3)
  10.  PRINT str2$
  11.    
  12.  str2$ = AsmMid$(str1$, 4)
  13.  PRINT str2$
  14.    
  15.  str2$ = AsmRight$(str1$, 3)
  16.  PRINT str2$
  17.    
  18.    
  19.  FUNCTION SubStr%(SrcStr AS LPSTR, SrcStrLen%, StartPosn%, DestStr AS LPSTR, DestStrLen%)
  20.    $ASM
  21.    nop
  22.    nop
  23.    nop
  24.    
  25.    ;save used registers except for eax:edx
  26.    push ebx
  27.    push ecx
  28.    push edi
  29.    push esi
  30.    
  31.    ;load the 32-bit registers from the stack
  32.    
  33.    mov esi, dword PTR [ebp+8]  ;SrcStr
  34.    mov edx, dword PTR [ebp+12] ;SrcLen
  35.    mov eax, dword PTR [ebp+16] ;StartPosn
  36.    mov edi, dword PTR [ebp+20] ;DestStr
  37.    mov ecx, dword PTR [ebp+24] ;DestLen
  38.    
  39.    ;check for zero-length Destination string
  40.    jecxz NoGood
  41.    
  42.    ;compensate StartPosn for 0 based count
  43.    dec eax
  44.    
  45.    ;check for negative offset of StartPosn
  46.    cmp eax, 0
  47.    jl NoGood
  48.    
  49.    ;point esi to start of SrcStr
  50.    add esi, eax
  51.    
  52.    ;and subtract StartPosn from SrcLen
  53.    sub edx, eax
  54.    
  55.    ;check if offset is past end of SrcStr
  56.    cmp edx, 0
  57.    jle NoGood
  58.    
  59.    ;move substring of esi into edi
  60.    rep movsb
  61.    
  62.    ;return this to indicate function worked
  63.    mov eax, edi
  64.    jmp ReturnAX
  65.    
  66.    NoGood:
  67.    mov eax, -1
  68.    
  69.    ReturnAX:
  70.    pop esi
  71.    pop edi
  72.    pop ecx
  73.    pop ebx
  74.    pop ebp
  75.    ret
  76.    $ASM
  77.    FUNCTION = 0
  78.  END FUNCTION
  79.    
  80.  'Left$ equivalent
  81.  FUNCTION AsmLeft$(fnstr1$, fnint1%)
  82.    DIM RAW startp% = 1
  83.    DIM RAW retstr$ * fnint1% + 256
  84.    SubStr(fnstr1$, LEN(fnstr1$), startp%, retstr$, fnint1%)
  85.    retstr[fnint1] = 0
  86.    FUNCTION = retstr$
  87.  END FUNCTION
  88.    
  89.  'MID$ equivalent
  90.  FUNCTION AsmMid$ OPTIONAL(fnstr1$, fnint1%, fnint2%=0)
  91.    DIM RAW startp% = fnint1%
  92.    DIM RAW lenfnstr1% = LEN(fnstr1$)
  93.    DIM RAW retstr$ * fnint2% + 256
  94.    IF fnint2% = 0 THEN
  95.      fnint2% = lenfnstr1% - startp% + 1
  96.    END IF
  97.    SubStr(fnstr1$, lenfnstr1%, startp%, retstr$, fnint2%)
  98.    retstr[fnint2] = 0
  99.    FUNCTION = retstr$
  100.  END FUNCTION
  101.    
  102.  'RIGHT$ equivalent
  103.  FUNCTION AsmRight$(fnstr1$, fnint1%)
  104.    DIM RAW lenfnstr1% = LEN(fnstr1$)
  105.    DIM RAW startp% = lenfnstr1% - fnint1% + 1
  106.    DIM RAW retstr$ * fnint1% + 256
  107.    SubStr(fnstr1$, lenfnstr1%, startp%, retstr$, fnint1%)
  108.    retstr[fnint1] = 0
  109.    FUNCTION = retstr$
  110.  END FUNCTION
  111.  

Quote
GOSUB ... RETURN statement

Purpose: GOSUB redirects program flow to a label. The flow continues from the label until a RETURN statement is encountered and the flow is returned to the line following the GOSUB Label statement.

 Syntax:

 GOSUB Label

 Label:
  Statements
 RETURN

Code: [Select]
DIM j
 
 CLS
 
 PRINT "Calling Subroutine 9 times ..." : PRINT
 
 FOR j = 1 TO 9
   GOSUB qwerty
 NEXT j
 
 PRINT : PRINT "All Done!"
 
 END                       ' Don't accidently fall into the subroutine
 
 QwErTy:                   ' BCX Subroutine Labels are NOT case sensitive
 
 PRINT " This is instance >>", j, " <<  of the subroutine call."
 
 RETURN                    ' Return from subroutine back to main program
Title: Re: Alternative O2 Builder
Post by: efgee on September 23, 2010, 05:11:17 PM
Charles,

Oxygen makes intensive use of FreeBasic Inline Assembler (which is slightly preprocessed GNU GAS using Intel notation). The Oxygen Assembly language is largely compatible with this. We also have a high degree of compatibility between FreeBasic and OxygenBasic language.

If the Oxygen Assembly language is largely compatible with gas, maybe you could have saved yourself quite some time and utilize the whole "BASIC language" frontend from freeBasic.
Like:

Code: [Select]
fbc -gen oxygen hello.bas

if oxygen is chosen than co2.exe is used instead of gas.exe.
The code to make this possible could have been done in one afternoon...

With that you could have had tons of new users/testers in one night!

efgee
Title: Re: Alternative O2 Builder
Post by: JRS on September 23, 2010, 10:26:59 PM
I'm going to take a guess here but I think Charles has his own ideas how a Basic and a hand coded compiler should work.

I don't see why O2 ASM as defined in FreeBASIC and using the GNU Assembler (http://en.wikipedia.org/wiki/GNU_Assembler) should not translate well to BCX Inline ASM and compiled with MinGW-gcc under windows for 32 and 64 bit.

Quote
The GNU Assembler (GAS) is the default back-end to the GNU Compiler Collection (GCC) suite. As such, GAS is as portable and retargetable as GCC is. However, GAS uses the AT&T syntax for its instructions, which some users find to be less readable than Intel syntax. As a result, assembly code written inline in a C code file for GCC must also be written in GAS syntax.

GAS is developed specifically to be used as the GCC backend. GCC always feeds GAS syntactically-correct code, so GAS often has minimal error checking.

GAS is available as a part of either the GCC package or the GNU binutils package.
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 24, 2010, 12:31:22 AM

The important thing at present is to work with a single "incarnation" of Oxygen when making a large number of changes. But you have given me ideas about emitting various types of code. With a few adjustments Oxygen could easily emit C or Assembly code, as well as binary. :)

One curious phenomenon: When Oxygen was just an Assembler it was around 150K in size. Adding Basic produced a rapid escalation to around 370k. But while I continue to add new functionality and refactor at the same time, the size has not changed significantly in several months. It seems to have reached a critical mass where only small changes are required to add new functionality. I can always find ways to reduce the code before adding new things. Its a bit like file zipping by hand.

Charles
Title: Re: Alternative O2 Builder
Post by: JRS on September 24, 2010, 01:11:46 AM
Quote
The important thing at present is to work with a single "incarnation" of Oxygen when making a large number of changes.

All I'm saying is that it would seem easier to integrate C libraries (reading headers, inline C in O2Basic, ...) if you used a C compiler to compile your initial compiler. (say that 3 times in a row)  :)

I hate to see you wasting your time when needed functionality already exists when using the correct compiler.



Title: Re: Alternative O2 Builder
Post by: cevpegge on September 24, 2010, 02:17:16 AM
John,

I am considering porting to C but this is not a priority right now. It would disrupt the work flow. Much of my design effort at the moment is in getting the architecture of the program right. This is more or less language-independent. With FreeBasic firmly embedded in my subconsciousness it's one less thing to think about.

C headers are not a problem. Most of the work to read them has been done.

Charles
Title: Re: Alternative O2 Builder
Post by: JRS on September 24, 2010, 02:28:59 AM
I understand. FreeBasic is a vessel that lets you construct an independent compiler in BASIC.

I'll leave you alone for now.  :-X   I'm having too much fun turning the SB webserver into a multi-threaded desktop application server.  ;D

Actually this concept should work just as well using Apache with PHP or Python. Gtk has bindings for both.
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 24, 2010, 03:03:56 AM

Okay John :) What is your time zone by the way. Mine is GMT though I tend to work in 4 hours shifts around the clock.

Charles
Title: Re: Alternative O2 Builder
Post by: JRS on September 24, 2010, 03:06:02 AM
It's 3 am and I just exhausted my last wind so I'm going to bed.



Title: Re: Alternative O2 Builder
Post by: cevpegge on September 28, 2010, 05:41:26 AM
I tried compiling Oxygen with FreeBasic version 0.21 instead of the usual 0.20b. But it was not successful unfortunately. Something fundamental may have changed. Not to mention the indignity of getting locked out of the FB forum about 7 hours ago, and no administrator to be found for assistance, I will make haste and start to plan a C migration. It makes a lot of sense in the long term.

The state-of-the-art parallel processing APIs which make use of the Graphics Hardware cores, OpenCL and Cuda are based on C. There is no escape from C!

Charles
Title: Re: Alternative O2 Builder
Post by: JRS on September 28, 2010, 07:22:35 AM
Charles,

Moving to C is a wise and safe choice. The FreeBASIC group can't even get it together to move their own project along. FreeBASIC had it's day but I don't see a future at this time for the language. There also seems to be a bitter and angry side to many of the users and they aren't that friendly to new users of the language that ask stupid questions.

Title: Re: Alternative O2 Builder
Post by: cevpegge on September 28, 2010, 07:37:14 AM

The dark areas of C, for me are linking libraries and building packages. There are some unfamiliar mumbojumbo invocations involved, but the language will port fairly cleanly, once the necessary string support functions are in place.

Charles
Title: Re: Alternative O2 Builder
Post by: jcfuller on September 28, 2010, 08:03:36 AM
Charles,
  With your assembler expertise I would recommend giving JWasm a look-see. No stupid {};crap!!

James
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 28, 2010, 09:09:40 AM

Thanks James,

I'm just looking at what's avaiable now, but I think the curly braces and the semicolons have us in an inescapable grip.

The Pelles C compiler looks attractive and coding to the C99 standard should ensure good portability. I can keep library dependency down to a minimum.

Charles
Title: Re: Alternative O2 Builder
Post by: efgee on September 28, 2010, 10:35:48 AM
Pelles C restricts OxygenBasic to Windows, but on the other side there is a ARM version too (WinCE).

---

When I fumble around with C I find myself battling compiler issues rather than language issues.

Everybody talks about how portable C is... well the language itself is, but the compilers are not.

If one keeps the code clean from compiler specific nonsense then code can be ported to different compilers/OS.

---

Charles, have you looked into tcc?
It has Windows/Linux/32bit/64bit/ARM on a small footprint.

Take care
efgee
Title: Re: Alternative O2 Builder
Post by: jcfuller on September 28, 2010, 10:40:19 AM
efgee,
  I agree and that's why I adopted to go with MinGwTD for use with Bcx on Windows. It's the closest to the Linux versions.

http://tdm-gcc.tdragon.net/

James
 
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 28, 2010, 01:37:47 PM

The main problem with these is the inline-Assembler facilities. TCC demands the use of AT&T notation and although GCC will accept Intel notation, the examples I saw were very clumpy - each instruction had to be quoted and end with \n.

This is worrying:
http://www.reversing.be/article.php?story=20051203194931893

Pelles has it absolutely right and supports ARM Assembly code as well. It is all neatly contained between curly braces without extraneous syntax.

But I have not ruled out Oxygen self-compilation if C presents too many obstacles :)

Charles
Title: Re: Alternative O2 Builder
Post by: jcfuller on September 28, 2010, 03:14:56 PM
Charles,
  I had forgotten how ugly intel gcc asm is. Just bite the bullet and go with JWasm. With all the high level syntax and the superb macro facilities it's very basic - like. There is even a masmbasic which is just macros. JWasm is 100% compatible with Masm. It's closer to basic that "c" will ever be.

No Linux if you go with Pelles.

James
Title: Re: Alternative O2 Builder
Post by: JRS on September 28, 2010, 03:52:08 PM
Quote
But I have not ruled out Oxygen self-compilation if C presents too many obstacles ...

You would be in control of your destiny and your code with a potential of establishing new standards in Basic.

SB is only an interpreter but the ANSI C route Peter Verhas chose from the beginning has held up for over 10 years with no obsolescence in sight for the near future.

It might be worth porting your FreeBASIC code to BCX and tweak the C after the translation if needed. Nothing ugly about BCX ASM inclusion. This is the fastest way out the FreeBASIC door.

It would be great if Armando and James could help Charles out with a port of O2 to BCX.
Title: Re: Alternative O2 Builder
Post by: jcfuller on September 28, 2010, 05:04:28 PM
Quote
But I have not ruled out Oxygen self-compilation if C presents too many obstacles ...


 Nothing ugly about BCX ASM inclusion.



John,
  I use g++ with Bcx and it needs g++ asm -> UGLY
The help file I believe shows asm if code is compiled with Pelles.

James
Title: Re: Alternative O2 Builder
Post by: JRS on September 28, 2010, 05:11:24 PM
Quote
The help file I believe shows asm if code is compiled with Pelles.

Okay, now I'm on the same page.
Title: Re: Alternative O2 Builder
Post by: efgee on September 28, 2010, 06:48:09 PM
I actually don't see the hurry to leave FreeBasic at all. It served Charles well and it can do so for years to come.

On the other hand Oxygen self-compilation would be a very good move, as you could see for yourself how solid the compiler is  ;D

efgee
Title: Re: Alternative O2 Builder
Post by: JRS on September 28, 2010, 08:26:53 PM
I added a Poll to the thread asking the question ...  (see top of page to vote)

What is your language preference to build OxygenBasic?
Title: Re: Alternative O2 Builder
Post by: efgee on September 28, 2010, 08:43:16 PM
I added a Poll to the thread asking the question ...  (see top of page to vote)

What is your language preference to build OxygenBasic?

But in the end it's his baby - his choice.
Title: Re: Alternative O2 Builder
Post by: JRS on September 28, 2010, 11:09:32 PM
True but OxygenBasic is the ASM code that makes the Basic what it is. FreeBASIC is a shell and support framework to generate an executable.

I don't think Charles would mind hearing from the troops what weapon of choice is preferred.
Title: Re: Alternative O2 Builder
Post by: JRS on September 29, 2010, 01:39:46 AM
Does Oxygen Basic generate an assembly source which is compile by the GAS assembler?

If this is true, just about anything could be used to do what FreeBASIC provides. It sounds to me that string support is the holy grail that keeps Charles from letting go of his comfortable environment. Reminds me of the PowerBASIC author that was generating 32 bit code with a 16 bit compiler.
Title: Re: Alternative O2 Builder
Post by: jcfuller on September 29, 2010, 03:10:45 AM
I actually don't see the hurry to leave FreeBasic at all. It served Charles well and it can do so for years to come.

efgee

Not so. It will not compile with the latest version.

James
Title: Re: Alternative O2 Builder
Post by: jcfuller on September 29, 2010, 03:26:23 AM
There might be an option to use Bcx(g++) without the ugly asm syntax but a quick look at the Fb Oxygen.dll dictates a bit of restructure. You won't be able to place an asm{} block in the middle of a procedure or in the Bcx code.The asm code could be written in jwasm and compiled to either object modules or a static library and then linked to the main code(dll) written in Bcx (or straight c++).Using g++ STL you have your dynamic strings too.
Just more food for thought.

James
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 29, 2010, 02:36:09 PM
Well I have cast my vote in favour of Oxygen self-compile :)

I've worked out how to do it incrementally, so there are no large chasms to leap over during development. The biggest concern is being able to test each component in a live system and minimise the chances of bugs getting buried in the new software.

One further advantage is that the self-compiled product could safely expose many of its internal parsing functions as a library. This would be very useful for many data processing applications. The compatibility between  the compiler internals and a program that needs to access then is assured.

Charles
Title: Re: Alternative O2 Builder
Post by: efgee on September 29, 2010, 04:19:24 PM
Charles,
on top of what you mentioned:

the runtime could then also be a part of the exe and not in an external dll.
(and only the parts that are really needed...)

8)
Title: Re: Alternative O2 Builder
Post by: cevpegge on September 29, 2010, 05:05:10 PM

Yes the runtime will be small and in the form of an include file of source code. The main advantage of having Oxygen as a DLL dependency is to have JIT compiler services on tap. For instance if you wanted to have an Eval  function available in your application. This is not usually possible in statically compiled code.

Charles