Blogs‎ > ‎Tech Stuff‎ > ‎

2015.06.22 GFA BASIC 32 mAlloc() Vs the Super Strings

posted Jun 21, 2015, 7:20 PM by Troy Cheek   [ updated Jun 21, 2015, 7:21 PM ]
Because I'm old and set in my ways and too close to death to learn a new programming language, GFA BASIC is the hammer I use on all my nails.  I currently use GFA BASIC 32.  The short version is that GFA BASIC 32 is a modern BASIC which looks and acts more like C than any BASIC someone my age grew up on.  Luckily, while I was in college I took a few C courses.  I also followed the evolution of GFA BASIC, from it's humble beginnings as Turbo-Basic XL for the Atari 8-bit line of computers, to GFA BASIC for the Atari ST line, to GFA BASIC 32 for Windows which was first released back about the time Windows 95 seemed like a good idea.  Development stopped about the time Windows XP came out, but with some minor hacks it still works on my Windows 7 64-bit Home Premium Bacon Ranch Edition computer.  There were also versions for the Amiga, MS-DOS, and 16-bit Windows.  All these were created by German programmer Frank Ostrowski in 1985, and it all started because he was between jobs after leaving the military.

But enough history.  Today, we're talking about memory allocation.  For speed and ease of use, if you want to work on a really big data set of some kind, it's easier to load it into the computer's RAM memory first.  To do that, you need to allocate a block of memory by requesting it from the operating system.  In GFA BASIC, the command for memory allocation is mAlloc(), which seems obvious in retrospect.  From the GFA BASIC 32 help file:

mAlloc(n) allocates memory to the program. n specifies the amount of memory to reserve in bytes. If n is -1, mAlloc(-1) returns the size of the largest available block of free memory. The reserved memory block is not initialized.

Actually, there are several negative values you can use for n, which will return things like total free memory, virtual memory (Windows page file), percentage of memory used, etc.  Unfortunately, none of these values are correct if you're using a 64-bit version of Windows.  I guess the Windows memory handler is different or something.

But mAlloc(n) still works with positive values of n.  If you want to read a file that is 1 MB long into memory, you just use addr%=mAlloc(1024*1024), and addr% is a 32-bit signed integer containing the memory address of the start of the memory block you just reserved.  If there's a problem, let's say you don't currently have a full 1 MB of free RAM in one contiguous block anywhere in your system, then addr% will be set to -1.  Always check to make sure that mAlloc() didn't return a -1 before you go on with your program.

Now, 1 MB was a pretty big file, back when 1.44 MB floppy disks were the norm and a computer with 16 MB of RAM was considered overkill.  Today, however, it's not unheard of to work with a video file that's 4 GB or more.  No problem.  My computer as 16 GB of RAM with over 9 GB free at the moment  We'll just try addr%=mAlloc(1024*1024*1024*4).  Hey, addr%<>-1, so we must have our 4 GB of memory reserved, right?  Unfortunately, wrong.

GFA BASIC 32 is, as the name suggests, a 32-bit program.  Even running on a 64-bit operating system, it's still limited to accessing memory addresses in a 32-bit range.  32 bits translates to 2^32 values, or 4096 MB, or 4 GB.  However, we're working with signed integers here, meaning that they can contain both positive and negative numbers.  It takes 1 bit to show positive or negative, so we're left with plus or minus 31 bits which translates to -2^31 to +2^31, or -2048 MB to +2048 MB, or -2 GB to +2 GB.  So, the largest possible memory block we can reserve is 2 GB.

But didn't we just successfully allocate 4 GB?  Well, no.  While mAlloc() didn't return the -1 error code, it did return 0.  I'm not sure what that means.  It's not in the help file, but it is a sign of failure of some kind.  I think, theoretically, a successful memory allocation could return an address of 0, just meaning that the memory block starts at the very first memory location (computers like to start counting at 0) this program has access to, but in practice it's an error.  You can't request a memory block bigger than what can be expressed as a 32-bit signed integer.

Now, you might be tempted to try mAlloc(1024*1024*2) to get a block of memory 2 GB long, but 32-bit integers can't actually express positive 2 GB because you have to have room for the 0 in the middle (remember, computers like to start counting a 0), so actually the biggest number you can use is 1024*1024*2-1 or 2 GB - 1.

But don't try that, either, because you'll get a -1.  You see, while GFA BASIC 32 programs can access 2 GB or so of memory, they reserve some memory for themselves for stacks and heaps and variables and file buffers and housekeeping.  To make a long story short (too late!), the biggest block of memory you'll ever successfully allocate is a bit over 1.25 GB.  I limit my memory blocks to 1 GB even, just to be sure.

However, I grew up not using mAlloc() for memory allocation.  You see, there's a corresponding command mFree() which tells the memory manager that you're finished working with this memory block and it's available for use by other programs.  Back in the Atari ST days, the memory manager wasn't very smart.  If a GFA BASIC program allocated memory, it stayed allocated until it was set free.  If the program crashed or abnormally ended or was just programmed incorrectly, it would not free the memory.  It then became impossible to free the memory because the location of the block was lost when the program ended and all it's variables were cleared, namely the addr% or whatever variable.  The block would stay allocated until the Atari ST was rebooted.  Since I was writing programs that would crash or at least end prematurely, they wouldn't always clean up after themselves, so I'd eventually run out of free memory and have to reboot my ST.  Luckily, modern versions of Windows have better memory management, and memory blocks are almost always freed when the program ends.  At worst, I've had to exit and restart the GFA BASIC editor.

So, if you're not using mAlloc(), how do you allocate memory blocks?

Enter the Super Strings!  (fanfare!)

Back in the Atari 8-bit days, Atari BASIC (and, of course, Turbo-Basic XL) handled strings differently than other BASICs of the day.  For most home computer BASICs, string variables (a$ = "Hi there!") were limited to a length of 255 or so characters (bytes).  That's perfectly acceptable if you're using string variables to hold a person's name or address or a single line of text.  However, if you need a larger data structure, 255 bytes is a bit small.  Atari BASIC allowed you to have "super" strings.  As it was described at the time, a single string variable could be as long as available memory.  Of course, back then home computers had 16 KB of RAM, but that's still pretty long for a string.  Later Atari computers had 32 or 48 or even 64 KB, and we learned that Atari's super strings were limited to 32,767 bytes or 32 KB - 1 (remember, you have to leave room for the zero).  Still, that's pretty darn big.  Eventually, pretty much all BASICs allowed strings this big, but Atari was the first in 1979!

Why use strings?  Well, back then we didn't have mAlloc() or similar memory management tools.  Strings were a good way to allocate memory without having to pull out the memory map (seriously, we had a book describing all 65536 possible memory locations) and find an open area (and hope that some other program didn't also want to put data there).  You could use the instr command to rapidly search for text or byte sequences.  You could put sprite (primitive graphic object) information in a string and move it around the screen by moving bytes within the string using very fast BASIC commands.  Short machine language routines could be placed in string variables.  Screen graphics could be buffered by putting them in string variables.  Super strings were great!

GFA BASIC for the Atari ST continued the trend, and I remember using string variables for file buffers and the like instead of mAlloc().  I continued the practice when I started using GFA BASIC 32.  It was easier to use buf$ = space$(1024) for a short file buffer than to fiddle with mAlloc() and the like.  I got to thinking, though, what is the limit of super strings in GFA BASIC 32?

Trying to create a string variable longer than 2 GB fails, of course.  I tried various other big numbers.  They resulted in messages like Out of String Memory, Access Violation, etc.  To make a long story short, the longest super string in GFA BASIC 32 is 256 MB - 1.  Now, that's an odd number, and maybe even a prime number, so I generally limit my super strings to 128 MB to make things even.  However, you can create multiple string variables.  You can create ten 128 MB strings before you run out of memory, for a combined total of 1.25 GB.  This is the same limit as we discovered for mAlloc().

Of course, strings aren't the only variables.  You can create an array of 32-bit integers, or 4 bytes per array element.  Dim a%(1024*1024/4) would give you 1 MB of memory.  I'll save you the trouble of trying it out.  The largest amount of memory you'll be able to reserve this way is about 1.25 GB.

The main difference between using variables and mAlloc() is that if you try to create a variable too big for available memory, your program stops with an error message.  If you try to a create a memory block with mAlloc() that's too big for available memory, you just get an error code, which you can check for and program around.  What follows is a simple routine for a large memory block that fits within memory constraints:

dim memloc%, memlen% as int32
memlen% = 1024 * 1024
memloc% = mAlloc(memlen%)
while memloc%<1
    memlen% = memlen% / 2
    memloc% = mAlloc(memlen%)

This starts by trying to allocate 1 GB.  If that fails, it tries 512 MB, then 256 MB, 128 MB, and so on.  I like programming with round numbers.  Very quickly, the largest block of free memory will be determined, located at memloc% and memlen% bytes long.  Remember to mFree(memloc%) when you're finished!