Understanding Address Spaces and the 4GB Limit

mikeblas

[H]ard|DCer of the Month - May 2006
Joined
Jun 26, 2004
Messages
12,777
Understanding Address Spaces and the 4GB Limit

When the first 32-bit versions of Windows were released back in the early 90's, their large address space was touted as a great feature. At the time, when most machines had less than eight megabytes of memory, a four gigabyte address space seemed incredibly large.

Over the years, advances in semiconductor fabrication technology have made gigabyte memory parts not only a reality, but common place. Desktop machines with a gigabyte of memory common, and boxes configured with four gigabytes of physical memory are not unheard of. These machines are pushing the limits of the system, then, and causing interesting effects of the limitation to be shown.

This article examines the four gigabyte memory addressing limit of Windows 32 operating systems.

Memory Addresses

Memory is the most readily accessible large-volume storage available to the processor. Bytes of data readily accessible to the processor are identified by a memory address, which by convention starts at zero and ranges to the upper limit addressable by the processor. 32-bit processors typically use memory addresses that are 32 bits wide. The 32-bit wide address allows the processor to address 2^32 bytes of memory, which is exactly 4,294,967,296 bytes, or four gigabytes. This is the direct source of the four-gigabyte memory addressing limitation.

Further, all current desktop and server versions of Windows (that is, Windows XP Home, Windows XP Pro, Windows XP Media Center Edition, and all non-embedded editions of Windows 2003 Server, all and editions of Windows 2000) support addressing four gigabytes of physical memory.

Yet accessing four gigabytes of physical memory isn't possible on all machines. Why is that?

While there's no limit in Windows beyond the four-gigabyte limit of the address space, the address space might contain things other than memory. Hardware designers frequently use a technique called memory-mapped I/O. Memory-mapped I/O places memory which is visible to a peripheral on the system bus, in the address space of the processor. To communicate with the device, the processor simply writes data into the range of memory the device has mapped into its address space. This mechanism is significantly faster than using a specialized bus for I/O access, and provides additional benefits the software and hardware designers alike.

Windows will work with the hardware to map as much physical storage memory as possible into the spaces around the device-provided memory mapped features. While the total memory available to the operating system will be less than four gigabytes, the system certainly is capable of accessing four gigabytes of memory space. For applications where memory is important, care must be exercised in selecting and configuring peripherals in order to assure the maximum amount of memory is available.

One easy way to see what memory mapping is happening is to use the Device Manager on your system and have it show what mapping is occurring. On a machine with three gigabytes of memory,

  1. Right-click on the "My Computer" icon
  2. Select "Properties" from the Context menu
  3. In the resulting "System Properties" dialog, activate the "Hardware" tab
  4. Click on the "Device Manager" button
  5. In the "View" menu, select the "Resources by Connection" command
  6. select the "Memory" node in the tree
  7. Press the "*" key on the numeric keypad to expand all the nodes in the memory tree

On a desktop machine with three gigabytes of memory, the memory map provided by the Device Manager looks like this:
MemoryMap_Enduro.jpg


When this machine, named ENDURO, boots, it counts 3072 megabytes of memory in the BIOS, and shows 3.0 GB in the general properties for the computer. But using the "Performance" tab of Task Manager, this machine reports 3,144,940 kilobytes of available physical memory. This isn't quite enough, as 3072 megabytes of memory is 3,145,728 kilobytes of memory. Where did the missing 788 kilobytes of memory go?

Addresses shown in the Device Manager window are in hexadecimal. We don't care about many of them, but one of the important values we're interested in is 0xC0000000, since that corresponds to three gigabytes. We can see that Windows has allocated addresses 0x00100000 through 0xBFFFFFFF to the system board, and that totals 3,144,704 kilobytes of memory. There's also the range 0x000E0000 through 0x000FFFFF, which is another 128 kilobytes of memory. That brings us within 788 kilobytes of our expected total.

One hole of memory is evident at 0x000A0000 through 0x000BFFFF; this memory is mapped to the display adapter for its text mode and lower graphics modes. These addresses have been used by the dispaly adapter by convention since the original IBM PC in the early eighties, and are retained for compatibility. Adding the memory around it brings us within a few kilobytes of our physical memory goal.

The hole the graphics adapter causes in physical memory is called its aperture. This value is settable in the system BIOS, at the cost of being able to access memory on the video card directly and rapidly. With very large buffers on video cards, and the advent of SLI systems, the memory aperture taken out of the system address space can be huge, and reduces the usable general-purpose memory accessible by the system.

This machine was booted with the BIOS set to a 64MB AGP aperture:

MemoryMap_Burner64.jpg


Rebooting the machine and setting the aperture to 256MB makes a noticable difference in the memory map:

MemoryMap_Burner256.jpg


While the card still reserves 256 megs for its video memory, the aperture now takes much more room from the system address space. In the first screen shot, we see the aperture is from 0xF8000000 to 0xFBFFFFFF, which is 64 megabytes. On the second screen shot, the aperture extends from 0xE0000000 to 0xEFFFFFFF, which is 256 megabytes.

Had this system four gigs of memory, the system would show at most 3.94 gigabytes of physical memory available after it booted with the 64 megabyte aperture. With the 256 megabyte aperture, the maximum is 3.75 gigabytes. The true maximum is substantially less, since the 256 megabyte card memory also comes from the physical address space, as well as memory for other devices on the system.

Note that all the addresses in this section of the document are physical. The OS is free to map a physical address range to a logical address range in any way it sees fit, and that's important for understanding the next section, where all the addresses described are logical.

The Process Address Space

When a program loads under Windows, it is managed as a process. One of the most important features of a process is its virtual address space, which holds all of the code and data the process will directly access.

In the 32-bit versions of Windows I enumerated above, processes are given a default memory limit of at least two gigabytes. This limit means has the logical addresses that any process touches varying between 0x00000000 and 0x7FFFFFFF. These numbers are interesting because they're the full range of non-negative integers that can fit in 32 bits. Considered as a signed integer, the quantity 0x80000000 is -2147483648,
and the quantity 0xFFFFFFFF is considered -1.

Considering pointers to memory as signed values, even inadvertently, can have disastrous effects. Doing math between two pointers where one is unexpectedly negative results in wildly different results than correctly doing the signed math.

To defend the system against poorly written applications and device drivers, Windows allocates memory for applications from 0x00000000 to 0x7FFFFFFF. It maps the operating system itself in the range 0x80000000 to 0xFFFFFFF; as much as it can, at any rate. This gives user processes two gigabytes of virtual address space, and gives the system two gigabytes of virtual address space.

An application can declare that it has been tested and works correctly with large addresses. Many applications, from Microsoft SQL Server and all its tools, to Adobe Photoshop and Illustrator, have a bit set in their executable image which tells the operating system that the program can load in high address ranges, and be exposed to high memory values—that it doesn't do incorrect signed arithmetic on pointers, and won't get sick if it sees such values.

On a system with more than two gigabytes of memory, the system can be booted with the /3GB option to expose more than two gigabytes of address space to each process that's marked with the large-address aware bit in its executable image. User processes receive three gigabytes of space, then, while the system can use a single gigabyte of address space.

The benefit of using this extra space can be mitigated by the fact that the system has to compress itself into the remaining one gigabyte of address space. Memory-mapped I/O and the video aperture space also come from that single gigabyte. Raymond Chen's excellent Old New Thing blog contains a couple of articles about this tradeoff.


Workarounds and Enhancements

All Pentium-class processors support at least 32-bits of addressing space. Some support more, using a 36-bit address bus. This extension is called PAE, which stands for Physical Address Extensions. A machine with such a processor installed on a motherboard that has a chipset supporting the extensions can use 36 (or more!) bits to address memory. These extra bits are controllable by special registers in the processor and the chipset, and allow the mapping of additional memory into the virtual address space of a process.

Common mostly on server and advanced workstation machines, PAE provides the ability to break through some interesting barriers.

PAE is also used to enable a very interesting feature on some of the newsest processors: data execution protection, or DEP. Data Execution Protection lets the operating system mark ranges of memory as data, which is strictly not executable. Ranges not marked as data are assumed to contain code, which is freely executable. DEP help prevent data from entering the machine and executing as code through buffer overruns, for example.

dep.jpg


When DEP is enabled, the system may use /PAE to relocate device drivers above the four gigabyte limit. This enables a simpler tagging mechanism for the data-marked memory to work with PAE, relieving a little bit of address space pressure in the problem. During the development and testing of SP2, it was found that many device drivers didn't work correctly with /PAE enabled when they had to work with DMA transfers that were targeting the address space above four gigs of memory. As such, Windows SP2 turns off PAE and forces the drivers (and possibly their associated mapped regions of memory) into the four gigabyte address space, causing the system to see less available physical memory.

This issue is described in a Microsoft KB article.

Does Win64 Help?

Some people believe that Win64 is of no use to desktop users unless they need to manage monumental amounts of data. This might be true, though the definition of "monumental" might be smaller than we think. With four gigabytes of physical memory, a 32-bit system's physical memory space is rather crowded. Win64 lets the 64-bit processors spread their physical memory out through a much larger logical memory space, resulting in less unusable ranges and fewer conflicts.

Hopefully, vendors will see the light soon and begin focusing on driver support (and stability!) for Win64 operating systems in the desktop and workstation markets.


Disclaimer
This posting is provided "AS IS" with no warranties and confers no rights. I do not speak on behalf of Microsoft. This post contains material protected by copyright.

References


Version 1.0, 25 March, 2006
  • First version.
Version 1.01, 19 July, 2006
  • Added some links, fixed some others
Version 1.02, 6 December, 2006
  • Added another reference to RayChen's blog
Version 1.03, 19 December, 2006
  • Added Channel9 reference.
Version 1.04, 18 February, 2007
  • Fix typos.
Version 1.05, 24 August, 2009
  • Fix more typos.
 
Last edited:
Thanks Mike, this was really informative. :) I already new a fair amount about the 4GB limit, didn't know the arpeture fact though. Also, I remeber hearing that there was a problem with the PCI-Express video cards and Memory on 32-bit systems posted a while back (users would install 3GB of memory, but only 2.5 was being reported). If you have time, could you research that subject and include it in your post?
 
Never heard of that PCI Express issue before; you'd have to give me a little more to go on. I've only got one PCI Express machine, and it reports 3.0 gigs of memory with 3.0 gigs of memory installed, so I can't reproduce it myself.
 
Ill try to search the forums, because I know I heard it some where here. Might have been a hardware specific problem then.
Took a look around, seems as though the barrier isn't at 3GB, but 4GB. It may simply have to do with the Apreture you mentioned though, not PCI-Express itself. Some people on various forums were reporting that only 3.5GB (sometimes 3.3GB) was showing up in the system properties, instead of there full 4GB they had installed.
 
on the A8N SLi Premium , it was a hardware limitation fixed with recent bios releases

nice article , thanks

 
Xipher said:
Some people on various forums were reporting that only 3.5GB (sometimes 3.3GB) was showing up in the system properties, instead of there full 4GB they had installed.

If you install SLI, do you end up using two apertures? Reserving twice the space for video memory? If so, it'll add up quick: two 256 meg cards would take 512 megs out of the address space, plus the aperture space. So that's the 0.50 to 0.75 gigs, right there.

There's plenty of other explanations, though, so it's hard to diagnose without any specifics. I'd just be guessing blindly.

ThreeDee said:
on the A8N SLi Premium , it was a hardware limitation fixed with recent bios releases

nice article , thanks

I'm glad you like it. What is the "it" you're referring to?
 
nice post. did you have to do this for another reason and post it here, or did you just decide to write it for us? either way, thanks.
 
Very informative Mike. I appreciate you sharing this information and taking the time to type it all out.
 
santaliqueur said:
nice post. did you have to do this for another reason and post it here, or did you just decide to write it for us? either way, thanks.
I wrote it to post here. I see many people who insist that "XP wastes memory", when it doesn't ; I grew weary of responding to them (and getting flamed for taking the host thread off topic), so I put together this essay with the expanation. It's the fourth or fifth such essay I've written and posted here, if I'm counting right.

Thank you for your kind feedback.
 
I appreciate you sharing this information and taking the time to type it all out.
Yea, thanks. I don't think I would have had the patience to do it.
 
Can we get a sticky on this?

mikeblas, the link for "RAM, Virtual Memory, Pagefile and all that stuff" is broken.

edit: odoe, thanks for the sticky.
 
I've updated this article to add a couple interesting links, and fix one that I didn't notice being busted.
 
That's a good read, thanks mikeblas. I was just given a lecture in another thread about how XP32 had a 2 GB limit. :( Have yourself a nice cold Moretti for this one.
 
Xipher said:
Thanks Mike, this was really informative. :) I already new a fair amount about the 4GB limit, didn't know the arpeture fact though. Also, I remeber hearing that there was a problem with the PCI-Express video cards and Memory on 32-bit systems posted a while back (users would install 3GB of memory, but only 2.5 was being reported). If you have time, could you research that subject and include it in your post?


Yeah, I had heard this too, infact on my PC (with 4GB installed, and SLI cards) when I go into my BIOS and "tweek" the PCI-E time, I can get the system to see 2.75GB, but at stock it will only see 2.5GB. I think it's time to get XP 64 on my system.


-Proxy
 
Wow.

That is an extremely interesting read. Very informative, especially the PCI-E/SLI information.

I'll definitely be using a 64-bit OS when I start to push the memory envelope though, I'm only 1/2 way there right now.
 
Thank you for your kind words.

I need to write this up to fit into the article, but I also know one of the guys at work is preparing a KB article to cover the issue. Thing is, X64 only helps if you have a motherboard (eg, a chipset) that supports remapping and a BIOS that allows you to turn it on. All the AMD processors with their on-chip memory controller support it; only certain Intel-compatible chipsets support it.
 
Thanks! I'm glad ya'll find this useful. Maybe I can update it soon with notes about Vista...
 
It already is in the sticky, and has been for a very long time.
 
It's in the "FAQ LINKS - This is where the forum FAQs are." sticky.
 
Now i have something to read for my Saturday afternoon procrastination time :)

Thanks Mike!
 
I have a question for you Mike. And please don't take this as me trying to surmount your superiority, as you are obviously extremely knowledgeable in this topic. You use the word "arperture" a few times in your article. You use it to mean a hole, which is exactly what aperture usually describes. Is this a specialized spelling of aperture for this particular topic?
 
Nah, it's just a really consistent typo. I'll see if I can fix it later this afternoon; thanks for letting me know!

LATER: Fixed. Let me know if you notice any others.
 
There are ways that a 32bit OS can address alot more RAM then the 4Gbyte
it uses a similar method that OS's use to get around a fuckup by BIOS writers

There are two limits on the amount of space that can be address on harddrives (via the CHS method). The first is after 1024sectors (~8gig) and the other is around 137Gig
This is purly downto BIOS missing off one bit (as agreed between HD manu and BIOS writers) in one of the CHS addressing.

Problem?
Originally no OS could be installed or be used on large disks
Workaround?
as long as a part of the OS (bootloader, 2nd stage bootloader, kernel) was below the 8gig limit the OS could provide paging to re-address the full size of the HD.

ultimate fix was when addressing went to LBA

its not pretty but there is a way to address more then 4Gig of RAM (32bit servers are now!). Not to say that 64bit isn't better, but just to say it is possible
 
There are ways that a 32bit OS can address alot more RAM then the 4Gbyte
it uses a similar method that OS's use to get around a fuckup by BIOS writers
Can you be more specific?

its not pretty but there is a way to address more then 4Gig of RAM (32bit servers are now!).
I think you're referring to Physical Addressing Extensions, which are discussed above.
 
Nah, it's just a really consistent typo. I'll see if I can fix it later this afternoon; thanks for letting me know!

LATER: Fixed. Let me know if you notice any others.

oooo... i've successfully contributed to the faq :p
 
I've taken another run through this essay to fix some more typos.
 
Hi 2007! :D

OT:
You know what makes this forum so great? The color scheme. Reading the white text on that grayish background's SO relaxing compared to the crap on Anandtech. Those forums are TERRIBLE! Anyway, that's probably why I've got this and many other H info threads bookmarked.
 
One thing worth noting is that even with PAE, Windows Client (XP, Vista, 7) is still (artificially) limited to 4GB of total address space.

Windows Server 32bit, on the other hand works slightly differently. For example Windows server 2003 Std supports a max of 4GB of ram, but it can 'see' all 4GB in 32bit using PAE. Windows XP, even with PAE, wont ever get all 4GB. Windows server Enterprise and whatnot can support even more than 4GB in 32bit using PAE.
 
Back
Top