Use more RAM for variable, array

[Victorio]

Hi,
I have problem with memory used my application.
For boost speed processing large text files I "load" them to arrays. When this files are large about 450MB and more, my program crashed with MOM error .
When I look to taskmanager, I see, limit for memory exe file is about 1.1 - 1.2GB.
However for 32bit Xbase++ is limit 3GB, my app crashed when exceed about 1GB .
Ok, I modified my app to break large files to smaller parts and work good.

But I am interested if is some solution how use more memory than 1GB , some system, when array store to more free memory. Some as RAM disk.
Is it possible without create virtual RAM disk in OS ? Best will be programmatically from Xbase++.

The reason is faster processing than load from HDD.
Also however I have SSD PCIe M2 disk, still it is slower than store in array.
Note : I must use txt files because I need full text search in its content.

[Auge_Ohr]

hi,

However for 32bit Xbase++ is limit 3GB, my app crashed when exceed about 1GB .

as i can say : 32 bit App under 64 Bit OS can access 2 GB, NOT 3 GB !

try this

Code: Select all

#IFDEF __XPP__
#ELSE
   REQUEST HB_GT_WIN_DEFAULT           // harbour Console
#ENDIF
PROCEDURE MAIN
LOCAL i
LOCAL aString := {}
LOCAL bOldError := ERRORBLOCK( { | e | BREAK( e ) } )

   BEGIN SEQUENCE
   FOR i := 1 TO 1600
      ? i
      AADD(aString, REPLICATE("A",1024*10000))
   NEXT
   END SEQUENCE
   ERRORBLOCK( bOldError )
WAIT
RETURN

under harbour 64 Bit i can use 16 GB RAM ( - 2.8 GB OS )

[Victorio]

Hi Jimmy,
Thansk for advice

I test it, counting stops at 130, this says max memory for array is 1.33 GB.
Ok, also at this limit my app crashed.

You say Harbour 64bit can use 16GB -2.8, this is interesting. I do not know Harbour, I know exist but not that is in 64 bit version.
So this can be problem, when client run app in Harbour on 32bit PC, but I can test it.

This is good advice from you, but I´m thinking how realise it . At this moment I must create some external utility in Harbour, and use it for processing this large files .
Is it good idea ?

Viktor

[Koverhage]

maybe it is not a memory problem rather than a handle problem.
Ask Alaska for a description of the CDF_* settings

[Auge_Ohr]

Hi Klaus,

maybe it is not a memory problem rather than a handle problem.
Ask Alaska for a description of the CDF_* settings

have you run my Demo Code ?
you can´t get > 2 GB using 32 Bit App (without Tricks)

under 32 Bit OS hole 4 GB is "split" into 2 GB for System (or Kernel Space) and 2 GB for the Application (or User Space)

---

there is a /3GB or /userva in BOOT.INI but Application need /LARGEADDRESSAWARE flag for Compiler
other is /PAE (since PentiumPro) used by Server 2003 Data Centre Edition which can use 64 GB with 32 Bit OS

i have used those "Tricks" under Windows XP 32 Bit but it seem not to work with Windows 10 32 Bit or "SubSystem" in 64 Bit OS
but now there is no need for me to use 32 Bit any more while i can use 64 Bit Compiler to build 64 Bit Apps using xBase Syntax

btw. "locking" under Cl*pper / Xbase++ is Offset 2 GB +1 for 32 Bit OS.
under 64 Bit i can increase Offset so DBF have 1 TB +1 ... but it is not compatible !

[Koverhage]

@Jimmy,
please read the post correctly!
The file from victorio has a size about 450MB.
The Locking mode has nothing to do with arrays and text files!
The MOM Error: There are settings in the cdf.ch to deal with this.
You should know about that, you make a post of this here: https://www.xbaseforum.de/viewtopic.php ... 81#p126881

@Victorio
How many elements has the (should have) the array?

[Victorio]

Hi Klaus,

When my app processing data from text file, read his content to array, file can have about 500MB and more, but not only this file is read in memory, also other variables occupied memory.
I test it and program crashed about 1.1 GB and more .
because text file can have about 8.000.000 rows, it is also number of array element.
Now I divide file to smaller parts, text file read by 100MB blocks, and also number of elements I controlling and when number of elements exceed about 100.000 save content array to output file,
and release array and start reading other part. (arraytxt:={})
After many modifications now running process in my app and after several hours I will see, if problem solved or crash again. Processing need about 20 hours !, because it is difficult test it many times.

I read about handles , as you wrote about PDR, and it is interesting, that every element of array need handle. This can be my problem , when 2000000 elements is limit.

Note : My app need work on Windows 2012 server with 12 CPU cores, but I developed and test it on Windows 10 64bit. Sometimes on my W10 PC process end correctly, but on Windows 2012 crashed.

I will write info, how end processing now after some other modifications.

[Auge_Ohr]

hi Klaus,
it doesn't matter how many "Handle", which are limited by Alaska, while it still use (System) RAM which IS limited by Windows OS. so it will NOT help to increase "Handle" to break 2GB (Apps) RAM.

---

@Victorio
8.000.000 rows is simple "to much" for a 32 Bit OS
if you need 20 hours (!) for that work you should think about other Way like SQL to import hole Text File and use "build-in" Full-Text-Search.

[Koverhage]

Victorio,
each row need a handle, so you way is the only way you can handle it.

[skiman]

Victorio,

I don't know what you really need, but 20 hours of processing on a 500 Mb. file looks very long. I have text search in files, and it is very fast.

I just did a test in a 338 Mb. file. I searched for a word, and I found 50 results in about 6 seconds.

Just tell us what you really need, show some code, and maybe someone can help you.