Use more RAM for variable, array

[unixkd]

Hi Skiman

Can you favor us with the function you used to do the text search

Joe

[Victorio]

Hi skiman,

My app search in many text files, not only one ,standard is about 60000 by number of years of history when processing 10 years. Every year have 6000 files, and in database files I have 27 years !!!. but some from it have more than 300-400MB, but not all. Some has several kB.
Not only search , when searching, from text removing diacritics and comparing with one to x keys.
And many other operations, as creating XLS outputs, DBF outputs, etc.

Problem is that some of max size files fills RAM.

At this moment I have mechanism that divide large files and processing by parts, and after today test looks will work good.
So always some problem I have when generate XLSX from DBF because DBF has also millions rows, and also need divide to smaller parts to eliminate memory exhauting.

I did some modifications and must try again processing but I must wait to "computer time" on server where other processes is running.

[skiman]

Hi Joe,

The function is integrated in my app, and will not work as a separate function.

It is using the fopen and fread functions, which are very fast. This way you don't need to load it in memory, and size doesn't matter.

I open the file with fOpen, and I determine the lenght.
nBlock := 1024
nHandle := fopen(cFile, FO_READ + FO_SHARED)
nLen := fseek(nHandle, 0, FS_END)
nPages := int(nLen/nBlock) - 1
fseek(nHandle,0)

With a for next loop, the search is done.
FOR X := 1 to nPages
fread(nHandle, @cBuffer, nBlock)
if at( searchkey,cBuffer) != 0
// found
endif
next

This is the basic idea of the function.

Another possibility would be to use the DELDBE database driver to open the text files.

[Victorio]

Hi skiman,

nBlock why 1024 ? have you formatted text ?
when text is not formatted and not all rows have 1024 characters then can be problem with CRLF.

My experience is reading file by rows are quick when file is not large. Better speed is when read file or part of file to variable and then comparing strings in memory.
With this system I can read file any size but I have problem with store processed content in variable before save it to file with fwrite(). Because this I must also save file by parts.
Today after last test whis system works, only I must control how many rows I store to output string and this need some cpu time .

[Eugene Lutsenko]

I once wrote a very long time ago on CLIPPER such a program for text conversion. This program read the text by pointer and it had no limit on the file size. I made it because I needed to convert large amounts of text from DOS to Windows, which had just appeared.... It was a very popular program with us. everyone used it

Code: Select all

*** Преобразование текстового файла DOS -> WIN, Луценко Е.В., 10/05/96 06:09am


PARAMETERS File_name

scr_start=SAVESCREEN(0,0,24,79)

SHOWTIME(0,58,.T.,"rb/n")

FOR j=0 TO 24
    @j,0 SAY SPACE(80) COLOR "n/n"
NEXT

Fn = File_name
Ext = "NO"

IF .NOT. EMPTY(Fn)
   Pos_p = AT(".",Fn)
   IF Pos_p > 0
      Ext = UPPER(SUBSTR(Fn,Pos_p+1))
   ENDIF
ENDIF

IF EMPTY(Fn) .OR. .NOT. FILE(Fn) .OR. Ext = "OUT"

   SET CURSOR OFF
   SETCOLOR("g/n")
   *                    10        20        30        40        50        60        70
   *          0123456789012345678901234567890123456789012345678901234567890123456789012345678
   @ 1,0 SAY "█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█"
   @ 2,0 SAY "█      All Rights Reserved. (c) Scientific & inductrial enterprise AIDOS C°    █"
   @ 3,0 SAY "█      All Rights Reserved. (c) Евгений Луценко, 1995, 1996 7(8612) 311909     █"
   @ 4,0 SAY "█                                                                              █"
   @ 5,0 SAY "█      IF YOU WORK IN THE EDITOR DOS-TEXTS OR ENTER TEXTS WITH A SCANNER,      █"
   @ 6,0 SAY "█         AND THEN PROCESS THEM IN WINWORD, THIS PROGRAM - FOR YOU !!!         █"
   @ 7,0 SAY "█                                                                              █"
   @ 8,0 SAY "█      The given program executes following functions, which are necessary     █"
   @ 9,0 SAY "█   at transformation of text from a format DOS-TEXT in the format WinWord:    █"
   @10,0 SAY "█                                                                              █"
   @11,0 SAY "█    - Removal of symbols of the ends of lines in all cases, except the        █"
   @12,0 SAY "█      end of the paragraph;                                                   █"
   @13,0 SAY "█    - Correct transformation of the various lists of lines {-, *, Let, Num};  █"
   @14,0 SAY "█    - Removal of DOS-carries of words;                                        █"
   @15,0 SAY "█    - Removal ~clear of blanks~;                                              █"
   @16,0 SAY "█    - The exception ESC-sequences management of fonts, LEX-fonts and so on... █"
   @17,0 SAY "█                                                                              █"
   @18,0 SAY "█                                      Format of start: DW.EXE <FILE_NAME>     █"
   @19,0 SAY "█                                                                              █"
   @20,0 SAY "█   ATTENTION! A name of a source file should not have extension: ~.OUT~,      █"
   @21,0 SAY "█   as this expansion is reserved for target file !!!                          █"
   @22,0 SAY "█                                                         We wish success!     █"
   @23,0 SAY "▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀"
   *          0123456789012345678901234567890123456789012345678901234567890123456789012345678
   *                    10        20        30        40        50        60        70

   @ 5,01 SAY "      IF YOU WORK IN THE EDITOR DOS-TEXTS OR ENTER TEXTS WITH A SCANNER,      " COLOR "g+/n"
   @ 6,01 SAY "         AND THEN PROCESS THEM IN WINWORD, THIS PROGRAM - FOR YOU !!!         " COLOR "g+/n"
   @18,56 SAY "DW.EXE <FILE_NAME>"            COLOR "rg+/n"
   @20,01 SAY "   ATTENTION! A name of a source file should not have extension: ~.OUT~,      " COLOR "r+/n"
   @21,01 SAY "   as this expansion is reserved for target file !!!                          " COLOR "r+/n"
   @22,58 SAY "We wish success!"              COLOR "g+/n"

   INKEY(0)

   RESTSCREEN(0,0,24,79,scr_start)
   SHOWTIME()
   QUIT

ENDIF

*****************************************************************************

File_name = ALLTRIM(LOWER(File_name))

Pos_p = AT(".",File_name)
Fn_out = SUBSTR(File_name,1,IF(Pos_p>0,Pos_p-1,LEN(File_name)))+".out"
IF FILE(Fn_out) .AND. File_name <> Fn_out
   ERASE(Fn_out)
ENDIF

File_size = FILESIZE(File_name)

Delta = MIN(File_size,MEMORY(1) * 1024 / 8)

Lc_buf = SPACE(Delta)

CSETSAFETY(.F.)

********** Массив начальных символов списков строк

List := {}
AADD(List,"-" )
AADD(List,"*" )
AADD(List,"1" )
AADD(List,"2" )
AADD(List,"3" )
AADD(List,"4" )
AADD(List,"5" )
AADD(List,"6" )
AADD(List,"7" )
AADD(List,"8" )
AADD(List,"9" )
AADD(List,"а)")
AADD(List,"б)")
AADD(List,"в)")
AADD(List,"г)")
AADD(List,"д)")
AADD(List,"е)")
AADD(List,"ж)")
AADD(List,"з)")
AADD(List,"и)")
AADD(List,"a)")
AADD(List,"b)")
AADD(List,"c)")
AADD(List,"d)")
AADD(List,"e)")
AADD(List,"f)")
AADD(List,"g)")

Nm = CHR(13)+CHR(10)          && Конец строки (абзаца) (CrLf)

@10,30 SAY "W a i t i n g  !" COLOR "rb+*/n"

@24,0 SAY REPLICATE("█",80) COLOR "rb/n"

x = 0
y = 0

DO WHILE x < File_size .AND. LASTKEY() <> 27

   ****** Загрузка фрагмента файла
   Lc_buf = ALLTRIM(FILESTR(File_name,Delta,x))


   ****** Кодирование "красных строк" (4 или более пробелов - кр.стр.)
   Lc_buf=STRTRAN(Lc_buf,Nm+CHR(09),CHR(01))      && TAB
   Lc_buf=STRTRAN(Lc_buf,Nm+Nm,CHR(01))           && Пустая строка
   Lc_buf=STRTRAN(Lc_buf,Nm+SPACE(4),CHR(01))     && 4 или более пробелов

   ** Исключение Esc-последовательностей управления шрифтами
   IF AT(CHR(27),Lc_buf) > 0
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"@" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"M" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"P" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"S0","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"T" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"I1","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"I2","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"I3","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"E" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"F" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"G" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"H" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"W1","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"W0","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"w1","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"w0","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"-1","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"-0","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"0" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"2" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"3/","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"37","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"3","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"h","")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"4" ,"")
      Lc_buf=STRTRAN(Lc_buf,CHR(27)+"5" ,"")
   ENDIF
   Lc_buf=STRTRAN(Lc_buf,"4","")
   Lc_buf=STRTRAN(Lc_buf,"5","")

   ** Исключение последовательностей управления шрифтами редактора LEXICON
   IF AT(CHR(255),Lc_buf) > 0
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"_","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"0","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"1","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"2","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"3","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"4","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"5","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"6","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"7","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"8","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)+"9","")
      Lc_buf=STRTRAN(Lc_buf,CHR(255)    ,"")
   ENDIF

   ******** Вставка пробелов после знаков
   Lc_buf = ALLTRIM(STRTRAN(Lc_buf,":",": "))
   Lc_buf = ALLTRIM(STRTRAN(Lc_buf,";","; "))
   Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"?","? "))
   Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"!","! "))
   *** Если после точки или запятой заглавная латинская буква
   FOR j=65 TO 90
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"."+CHR(j),". "+CHR(j)))
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,","+CHR(j),", "+CHR(j)))
   NEXT
   *** Если после точки или запятой маленькая латинская буква
   FOR j=97 TO 122
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"."+CHR(j),". "+CHR(j)))
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,","+CHR(j),", "+CHR(j)))
   NEXT
   *** Если после точки или запятой заглавная русская буква
   FOR j=128 TO 159
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"."+CHR(j),". "+CHR(j)))
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,","+CHR(j),", "+CHR(j)))
   NEXT
   *** Если после точки или запятой маленькая русская буква
   FOR j=160 TO 175
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"."+CHR(j),". "+CHR(j)))
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,","+CHR(j),", "+CHR(j)))
   NEXT
   FOR j=224 TO 239
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,"."+CHR(j),". "+CHR(j)))
       Lc_buf = ALLTRIM(STRTRAN(Lc_buf,","+CHR(j),", "+CHR(j)))
   NEXT

   ****** Удаление подряд идущих пробелов
   FOR j=10 TO 2 STEP -1
       Lc_buf=STRTRAN(Lc_buf,SPACE(j)," ")
   NEXT

   ****** Удаление пробелов в начале списков строк и абзацев,
   ****** а также после строк с DOS-переносами слов
   FOR i=1 TO 3
       FOR j=1 TO LEN(List)
           Lc_buf=STRTRAN(Lc_buf,Nm+SPACE(i)+List[j],Nm+List[j])
       NEXT
       Lc_buf=STRTRAN(Lc_buf,CHR(01)+SPACE(i),CHR(01))
       Lc_buf=STRTRAN(Lc_buf,"-"+Nm+SPACE(i),"-"+Nm)
   NEXT

   *** Исключение ВСЕХ управляющих символов с кодами от 0 до 31,
   *** кроме конца абзаца (1310) и кода абзаца 01
   FOR j=0 TO 31
       IF j <> 13 .AND. j <> 10 .AND. j <> 01
          Lc_buf=STRTRAN(Lc_buf,CHR(j),"")
       ENDIF
   NEXT

   ****** Кодирование списков строк через дефис "-"
   Lc_buf=STRTRAN(Lc_buf,Nm+"-" ,CHR(02))

   ****** Кодирование списков строк через дефис "*"
   Lc_buf=STRTRAN(Lc_buf,Nm+"*" ,CHR(03))

   ****** Кодирование списков пронумерованных строк
   Lc_buf=STRTRAN(Lc_buf,Nm+"1" ,CHR(04))
   Lc_buf=STRTRAN(Lc_buf,Nm+"2" ,CHR(05))
   Lc_buf=STRTRAN(Lc_buf,Nm+"3" ,CHR(06))
   Lc_buf=STRTRAN(Lc_buf,Nm+"4" ,CHR(07))
   Lc_buf=STRTRAN(Lc_buf,Nm+"5" ,CHR(08))
   Lc_buf=STRTRAN(Lc_buf,Nm+"6" ,CHR(09))
   Lc_buf=STRTRAN(Lc_buf,Nm+"7" ,CHR(11))
   Lc_buf=STRTRAN(Lc_buf,Nm+"8" ,CHR(12))
   Lc_buf=STRTRAN(Lc_buf,Nm+"9" ,CHR(14))

   ****** Кодирование списков строк, пронумерованных через русские буквы
   Lc_buf=STRTRAN(Lc_buf,Nm+"а)",CHR(15))      &&  1
   Lc_buf=STRTRAN(Lc_buf,Nm+"б)",CHR(16))      &&  2
   Lc_buf=STRTRAN(Lc_buf,Nm+"в)",CHR(17))      &&  3
   Lc_buf=STRTRAN(Lc_buf,Nm+"г)",CHR(18))      &&  4
   Lc_buf=STRTRAN(Lc_buf,Nm+"д)",CHR(19))      &&  5
   Lc_buf=STRTRAN(Lc_buf,Nm+"е)",CHR(20))      &&  6
   Lc_buf=STRTRAN(Lc_buf,Nm+"ж)",CHR(21))      &&  7
   Lc_buf=STRTRAN(Lc_buf,Nm+"з)",CHR(22))      &&  8
   Lc_buf=STRTRAN(Lc_buf,Nm+"и)",CHR(23))      &&  9

   ****** Кодирование списков строк, пронумерованных через латинские буквы
   Lc_buf=STRTRAN(Lc_buf,Nm+"a)",CHR(24))      &&  1
   Lc_buf=STRTRAN(Lc_buf,Nm+"b)",CHR(25))      &&  2
   Lc_buf=STRTRAN(Lc_buf,Nm+"c)",CHR(26))      &&  3
   Lc_buf=STRTRAN(Lc_buf,Nm+"d)",CHR(28))      &&  4
   Lc_buf=STRTRAN(Lc_buf,Nm+"e)",CHR(29))      &&  5
   Lc_buf=STRTRAN(Lc_buf,Nm+"f)",CHR(30))      &&  6
   Lc_buf=STRTRAN(Lc_buf,Nm+"g)",CHR(31))      &&  7

   ****** Удаление DOS-переносов
   Lc_buf=STRTRAN(Lc_buf,"-"+Nm,"")

   ****** Удаление ВСЕХ концов строк
   Lc_buf=STRTRAN(Lc_buf,Nm," ")

   ****** Восстановление "красных строк" (концов абзацев)
   Lc_buf=STRTRAN(Lc_buf,CHR(01),Nm)

   ****** Восстановление списков строк через дефис "-"
   Lc_buf=STRTRAN(Lc_buf,CHR(02),Nm+"-")

   ****** Восстановление списков строк через дефис "*"
   Lc_buf=STRTRAN(Lc_buf,CHR(03),Nm+"*")

   ****** Восстановление списков пронумерованных строк
   Lc_buf=STRTRAN(Lc_buf,CHR(04),Nm+"1")       &&  1
   Lc_buf=STRTRAN(Lc_buf,CHR(05),Nm+"2")       &&  2
   Lc_buf=STRTRAN(Lc_buf,CHR(06),Nm+"3")       &&  3
   Lc_buf=STRTRAN(Lc_buf,CHR(07),Nm+"4")       &&  4
   Lc_buf=STRTRAN(Lc_buf,CHR(08),Nm+"5")       &&  5
   Lc_buf=STRTRAN(Lc_buf,CHR(09),Nm+"6")       &&  6
   Lc_buf=STRTRAN(Lc_buf,CHR(11),Nm+"7")       &&  7
   Lc_buf=STRTRAN(Lc_buf,CHR(12),Nm+"8")       &&  8
   Lc_buf=STRTRAN(Lc_buf,CHR(14),Nm+"9")       &&  9

   ****** Восстановление списков строк, пронумерованных через русские буквы
   Lc_buf=STRTRAN(Lc_buf,CHR(15),Nm+"а)")      &&  1
   Lc_buf=STRTRAN(Lc_buf,CHR(16),Nm+"б)")      &&  2
   Lc_buf=STRTRAN(Lc_buf,CHR(17),Nm+"в)")      &&  3
   Lc_buf=STRTRAN(Lc_buf,CHR(18),Nm+"г)")      &&  4
   Lc_buf=STRTRAN(Lc_buf,CHR(19),Nm+"д)")      &&  5
   Lc_buf=STRTRAN(Lc_buf,CHR(20),Nm+"е)")      &&  6
   Lc_buf=STRTRAN(Lc_buf,CHR(21),Nm+"ж)")      &&  7
   Lc_buf=STRTRAN(Lc_buf,CHR(22),Nm+"з)")      &&  8
   Lc_buf=STRTRAN(Lc_buf,CHR(23),Nm+"и)")      &&  9

   ****** Восстановление списков строк, пронумерованных через латинские буквы
   Lc_buf=STRTRAN(Lc_buf,CHR(24),Nm+"a)")      &&  1
   Lc_buf=STRTRAN(Lc_buf,CHR(25),Nm+"b)")      &&  2
   Lc_buf=STRTRAN(Lc_buf,CHR(26),Nm+"c)")      &&  3
   Lc_buf=STRTRAN(Lc_buf,CHR(28),Nm+"d)")      &&  4
   Lc_buf=STRTRAN(Lc_buf,CHR(29),Nm+"e)")      &&  5
   Lc_buf=STRTRAN(Lc_buf,CHR(30),Nm+"f)")      &&  6
   Lc_buf=STRTRAN(Lc_buf,CHR(31),Nm+"g)")      &&  7

   *** Исключение ВСЕХ управляющих символов с кодами от 0 до 31,
   *** кроме конца абзаца (1310)
   FOR j=0 TO 31
       IF j <> 13 .AND. j <> 10
          Lc_buf=STRTRAN(Lc_buf,CHR(j),"")
       ENDIF
   NEXT

   ***** Запись фрагмента файла
   Len_x = STRFILE(Lc_buf,Fn_out,.T.,y,.F.)

   x = x + Delta
   y = y + Len_x

   p=x/File_size*100;p=IF(p<=100,p,100)
   @24,0 SAY STR(p,3)+"%" COLOR "rg+/r+"
   @24,4 SAY REPLICATE("█",p*0.76) COLOR "rb+/n"

ENDDO

RESTSCREEN(0,0,24,79,scr_start)
SHOWTIME()
QUIT

*                                 ▄▄▄▄▄▄▄▄▄▄▄
*██████████████████████████████████ The end ██████████████████████████████████
*                                 ▀▀▀▀▀▀▀▀▀▀▀

[Auge_Ohr]

hi Victorio,

when have a Xeon with 12 Core and 64 Bit OS you can run 10 x Xbase++ App on "different Core" !

Core 1 : Windows a usual
Core 2 : "Master App" call "Client App"
Core 3 : "Client App" run on Core 3
...
Core 12 : "Client App" run on Core 12

this Way you can reduce to 1/10 of Time

! Note : you need 2 GB for each Core

---
"Master App"

Code: Select all

PROCEDURE Main(cExe,cPara,cCpu)

// call Client App
   nCpu := VAL(cCpu)
   SmpSetCPU(2**(nCpu-1))
   // now RunShell()
   RunShell(cPara,cExe,.T.,.T.)

---
"Client Apps"

Code: Select all

PROCEDURE MAIN( cCPU )

   IF PCOUNT() > 0
      nCpu := VAL( cCPU )
      nWorkCPU := 2 ^ INT( nCpu - 1 )
   ELSE
      Msgbox("NEED run on which CPU Core ?")
      RETURN
   ENDIF

---

[Victorio]

Jimmy,
I am using 12 cores cpu, not for all operations but this work very good several years.
Not simple is communication between modules running by runshell but also this I have and using database when every module updating own status in field for cpu 1 to cpu 12.
when all fields cpu are in status passed main module know process full passed.
Just today admin change Ram from 8 to 16 GB, this maybe help to better performance.