Mmap to Read Files in Reverse Python

How to improve file reading performance in Python with MMAP function?

---

Introduction..

MMAP abbreviated equally memory mapping when mapped to a file uses the operating systems virtual memory to access the data on the file system directly, instead of accessing the data with the normal I/O functions. In that location past improving the I/O performance as it does not require either making a split up system phone call for each access or copying information between buffers.

To a matter of fact anything in memory for instance a SQLlite database when created in-memeory has improve performance compared to on disk.

Retentivity-mapped files can be treated as mutable strings or file-similar objects, depending on what you desire to exercise.

MMAP supports many methods, such every bit close(), affluent(), read(), readline(), seek(), tell(), write() and can very well work with piece operations and even regular expressions.

How to exercise information technology..

ane. Assume a text file with below contents. You can get this text by only using Google and searching for sample text. Copy these contents to a input.txt file.

Lorem ipsum dolor sit down amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens european union has, in natum meliore noluisse sea. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus ad sea, interesset delicatissimi et sit down. Purto molestiae cu eum, in per hinc periculis intellegam.

Id porro facete cum. No est veritus detraxit facilisis, sit ea clita decore essent. Ut eam labores fuisset menandri, ex sit animal viderer eleifend, altera argumentum vel ex. Duo at zril sensibus, eu vim ullum assentior, quando possit at his.

Te nam tempor posidonium scripserit, eam mundi reprimique dissentias ne. Vim te soleat offendit democritum. Nam an diam elaboraret, quaeque dissentias an has. Autem legendos dignissim ad vis, sea ex amet petentium reprehendunt, inermis constituam philosophia ne mel. Esse noster lobortis usu ne.

Nec reque postea urbanitas ut, mea in nulla invidunt ocurreret. Ei duo iuvaret numquam. Ferri nemore audire te est, mel et detracto noluisse. Nec eu habeo justo, id pro posse apeirian volutpat. Mea sonet quaestio ne.

Atqui quaeque alienum te vim. Graeco aliquip liberavisse pro ut. Te similique reformidans usu, te mundi aliquando ius. Meis scripta minimum quo no, meis prima fabellas eu eam, laoreet delicata forensibus ut vim. Et quo vocibus mediocritatem, atqui summo an eam.

2. We volition Use the mmap() function to create a memory-mapped file. We tin pass the filename either past fileno() method of a file object or from os.open().

Notation: The user is responsible for opening the file before invoking mmap(), and closing it.

The second argument to mmap() is a size in bytes indicating the portion of the file to map. If the value is 0, the entire file is mapped. There is also an additional argument you can use which is ACCESS_READ for read-just access, ACCESS_WRITE for write-through admission, and ACCESS_COPY for copy on write access.

import mmap  input_text = """Lorem ipsum dolor sit down amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens european union has, in natum meliore noluisse bounding main. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus advertising sea, interesset delicatissimi et sit. Purto molestiae cu eum, in per hinc periculis intellegam.  Id porro facete cum. No est veritus detraxit facilisis, sit ea clita decore essent. Ut eam labores fuisset menandri, ex sit down animate being viderer eleifend, altera argumentum vel ex. Duo at zril sensibus, eu vim ullum assentior, quando possit at his.  Te nam tempor posidonium scripserit, eam mundi reprimique dissentias ne. Vim te soleat offendit democritum. Nam an diam elaboraret, quaeque dissentias an has. Autem legendos dignissim ad vis, ocean ex amet petentium reprehendunt, inermis constituam philosophia ne mel. Esse noster lobortis usu ne.  Nec reque postea urbanitas ut, mea in nulla invidunt ocurreret. Ei duo iuvaret numquam. Ferri nemore audire te est, mel et detracto noluisse. Nec eu habeo justo, id pro posse apeirian volutpat. Mea sonet quaestio ne.  Atqui quaeque alienum te vim. Graeco aliquip liberavisse pro ut. Te similique reformidans usu, te mundi aliquando ius. Meis scripta minimum quo no, meis prima fabellas european union eam, laoreet delicata forensibus ut vim. Et quo vocibus mediocritatem, atqui summo an eam.  """  # create a inout file with some text input_file = 'input.txt' f = open(input_file, "w+") f.write(input_text) f.shut()  #Open the file in read style with open up(input_file, 'r') as f: with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m: impress(f"Output \n*** Output beginning 5 bytes of the {input_file} is {m.read(5)} ") print(f"*** Output Adjacent 10 bytes of the {input_file} is {m.read(10)} ")

Output

*** Output first 5 bytes of the input.txt is b'Lorem' *** Output Next x bytes of the input.txt is b' ipsum dol'

3. We have read the file and mapped to memory and used .read() to read the first 5 bytes. So the file pointer moves ahead of 10 bytes afterwards first read. Now if y'all do one more than read lets say read(10) bytes it gives y'all bytes from six - 15.

4. To prepare the memory mapped file to update, open information technology for 'r+' (not 'due west') before mapping it.

I will evidence you with an instance how to modify the part of a line in-place.

import mmap import shutil  input_file = 'input.txt' input_copy = input_file.replace('input','input_copy')  # Make a Re-create of the file just to make certain original is un-modified. shutil.copyfile(input_file,input_copy)  # word word = b'ipsum'  # modified word modified_word = word[::-1]  # Open the file to receive updates with open up(input_copy, 'r+') as f: with mmap.mmap(f.fileno(), 0) as m: print(f"output \n *** Line before updates \north {m.readline().rstrip()}")  # Rewind using seek m.seek(0)  # find the word and contrary information technology loc = m.find(word) one thousand[loc:loc + len(give-and-take)] = modified_word thousand.flush()  # Rewind using seek m.seek(0) impress(f" \n *** Line later updates \n {m.readline().rstrip()}")  f.seek(0) print(f" \n *** Final file \n {f.readline().rstrip()}")

Output

*** Line before updates b'Lorem ipsum dolor sit amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens eu has, in natum meliore noluisse sea. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus ad sea, interesset delicatissimi et sit down. Purto molestiae cu eum, in per hinc periculis intellegam.'  *** Line later updates b'Lorem muspi dolor sit amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens eu has, in natum meliore noluisse sea. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus advertising sea, interesset delicatissimi et sit down. Purto molestiae cu eum, in per hinc periculis intellegam.'  *** Final file Lorem muspi dolor sit amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens eu has, in natum meliore noluisse body of water. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus advertizement sea, interesset delicatissimi et sit down. Purto molestiae cu eum, in per hinc periculis intellegam.

5. The word "ipsum" is replaced in the heart of the showtime line in memory and in the file.

half-dozen. If, for whatever reason yous desire to come across the changes in-memory and don't want to update the file on disk, then use ACCESS_COPY.

import mmap import shutil  input_file = 'input.txt' input_copy = input_file.supercede('input','input_copy')  # Brand a Copy of the file just to make sure original is un-modified. shutil.copyfile(input_file,input_copy)  # discussion word = b'ipsum'  # modified word modified_word = word[::-1]  # Open up the file to receive updates with open up(input_copy, 'r+') every bit f: with mmap.mmap(f.fileno(), 0, admission=mmap.ACCESS_COPY) as yard: print(f"output \n *** Line before updates \due north {k.readline().rstrip()}")  # Rewind using seek grand.seek(0)  # find the word and opposite it loc = m.notice(give-and-take) m[loc:loc + len(give-and-take)] = modified_word m.flush()  # Rewind using seek m.seek(0) print(f" \n *** Line later updates \n {k.readline().rstrip()}")  f.seek(0) impress(f" \due north *** Concluding file \n {f.readline().rstrip()}")

Output

*** Line before updates b'Lorem ipsum dolor sit amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens eu has, in natum meliore noluisse sea. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus advertising sea, interesset delicatissimi et sit. Purto molestiae cu eum, in per hinc periculis intellegam.'  *** Line after updates b'Lorem muspi dolor sit down amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens eu has, in natum meliore noluisse sea. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus ad bounding main, interesset delicatissimi et sit. Purto molestiae cu eum, in per hinc periculis intellegam.'  *** Final file Lorem ipsum dolor sit amet, causae apeirian ea his, duo cu congue prodesset. Ut epicuri invenire duo, novum ridens european union has, in natum meliore noluisse sea. Has ei stet explicari. No nam eirmod deterruisset, nusquam electram rationibus advertising ocean, interesset delicatissimi et sit. Purto molestiae cu eum, in per hinc periculis intellegam.

7. Observe, the content in the input and output which are unchanged, while the changes applied only to in-memory.

Published on 09-Nov-2020 11:01:22

• Related Questions & Answers
• Better MySQL Search Performance with wildcards (%%)?
• Memory-mapped file support in Python (mmap)?
• How to improve the performance of JavaScript?
• How to improve the performance of my Python CGI script?
• How does pipelining improve performance in estimator architecture?
• Ameliorate operation of a HTML5 Canvas with particles bouncing around
• Reading and Writing CSV File using Python
• How can I better performance of a Laptop or PC?
• Opening and reading a file with askopenfilename in Tkinter?
• What are the best practices to improve jQuery selector performance?
• Reading a Text file in java
• How to open an Excel file with PHPExcel for both reading and writing?
• Dissimilar ways of Reading a file in C#
• Reading and writing binary file in C/C++
• Reading/Writing a MS Word file in PHP

strakerthadell43.blogspot.com

Source: https://www.tutorialspoint.com/how-to-improve-file-reading-performance-in-python-with-mmap-function

Share this post