Efficiency: printing n times vs append n times, then print

[Habby]

Hey guys  I'm working on a code that prints a huge amount of lines, to be more precise 1,080,000. Each line has around 15-18 variables in it , the thing is we are printing every line individually which is taking the whole process to be done in about 18 to 21 hours. I'm wondering if it'd be faster/more efficient to append each line to a list and then print the whole list in one go and if there could be any problems with the amount memory required to do this. The workstation has 60Gigs of ram.

Hope I was able to express myself clearly

[Fleetscut]

are you printing to the screen or a file? the reason printing to a screen is so slow is that the buffer used to hold the data has to be flushed constantly. if you dont want to see live updates of every line then look to see if the language you are using has a way to control how often the buffer is flushed to the screen. or just write to a file.

[reniat]

Is there a reason you need to print out all 1 million lines? It would probably be faster to write it out to a file in chunks than put it out via console I/O

[geo3]

1m lines should not take 18hours. Are you running on a 286? Printing to a mechanical printer?  Is anything else happening in the loop besides printing?

 

from datetime import datetime
def timeit():
    start =  datetime.now()
    for i in range(0, 1000000):
        print 'This is a line of text that is only so long. It shouldn\' take that long to print a million of them', i
    print start, datetime.now()

 

This take less than a minute on a i7 3770. Python is renowned for not being a fast language.

[Mira Yurizaki]

If you need to output that many lines, you should be putting it into a file, not printing it to the console. As mentioned by @Fleetscut, printing to the console or anywhere is a relatively expensive operation. On top of that, consoles can only retain so many lines before the previous ones are lost. Unless you need to monitor the output in real time, you're going to be missing basically 99% of the information you wanted printed out once the script is done.