Python and Oracle : Fetching records and setting buffer size

    Mar 5, 2018 11:10:00 AM by Brendan Tierney

    If you used other languages, including Oracle PL/SQL, more than likely you will have experienced having to play buffering the number of records that are returned from a cursor. Typically this is needed when you are processing more than a few hundred records. The default buffering size is relatively small and by increasing the size of the number of records to be buffered can dramatically improve the performance of your code.

    As with all things in coding and IT, the phrase "It Depends" applies here and changing the buffering size may not be what you need and my not help you to gain optimal performance for your code.

    There are lots and lots of examples of how to test this in PL/SQL and other languages, but what I'm going to show you here in this blog post is to change the buffering size when using Python to process data in an Oracle Database using the Oracle Python library cx_Oracle.

    Let us begin with taking the defaults and seeing what happens. In this first scenario the default buffering is used. Here we execute a query and the process the records in a FOR loop (yes these is a row-by-row, slow-by-slow approach.

    import time
    i = 0
    # define a cursor to use with the connection
    cur2 = con.cursor()
    # execute a query returning the results to the cursor
    print("Starting cursor at", time.ctime())
    cur2.execute('select * from sh.customers')
    print("Finished cursor at", time.ctime())

    # for each row returned to the cursor, print the record
    print("Starting for loop", time.ctime())
    t0 = time.time()
    for row in cur2:
    i = i+1
    if (i%10000) == 0:
    print(i,"records processed", time.ctime())

    t1 = time.time()
    print("Finished for loop at", time.ctime())
    print("Number of records counted = ", i)

    ttime = t1 - t0
    print("in ", ttime, "seconds.")

    This gives us the following output.

    Starting cursor at  10:11:43
    Finished cursor at 10:11:43
    Starting for loop 10:11:43
    10000 records processed 10:11:49
    20000 records processed 10:11:54
    30000 records processed 10:11:59
    40000 records processed 10:12:05
    50000 records processed 10:12:09
    Finished for loop at 10:12:11
    Number of records counted = 55500
    in 28.398550033569336 seconds.

    Processing the data this way takes approx. 28 seconds and this corresponds to the buffering of approx 50-75 records at a time. This involves many, many, many round trips to the the database to retrieve this data. This default processing might be fine when our query is only retrieving a small number of records, but as our data set or results set from the query increases so does the time it takes to process the query.

    But we have a simple way of reducing the time taken, as the number of records in our results set increases. We can do this by increasing the number of records that are buffered. This can be done by changing the size of the 'arrysize' for the cursor definition. This reduces the number of "roundtrips" made to the database, often reducing networks load and reducing the number of context switches on the database server.

    The following gives an example of same code with one additional line.

    cur2.arraysize = 500

    Here is the full code example.

    # Test : Change the arraysize and see what impact that has
    import time
    i = 0
    # define a cursor to use with the connection
    cur2 = con.cursor()
    cur2.arraysize = 500
    # execute a query returning the results to the cursor
    print("Starting cursor at", time.ctime())
    cur2.execute('select * from sh.customers')
    print("Finished cursor at", time.ctime())
    # for each row returned to the cursor, print the record
    print("Starting for loop", time.ctime())
    t0 = time.time()
    for row in cur2:
    i = i+1
    if (i%10000) == 0:
    print(i,"records processed", time.ctime())

    t1 = time.time()
    print("Finished for loop at", time.ctime())
    print("Number of records counted = ", i)
    ttime = t1 - t0
    print("in ", ttime, "seconds.")

    Now the response time to process all the records is.

    Starting cursor at 10:13:02 Finished cursor at 10:13:02 Starting for loop 10:13:02 10000 records processed 10:13:04 20000 records processed 10:13:06 30000 records processed 10:13:08 40000 records processed 10:13:10 50000 records processed 10:13:12 Finished for loop at 10:13:13 Number of records counted = 55500 in 11.780734777450562 seconds.

    All done in just under 12 seconds, compared to 28 seconds previously.

    Here is another alternative way of processing the data and retrieves the entire results set, using the 'fetchall' command, and stores it located in 'res'.

    # Test : Change the arraysize and see what impact that has
    import time
    i = 0
    # define a cursor to use with the connection
    cur2 = con.cursor()
    cur2.arraysize = 500
    # execute a query returning the results to the cursor
    print("Starting cursor at", time.ctime())
    cur2.execute('select * from sh.customers')
    t0 = time.time()
    print("starting FetchAll at", time.ctime())
    res = cur2.fetchall()
    t1 = time.time()
    print("finished FetchAll at", time.ctime())
    ttime = t1 - t0
    print("in ", ttime, "seconds.") 

    Tags: Oracle

    Brendan Tierney

    Written by Brendan Tierney

    Brendan Tierney, Oracle ACE Director, is an independent consultant and lectures on Data Mining and Advanced Databases in the Dublin Institute of Technology in Ireland. He has 22+ years of extensive experience working in the areas of Data Mining, Data Warehousing, Data Architecture and Database Design. Brendan has worked on projects in Ireland, UK, Belgium and USA. Brendan is the editor of the UKOUG Oracle Scene magazine and deputy chair of the OUG Ireland BI SIG. Brendan is a regular speaker at conferences across Europe and the USA and has written technical articles for OTN, Oracle Scene, IOUG SELECT Journal and ODTUG Technical Journal. Brendan has published the following books with Oracle Press book Predictive Analytics using Oracle Data Miner Oracle R Enterprise SQL & PL/SQL from the Experts These books are available on Amazon.